Datasets
Traditional Stereo Datasets
Due to the difficulty of getting labeled ground truth depth estimates, there are only a limited number of datasets. The most popular ones are KITTI, Middlebury, and ETH3D. However, these datasets are almost entirely focused on daylight with very few low-light environmental conditions.
Oxford Robot Car
Contains a large number of outdoor driving around Oxford at day and night time, as well as various weather conditions.
NREC Collected Data
The above datasets, while may be helpful to some extent, are certainly not in the same domain as the environments for our targeted application. Mainly, our environment contains not just low-light but also off-road environments for which there aren’t any publicly available datasets yet. As such, NREC has collected their own data at various locations that are closer to our actual domain. An example of one such locations during day-time is shown.
Results
Evaluation Metrics
We use standard evaluation metrics listed below from KITTI and Middlebury datasets.
- Average Error – The average error between ground truth disparity and predicted disparity over the entire image and dataset.
- Bad-X Error – The percentage of pixels in the entire image which have an error > X in disparity.
We trained and evaluated Deep HSM on the KITTI dataset. The Deep HSM model was initially trained on a large synthetic dataset, and then fine-tuned on KITTI. Below are the results.
| Distance (meters) | Average Error | Bad-1 % | Bad-4 % |
| 0-25 | 1.245 | 27.1 | 3.5 |
| 25-60 | 1.147 | 27.7 | 3.9 |
| 60-115 | 1.196 | 39.0 | 4.3 |
| 0-115 | 1.231 | 30.4 | 4.1 |
Visualization Results
We showcase the following:
Input Image
Predicted Disparity Map
Entropy
We observe that edges of objects have the highest uncertainty as well as overall there is higher entropy in darker images as shown in the examples below.
