The KAIST Dataset has multiple sequences of pairs of RGB and Thermal images. Initially built as a pedestrian detection benchmark, this dataset is one of the very few publicly available datasets to have RGB and Thermal together. Moreover, this data has images from all day and different weather conditions making it suitable for our experimentation.<\/p>\n\n\n\n NREC Collected Data<\/strong> Pseudo Ground Truth<\/strong><\/p>\n\n\n\n We use the Monodepth2 architecture to train our models. In addition to the re-projection loss, we also investigate adding extra supervision with the pseudo-ground truth for RGB and thermal models. All of our results can be seen below.<\/p>\n\n\n\n We train Monodepth2 with both RGB and thermal with both self-supervision and stereo pseudo-ground truth. We notice that the thermal model and the RGB model have a large difference in the self-supervised case for the strictest delta threshold. However, when we add the pseudo ground-truth, we are able to close the gap between RGB and thermal almost entirely and achieve really good performance. Since thermal images have less of a domain shift from day to night, our model is likely able to perform the same in day and night, meaning our model is nearly as good as if the RGB model was running on day time!<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Our Datasets KAIST Multi-Spectral Dataset The KAIST Dataset has multiple sequences of pairs of RGB and Thermal images. Initially built as a pedestrian detection benchmark, this dataset is one of the very few publicly available datasets to have RGB and Thermal together. Moreover, this data has images from all day and different weather conditions making … <\/p>\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/Screen-Shot-2022-12-16-at-11.39.45-PM-1024x405.png\")
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/ezgif.com-gif-maker-3-1024x474.jpg\")
The NREC dataset contains not just low-light but also off-road environments for which there aren\u2019t any publicly available datasets yet. As such, NREC has collected its own data at various locations that are closer to our actual domain. An example of one such location during daytime is shown. We seek to train our models on daytime data and evaluate them in both day and night conditions. Note, however, we do not have any ground truth depth. As such we rely on creating pseudo ground truth from the vertical stereo pair.<\/p>\n<\/div>\n<\/div>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/ezgif.com-gif-maker-4-1024x373.jpg\")
Results Overview<\/h2>\n\n\n\n
RGB-Based Depth Estimation<\/h2>\n\n\n\n
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/ezgif.com-gif-maker-5-1024x370.jpg\")
Thermal-Based Depth Estimation<\/h2>\n\n\n\n
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/ezgif.com-gif-maker-6-1024x393.jpg\")
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/ezgif.com-gif-maker-1.gif\")
Quantitative Analysis<\/h2>\n\n\n\n
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team7\/wp-content\/uploads\/sites\/62\/2022\/12\/ezgif.com-gif-maker-7-1024x265.jpg\")