Summary<\/h3>\n\n\n\n![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/omnimvs-1024x296.png\")
<\/figure>\n\n\n\n- Sphere Sweep (C) – Every pixel of the six fisheye images is projected onto n spheres. These spheres lie evenly between the minimum and maximum depth to be measured.<\/li>
- For every pair of images, the n spherical representations are integrated and then averaged over to form a 4D cost volume.<\/li>
- The cost volume is then passed through a 3D encoder-decoder to generate the depth map (D).<\/li>
- To save on computation and memory, sphere sweep is run on feature maps generated by a 2D CNN feature extractor (B) instead of the images themselves.<\/li><\/ul>\n\n\n\n
Uncertainty Prior<\/h3>\n\n\n\n
In case of hard matching cases such as textureless, reflective, or occluded regions, the depths predicted might not be accurate enough. In such cases, having a confidence measure of the model predictions would provide a good indication of where the model doesn’t perform well. This measure would provide the model a great training signal to penalize such scenarios and detections appropriately.<\/p>\n\n\n\n
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/occlusion.png\")
View occluded due to the battery pack<\/figcaption><\/figure><\/div>\n\n\n\nOne of the novel ideas presented in this paper included having an uncertainty prior. The uncertainty, measured using entropy, information would be used to filter out depths with uncertainty higher than a threshold. This information would also be used to enforce an uncertainty loss thus, forcing the model to learn such cases explicitly.<\/p>\n\n\n\n
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/uncertainty-prior.png\")
<\/figure><\/div>\n\n\n\nThe graphs above two scenarios of depth prediction. In (a), the depth is predicted with high probability. The predicted depth is close to the ground truth depth and so, the overall loss is low and the uncertainty loss is low as well. In (b), however, the depth predicted is very close to other depth candidates around it. Fortunately, the predicted depth turned out to be very close to the ground truth depth. So, the loss is low. However, due to the low and distributed probabilities, the entropy is high and the overall uncertainty loss is high. The model is thus, forced to disambiguate such scenarios.<\/p>\n\n\n\n
This uncertainty prior is yet to be implemented.<\/p>\n\n\n\n
Analysis<\/h2>\n\n\n\nDouble Sphere Camera Model<\/h3>\n\n\n\n
Due to the extreme distortion that a fisheye lens produces, the pinhole camera model doesn’t apply. There are several models such as the Fisheye Projection Model[1]<\/a><\/sup> and the Kannala-Brandt Camera Model[2]<\/a><\/sup> which could be used. However, the inverse projection (3D point to 2D) either requires solving an optimization problem or expensive trigonometric operations. Enter Double Sphere Camera Model[3]<\/a><\/sup>.<\/p>\n\n\n\n![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/double-sphere.gif\")
<\/figure><\/div>\n\n\n\nIn order to verify that this model worked and to test that the project and inverse projection matched, we performed a small test. By taking a mask similar to a fisheye image, we projected each pixel to 3D space and then reprojected these back to the image plane. Both these operations were performed using the formulae defined by the Double Sphere Camera Model. The resulting pixel locations were compared with the original pixel locations and the projection errors were plotted.<\/p>\n\n\n\n
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/ds-proj-err.png\")
<\/figure><\/div>\n\n\n\nAs can be observed, the maximum error was below a pixel separation of 0.0001 which was definitely acceptable. Thus, this camera model would work well for our project.<\/p>\n\n\n\n
Determining Output Size<\/h3>\n\n\n\n
As part of our pipeline, the six fisheye images are stitched together into a panorama image.<\/p>\n\n\n\n
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/pan-1024x453.png\")
<\/figure><\/div>\n\n\n\nThe input fisheye size is fixed due to the compute constraints and the requirements of the other modules. \u200b However, the output panorama resolution needs to be determined.\u200b <\/p>\n\n\n\n
- It can\u2019t be too low or else we wouldn’t be able to leverage the information captured by the input images.\u200b<\/li>
- It can\u2019t be too high or else there is a possibility that an input image pixel might correspond<\/strong> to more than 1 output pixel.\u200b This could lead to incorrect\/blurry depth estimation.<\/li><\/ul>\n\n\n\n
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/image27-1024x567.png\")
Calculating projection error<\/figcaption><\/figure><\/div>\n\n\n\nTo calculate this correspondence<\/p>\n\n\n\n
- An output pixel (denoted by
(theta, phi)<\/code>) is projected to the 3D space in the Panorama frame – (x, y, z)<\/code>.<\/li>- It is then moved to the 3D space in a particular Camera\u2019s frame –
(x', y', z')<\/code>.\u200b<\/li>- Using the Double Sphere model, it is inversely projected onto the fisheye image plane.\u200b<\/li>
- This is done for each output pixel and the difference with its neighbors is calculated.<\/li><\/ul>\n\n\n\n
Using the above methodology, we conducted our analysis.<\/p>\n\n\n\n![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/image34.png\")
Mean Separation for 1st 1.0 % rows vs Heights for an input size of 1224×1028<\/figcaption><\/figure>\n\n\n\nFor an input size of 1224x1028<\/code>, we found that at a height of 540px (and a width of 1080px – maintaining an aspect ratio of 2), the mean separation error for the 1st 1% rows just goes below 1px. Therefore, for any heights less than or equal to 540px, the minimum pixel separation would be above 1. This can be seen in the actual projection error heatmap as well.<\/p>\n\n\n\n![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/image35.png\")
Projection Error Heatmap for a top-facing camera. The below half is invalid due to the FoV of the fisheye restricting the camera to view beyond 1950<\/sup><\/figcaption><\/figure>\n\n\n\nThe reason why we consider the top rows is because that’s the area, in a panorama image, that constitutes maximum distortion. Unlike the part of panorama images that represent forward, left, or right, the top view is covered by the entire width and so, the limited input pixels corresponding to the top view need to make up for the entire top rows. This would lead to maximum duplication of information. <\/p>\n\n\n\n
Alternative Approach<\/h2>\n\n\n\nSliceNet<\/h3>\n\n\n\n
<\/figure>\n\n\n\n- Sphere Sweep (C) – Every pixel of the six fisheye images is projected onto n spheres. These spheres lie evenly between the minimum and maximum depth to be measured.<\/li>
- For every pair of images, the n spherical representations are integrated and then averaged over to form a 4D cost volume.<\/li>
- The cost volume is then passed through a 3D encoder-decoder to generate the depth map (D).<\/li>
- To save on computation and memory, sphere sweep is run on feature maps generated by a 2D CNN feature extractor (B) instead of the images themselves.<\/li><\/ul>\n\n\n\n
Uncertainty Prior<\/h3>\n\n\n\n
In case of hard matching cases such as textureless, reflective, or occluded regions, the depths predicted might not be accurate enough. In such cases, having a confidence measure of the model predictions would provide a good indication of where the model doesn’t perform well. This measure would provide the model a great training signal to penalize such scenarios and detections appropriately.<\/p>\n\n\n\n
View occluded due to the battery pack<\/figcaption><\/figure><\/div>\n\n\n\n One of the novel ideas presented in this paper included having an uncertainty prior. The uncertainty, measured using entropy, information would be used to filter out depths with uncertainty higher than a threshold. This information would also be used to enforce an uncertainty loss thus, forcing the model to learn such cases explicitly.<\/p>\n\n\n\n
<\/figure><\/div>\n\n\n\nThe graphs above two scenarios of depth prediction. In (a), the depth is predicted with high probability. The predicted depth is close to the ground truth depth and so, the overall loss is low and the uncertainty loss is low as well. In (b), however, the depth predicted is very close to other depth candidates around it. Fortunately, the predicted depth turned out to be very close to the ground truth depth. So, the loss is low. However, due to the low and distributed probabilities, the entropy is high and the overall uncertainty loss is high. The model is thus, forced to disambiguate such scenarios.<\/p>\n\n\n\n
This uncertainty prior is yet to be implemented.<\/p>\n\n\n\n
Analysis<\/h2>\n\n\n\n
Double Sphere Camera Model<\/h3>\n\n\n\n
Due to the extreme distortion that a fisheye lens produces, the pinhole camera model doesn’t apply. There are several models such as the Fisheye Projection Model[1]<\/a><\/sup> and the Kannala-Brandt Camera Model[2]<\/a><\/sup> which could be used. However, the inverse projection (3D point to 2D) either requires solving an optimization problem or expensive trigonometric operations. Enter Double Sphere Camera Model[3]<\/a><\/sup>.<\/p>\n\n\n\n
<\/figure><\/div>\n\n\n\nIn order to verify that this model worked and to test that the project and inverse projection matched, we performed a small test. By taking a mask similar to a fisheye image, we projected each pixel to 3D space and then reprojected these back to the image plane. Both these operations were performed using the formulae defined by the Double Sphere Camera Model. The resulting pixel locations were compared with the original pixel locations and the projection errors were plotted.<\/p>\n\n\n\n
<\/figure><\/div>\n\n\n\nAs can be observed, the maximum error was below a pixel separation of 0.0001 which was definitely acceptable. Thus, this camera model would work well for our project.<\/p>\n\n\n\n
Determining Output Size<\/h3>\n\n\n\n
As part of our pipeline, the six fisheye images are stitched together into a panorama image.<\/p>\n\n\n\n
<\/figure><\/div>\n\n\n\nThe input fisheye size is fixed due to the compute constraints and the requirements of the other modules. \u200b However, the output panorama resolution needs to be determined.\u200b <\/p>\n\n\n\n
- It can\u2019t be too low or else we wouldn’t be able to leverage the information captured by the input images.\u200b<\/li>
- It can\u2019t be too high or else there is a possibility that an input image pixel might correspond<\/strong> to more than 1 output pixel.\u200b This could lead to incorrect\/blurry depth estimation.<\/li><\/ul>\n\n\n\n
Calculating projection error<\/figcaption><\/figure><\/div>\n\n\n\n To calculate this correspondence<\/p>\n\n\n\n
- An output pixel (denoted by
(theta, phi)<\/code>) is projected to the 3D space in the Panorama frame –(x, y, z)<\/code>.<\/li>- It is then moved to the 3D space in a particular Camera\u2019s frame –
(x', y', z')<\/code>.\u200b<\/li>- Using the Double Sphere model, it is inversely projected onto the fisheye image plane.\u200b<\/li>
- This is done for each output pixel and the difference with its neighbors is calculated.<\/li><\/ul>\n\n\n\n
Using the above methodology, we conducted our analysis.<\/p>\n\n\n\n
Mean Separation for 1st 1.0 % rows vs Heights for an input size of 1224×1028<\/figcaption><\/figure>\n\n\n\n For an input size of
1224x1028<\/code>, we found that at a height of 540px (and a width of 1080px – maintaining an aspect ratio of 2), the mean separation error for the 1st 1% rows just goes below 1px. Therefore, for any heights less than or equal to 540px, the minimum pixel separation would be above 1. This can be seen in the actual projection error heatmap as well.<\/p>\n\n\n\nProjection Error Heatmap for a top-facing camera. The below half is invalid due to the FoV of the fisheye restricting the camera to view beyond 1950<\/sup><\/figcaption><\/figure>\n\n\n\n The reason why we consider the top rows is because that’s the area, in a panorama image, that constitutes maximum distortion. Unlike the part of panorama images that represent forward, left, or right, the top view is covered by the entire width and so, the limited input pixels corresponding to the top view need to make up for the entire top rows. This would lead to maximum duplication of information. <\/p>\n\n\n\n
Alternative Approach<\/h2>\n\n\n\n
SliceNet<\/h3>\n\n\n\n
- It is then moved to the 3D space in a particular Camera\u2019s frame –
- An output pixel (denoted by
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/image38-1024x362.png\")
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/image39.png\")
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/image40.png\")
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-18-02-05-1024x290.png\")
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-18-08-10.png\")
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/data.gif\")
<\/figure><\/div>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-18-24-00.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-18-27-07.png\")
![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-52-37.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-52-44.png\")
<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/gt_0.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/pred_0.png\")
<\/td><\/tr>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/gt_1.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/pred_1.png\")
<\/td><\/tr>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/gt_2.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/pred_2.png\")
<\/td><\/tr>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/gt_3.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/pred_3.png\")
<\/td><\/tr>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/gt_7.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/04\/pred_7.png\")
<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-52-21-1.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-52-28-1.png\")
<\/td><\/tr>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-53-06.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-53-13.png\")
<\/td><\/tr>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-52-51.png\")
<\/td>![\"\"](\"http:\/\/mscvprojects.ri.cmu.edu\/2022team6\/wp-content\/uploads\/sites\/61\/2022\/12\/Screenshot-from-2022-12-18-17-52-57.png\")
<\/td><\/tr>