{"id":25,"date":"2023-05-08T21:10:40","date_gmt":"2023-05-08T21:10:40","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team16\/?page_id=25"},"modified":"2023-12-18T23:17:03","modified_gmt":"2023-12-18T23:17:03","slug":"spring","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team16\/spring\/","title":{"rendered":"Spring 2023"},"content":{"rendered":"\n

Overview <\/h1>\n\n\n\n

Our goal is to frame a methodology to perform learning-based multi-view vision-only depth estimation <\/strong>with large distortion lenses <\/strong>in real-time <\/strong>on edge devices<\/strong>.<\/p>\n\n\n\n

Baseline<\/h2>\n\n\n\n

One of our baseline methods is OmniMVS<\/strong> [1], which proposed an end-to-end network architecture for the omni-directional stereo depth estimation. The graph below gives an overview of OmniMVS architecture:<\/p>\n\n\n\n


\"\"

<\/p>\n\n\n\n

First, the input images are processed by a 2D CNN to extract feature maps. These unary feature maps are then projected into spherical features to create a matching cost volume. The final depth is obtained by computing the cost volume using a 3D encoder-decoder architecture and softargmax.<\/p>\n\n\n\n

To generate the spherical features, OmniMVS uses a technique called spherical sweeping<\/strong>, which is a modified version of the plane-sweeping algorithm designed for ultra-wide FOV images. As shown by the graph below, the coordinate system used for this method is based on the rig, with the y-axis defined as perpendicular to the plane closest to all camera centers, and the origin located at the center of the projected camera centers. The spherical sweeping generates a set of spheres with varying radii, representing different depth candidates, and builds the spherical images of each input image.<\/p>\n\n\n\n


\"\"\"\"
<\/p>\n\n\n\n

The pixel value of the equirectangular spherical image warped onto the n-th sphere is determined as, 
\"\"\"\"
The depth results from previous methods such as SweepNet often struggle to <\/strong>handle multiple true matches<\/strong>, resulting in incorrect depth estimates for thin objects that are overridden by the background depth. These methods also face challenges when dealing with large textureless regions<\/strong> such as ceilings. In contrast, OmniMVS leverages multi-view observations and global context information to address these issues, leading to improved depth estimation results as shown in the last row of the results.<\/p>\n\n\n\n


\"\"<\/p>\n\n\n\n

Despite its advantages, OmniMVS is not without limitations<\/strong>. There are instances where it may mispredict depth, such as when stairs are parallel to the stereo baseline<\/strong> or when objects<\/strong> are very close to the rig<\/strong>. However, the paper proposes the use of an uncertainty measure<\/strong> that can indicate complex and ambiguous regions in depth mapping, which can help mitigate these problems [1].<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Baseline results<\/h2>\n\n\n\n

The results from the baseline method on the 6-camera rig:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Approaches<\/h1>\n\n\n\n

1. Temporal Sequence Information<\/h2>\n\n\n\n


To improve the efficiency<\/strong>, the top k depth estimation from past keyframes<\/strong> with offset learning will be incorporated when building the spherical volume. Similar approaches were shown to have SOTA MAE\/MRE results in Matterport3D and 360D datasets by [2]. In addition, 2D context information<\/strong> from the input fisheye images will also be considered to further enhance the accuracy<\/strong> of the depth estimation.<\/p>\n\n\n\n


\"\"
<\/p>\n\n\n\n

2. Fisheye Space Convolutions<\/h2>\n\n\n\n

Traditional convolutions assume spatial feature invariance (i.e. content-agnostic) which doesn’t apply when working with fisheye images. Instead, we plan to use,<\/p>\n\n\n\n