{"id":90,"date":"2025-05-09T19:50:01","date_gmt":"2025-05-09T19:50:01","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/?page_id=90"},"modified":"2025-12-10T22:15:43","modified_gmt":"2025-12-10T22:15:43","slug":"background","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team14\/background\/","title":{"rendered":"Background"},"content":{"rendered":"\n

Pointmap formulation for 3D reconstruction<\/h3>\n\n\n\n

The pointmap formulation reimagines 3D reconstruction as a dense prediction task, where each pixel in an image is mapped to a corresponding 3D point in space, resulting in a (H, W, 3) tensor. This dense 2D-3D correspondence framework facilitates the extraction of various geometric properties, such as depth estimation, camera calibration, and pose estimation, all from the same representation.<\/p>\n\n\n\n

The DUSt3R [1] model (Dense and Unconstrained Stereo 3D Reconstruction) leverages this approach by predicting pointmaps for pairs of images, aligning both in the coordinate frame of the first image. This method eliminates the need for prior camera calibration or known poses, simplifying the 3D reconstruction pipeline.<\/p>\n\n\n\n

Although the DUST3R architecture is tailored for 3D reconstruction from a single image pair, it can be scaled to multi-view settings by applying a non-linear optimization strategy to globally align multiple image pairs.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

VGGT<\/h3>\n\n\n\n

VGGT [2] improves upon pointmap-based methods like DUSt3R by eliminating the need for expensive global alignment. Instead, it employs a streamlined pipeline with 24 pairs of frame-wise and global self-attention layers, allowing it to jointly reason across all views. VGGT also predicts dense depth, camera poses, pointmaps, and point tracks in a single forward pass. This unified design enables state-of-the-art performance in multi-view 3D reconstruction\u2014without post-processing\u2014using a multitask loss to supervise all outputs jointly.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Comparing 3D reconstruction methods<\/h3>\n\n\n\n

To support our goal of leveraging 3D reconstruction for robotic manipulation, we compared several 3D reconstruction method. Key evaluation metrics include reconstruction accuracy, inference time, temporal consistency across frames, and the ability to recover absolute depth. The results are presented in the table below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n


RaySt3R \u2013 single-view RGB-D to full 3D shape completion<\/h3>\n\n\n\n

RaySt3R [3] is a model for zero-shot 3D object completion<\/strong>: it completes the 3D shape of partially observed objects by predicting what their full geometry should look like.\u00a0<\/p>\n\n\n\n