EXPERIMENTAL SETUP

Data-Driven SfM pipeline

As mentioned in the introduction, we want to solve SfM using data-driving priors. We created a driver which utilizes different learning feature extractions and learning feature matching methods to see how the data-driven priors can help with sets of few Images.

Datasets

Custom Daily Common Objects

We created our own dataset which consists of 360-degree images of 11 objects that are found in our day-to-day activities. Each category consists of multi-view images with 10-23 viewpoints. To establish a baseline we first tested the data-driven SfM pipeline on our own datasets. Link to the dataset.

Example of our own dataset

Results

This is the visualization of the camera pose (pyramids) and 3D point clouds (red-points) estimated using hloc

Co3D v2

We tested the data-driven SfM pipeline on the Co3D v2 dataset. The Co3D V2 dataset is designed for learning category-specific 3D reconstruction and new-view synthesis using multi-view images of common object categories.

The dataset contains a total of 1.5 million frames from nearly 19,000 videos capturing objects from 50 MS-COCO categories, and, as such, it is significantly larger than alternatives both in terms of the number of categories and objects.

Co3D v2 dataset categories

We chose one of the hydrant categories from the Co3D v2 dataset and ran the data-driven SfM pipeline on it. We randomly sampled different numbers of images from the dataset with the provided ground-truth cameras and compared the average rotation error and the number of estimated poses with different inputs. 

Results

Number of imagesAverage rotation errorNumber of poses estimated
500.05650
200.66514
10Not Converged0
Visualize results
the blue cameras are the ground truth cameras provided in the dataset, and the red ones are predictions from the SfM pipeline.