Multiface: A Dataset for Neural Face Rendering
The Multiface dataset consists of multi-view data corresponding to 13 different subjects that are captured using the Mugsy (as mentioned previously). The dataset is a combination of raw images, unwrapped textures, tracked meshes, head poses, audio and metadata consisting of camera calibration parameters, texture mean and variance, etc. The table given below gives an overview of the content of the dataset belonging to Mugsy v1 and v2.
This high-quality multi-view codec-avatar dataset is released to serve as a benchmark dataset for facial reconstruction, and neural view synthesis, expression synthesis.
We use this dataset as the means of our experimentation because it has ground truth camera calibration data.
Structure-from-Motion Revisited
We utilize the proposed pipeline in great detail to refine the camera intrinsics of our dataset, so we discuss the working of this approach in detail on the next page.
Major Contributions of this work:
- Deploy a general-purpose algorithm based on incremental Sfm that is much more stable, robust, complete and scalable.
- The approach of next-best view selection maximizes the robustness and accuracy of the incremental reconstruction process.
- Iterative Bundle Adjustment (Local BA) followed by a Global BA, helps outperform the current state of the art in terms of robustness and completeness while preserving its efficiency.
Camera Self-Calibration Using the Kruppa Equations and the SVD of the Fundamental Matrix: The Case of Varying Intrinsic Parameters
The paper proposes a purely algebraic derivation of the Kruppa equations that relies only on Singular Value Decomposition of the fundamental matrix. The approach is more robust due to avoiding calculation of noise-sensitive quantities, such as epipoles.
Let A, A’ be the intrinsics of a camera pair,
Then,
After computing and substituting epipoles:
To eliminate γ (unknown scalar), the above equation is decomposed into the following scalar equations:
There are only 2 independent equations out of these 6 equations. Non-linear optimisation is used to solve for the unknown K, K’ and they are further decomposed into intrinsics A, A’ using Cholesky decomposition.
The paper reports the result on real and synthetic dataset, and intrinsics parameters were evaluated through 3D Reconstruction. The approach provides satisfactory results.