Group Composition

Portrait of Yuan Dong

Yuan is an MSCV student of class Fall 2018. He received his bachelor’s degree in automation from Tsinghua University. His research focuses on realistic human reconstruction and human keypoint detection. He built the pipeline for this project, as well as some major components like visual hull constrained Poisson reconstruction, body template fitting and personalized optimization.


Portrait of Congrui Hetang

Congrui is an MSCV student of class Fall 2018. He received his bachelor’s degree in computer science from Beijing University of Aeronautics and Astronautics. His research focuses on semantic segmentation and object detection. He developed a solution for high-resolution human segmentation for this project, and also developed some parts of mesh refinement.


We build a reconstruction pipeline to generate photo-realistic human model from high resolution multi-view RGB images, by the composition of several tranditional methods, from multi-view stereo to poisson to linear-blend-skinning template fit and finally refinement, and thre result is very well.

Detailed Description

  • Introduction

We’re trying to build realistic human models for virtual reality. The goal is to build human body models from multi-view RGB images given camera parameters, with highest possible level of detail. The high quality human we reconstruct may help overcome the uncanny valley problem, and eventually enable the next generation of gaming and social applications in VR context.


  • Device and Setting

There are more than one hundred cameras, all of 4K resolution, half of which are colored and the other half are grayscale. All cameras are pre-calibrated. The subject being captured is inside a capture room with static background. 

On the left is an image of our capture room. All the black dots are cameras. On the right is a showcase of a standard capture setup, with a subject standing inside the room.


  • Pipeline

The pipeline consists of several components. First, dense multi-view reconstruction is performed on all the images captured. We perform foreground segmentation for each image to acquire a high-resolution human mask, which builds a visual hull that constraints the reconstruction. After that, a parametrized, personalized human template mesh will be optimized to fit its shape and pose to the reconstructed mesh. This involves an optimization process developed by us, including several loss terms which will be introduced later. Finally, some additional optimization will be applied to the human template, which further improves the level of details.


  • High-Resolution Foreground Segmentation

In order to deal with all the difficulty induced by shadow, color confusion and grayscale images, we developed an end-to-end encoder-decoder network for high-resolution human foreground segmentation. We trained this network with synthetic data and it’s able to generate masks with pixel-level accuracy regardless of color or grayscale. Checkout the detailed structure in our slides.  Here are some example results:


  • MVS and Poisson Reconstruction

Standard multi-view stereo reconstruction only gives us a pointcloud. We use Poisson reconstruction to turn that into a mesh. In this process we leverage a visual hull generated by the segmentation masks to improve the result mesh: outlier points with wrong normals can be filtered out. This is implemented by inspecting the indicator oct-tree nodes. Here are some examples of colored Poisson reconstruction mesh.



  • Fitting Personalized and Parametrized Body Template

In this step we optimize the pose and shape parameters of a personalized body template to fit it closely with the reconstructed mesh. The template mesh uses linear blend skinning model to model how the skeleton affects surface shape. There is a kinematic chain as shown in the figure below. We also demonstrate a visualization of the human-designed skinning weights.


Here is an example of a reconstructed mesh and the fitted mesh.


  • Further Refinements

There are two additional refinement steps after fitting the parametrized body mesh. The first one is called local blendshape model. The body mesh is divided into a large number of small patches, each of which has its own blendshape model and parameters. The second one is per-vertex deformation, where each vertex will have a 3-vector assigned to it to further model the finest surface deformation. Here we show the comparison with and without these refinements.


  • Results


A video example:

Final presentation

See our slides at