{"id":84,"date":"2024-12-06T22:28:56","date_gmt":"2024-12-06T22:28:56","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2024team16\/?page_id=84"},"modified":"2024-12-06T22:47:14","modified_gmt":"2024-12-06T22:47:14","slug":"jing-project-page","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2024team16\/","title":{"rendered":"Jing’s Project Page"},"content":{"rendered":"\n

DiSRT-In-Bed: Diffusion-Based Sim-to-Real Transfer Framework for In-Bed Human Mesh Recovery<\/strong><\/h1>\n\n\n\n
\n\n\n\n
\n
\n
\n
\n
\"\"<\/figure>\n<\/div><\/div>\n<\/div>\n\n\n\n

Jing Gao<\/em><\/p>\n<\/div>\n<\/div><\/div>\n\n\n\n

Introduction<\/strong><\/h2>\n\n\n\n

In-bed human mesh recovery can be crucial and enabling for several healthcare applications, including sleep pattern monitoring, rehabilitation support, and pressure ulcer prevention. However, it is difficult to collect large real-world visual datasets in this domain, in part due to privacy and expense constraints, which in turn presents significant challenges for training and deploying deep learning models. Existing in-bed human mesh estimation methods often rely heavily on real-world data, limiting their ability to generalize across different in-bed scenarios, such as varying coverings and environmental settings. To address the challenge of limited training data, utilizing synthetic data presents a promising solution. Large-scale simulated depth datasets can be efficiently generated without preserving any personally identifiable information, leading to eliminating the need for sensitive real-world data. Building on this strategy, prior work has demonstrated good performance in the in-bed human mesh recovery task. However, they struggle to effectively bridge the domain gap between synthetic and real-world data, leading to significant performance degradation when the proportion of real-world data in the training set is low, as shown in the following figure.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

To address this, we propose a Sim-to-Real Transfer Framework for in-bed human mesh recovery from overhead depth images, which leverages large-scale synthetic data alongside limited or no real-world samples. We introduce a diffusion model that bridges the gap between synthetic data and real data to support generalization in real-world in-bed pose and body inference scenarios. <\/p>\n\n\n\n

Method<\/strong><\/h2>\n\n\n\n

We utilize the SMPL model, a parametric human body model representing 3D human bodies as a mesh of 6,890 vertices, controlled by pose and shape parameters. Our framework tackles the challenge of in-bed human mesh recovery with minimal real-world data by leveraging synthetic data and limited real-world samples. It consists of three stages: synthetic data generation, model design, and pipeline training and fine-tuning, as illustrated in the following figure.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Synthetic Data Generation<\/h3>\n\n\n\n

Obtaining labeled data for in-bed healthcare scenarios is challenging, limiting deep learning deployment. Simulation offers a low-cost solution, generating high-quality depth data with ground truth annotations for human mesh in resting positions. Using prior information like bed dimensions and camera-to-bed distance, we simulate realistic environments. Following BodyPressure, a physics-based simulation pipeline, we generate synthetic in-bed human depth data by simulating SMPL-based bodies resting on a soft mattress with a fixed camera. By sampling human shapes, joint angles, and translations, we create diverse data representing various body types and coverings.<\/p>\n\n\n\n

Diffusion-Based In-Bed Mesh Recovery<\/h3>\n\n\n\n

In contrast to the diffusion process used in image generation, which operates directly on images, we conduct forward noise-adding and reverse denoising processes on the SMPL body parameters for in-bed human mesh recovery.<\/p>\n\n\n\n

In the forward<\/em> process, we follow<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

to obtain noisy versions xt<\/sub> of the initial SMPL body parameters x0<\/sub> over t timesteps.<\/p>\n\n\n\n

In the reverse<\/em> process, we incorporate depth image c as a conditional input to the diffusion model, modifying the equation as follows:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

we diffuse toward the initial sample x0<\/sub> to ensure consistency throughout the denoising process, as illustrated in the framework. The estimated initial SMLP parameters can be represented as:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

The objective of training and fine-tuning the diffusion model D for in-bed human mesh recovery is to minimize:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

In the inference, the learned mean can be formulated as:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Then, we sample from the transition distribution in each denoising step to compute xt-1<\/sub> as follows:<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Following this reverse process, we iteratively denoise the SMPL latent from noise xT<\/sub> at timestep T down to the target SMPL latent x0<\/sub> at timestep 0.<\/p>\n\n\n\n

Model Architecture<\/h3>\n\n\n\n

We introduce a network that takes noisy SMPL parameter latent, depth images , and timestep as inputs and outputs the denoised SMPL parameters, as illustrated in the following architecture figure. The depth images condition the reverse diffusion process, while an MLP encoder processes noisy latent, a uniform sampler generates the time embedding. Inspired by the diffusion U-Net design, the network employs residual and attention blocks with adaptive normalization layers initialized to zero. The SMPL latent and time embedding are aligned to enable dynamic adjustments of normalization parameters via a linear MLP. The architecture consists of down-sampling residual blocks, attention blocks, and a regressor that maps output latents to SMPL parameters. The SMPL model then generates the human mesh and 3D joint positions, controlled by gender flags. This design effectively processes conditional information, enabling accurate human mesh recovery.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Training Strategy<\/h3>\n\n\n\n