{"id":24,"date":"2025-05-06T08:51:29","date_gmt":"2025-05-06T08:51:29","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team13\/?page_id=24"},"modified":"2025-12-13T03:23:32","modified_gmt":"2025-12-13T03:23:32","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team13\/method\/","title":{"rendered":"\u200b"},"content":{"rendered":"\n

Method<\/strong><\/h1>\n\n\n\n

Our work focuses on to reducing the embodiment gap between humans and humanoid robots by performing retargeted reconstruction. Conventional learning-from-demonstration pipelines first reconstruct a human using a parametric body model such as SMPL, followed by a separate retargeting step to obtain a humanoid pose. This paradigm becomes limiting when an appropriate parametric mesh is unavailable, particularly for non-humanoid robots.<\/p>\n\n\n\n

In contrast our approach leverages meshes that already exist in physics simulators and directly reconstruct the geometry of a posed robot from an image of its biological counterpart (e.g., reconstruct a quadruped from a dog or a humanoid from a human). <\/p>\n\n\n\n

By avoiding an intermediate reconstruction and retargeting step, our approach produces outputs directly in the target robot geometry, enabling pose learning in an interpretable and end-to-end manner. <\/p>\n\n\n\n

We demonstrate a specific example use case of our approach by reconstructing a G1 humanoid from humans.<\/p>\n\n\n\n

Method Overview<\/strong><\/h2>\n\n\n\n

We propose a reconstruction pipeline trained on a wide diversity of labelled data, generated using generative models. We provide an overview of our proposed method below:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Our pipeline consists of three stages: <\/p>\n\n\n\n

    \n
  1. Data generation<\/li>\n\n\n\n
  2. Point map prediction<\/li>\n\n\n\n
  3. Joint angle estimation<\/li>\n<\/ol>\n\n\n\n

    For training, we generate paired data of synthetic humans with their corresponding ground-truth humanoid pose and geometry. We then use this data to train our point map predictor and joint angle estimator. <\/p>\n\n\n\n

    At inference time, we take in image sequences of real humans and predict out the corresponding equivalent robot geometry and robot joint angles.<\/p>\n\n\n\n

    Data Generation<\/strong><\/h2>\n\n\n\n

    Our data generation pipeline synthesizes humans of a similar morphology as the G1 humanoid by leveraging generative models. We show an example of a generated sequence below:<\/p>\n\n\n

    \n
    \"\"<\/figure>\n<\/div>\n\n\n

    Creating robot motions<\/strong><\/h4>\n\n\n\n

    We leverage motion is sourced from the MoVi dataset, which provides paired videos and SMPL-H body meshes. We retarget SMPL-H to the G1 humanoid robot using Perpetual Humanoid Control to obtain robot joint angles that faithfully reproduce the original motion. This provides us with a diverse set of robot motions which we leverage for training and real human-robot pairs for validation.<\/p>\n\n\n

    \n
    \"\"<\/figure>\n<\/div>\n\n\n

    Generating Diverse Images<\/strong><\/h4>\n\n\n\n

    We generate a diverse set of human images by prompting Image GPT to generate images of people of different ages, ethnicity, clothing and backgrounds.<\/p>\n\n\n

    \n
    \"\"<\/figure>\n<\/div>\n\n\n

    Editing to create Human-Robot data pairs<\/strong><\/h4>\n\n\n\n

    We then reanimate these images of people such that they are in the same morphology and pose of the G1 robot using MagicDance. We use the driving motion in the video below to demonstrate how we animate a G1 robot and then use the tracked keypoints to generate humans of a robot’s morphology for training our model.<\/p>\n\n\n

    \n
    \"\"<\/figure>\n<\/div>\n\n\n