Improving HOSIG
We are building a baseline HOI/HSI pipeline on top of HOSIG[2], whose base model was trained on only 3 objects using the right hand. To improve generalization, we are retraining it with a larger, filtered dataset from TRUMANS[5] covering greater object diversity and closer grasping poses.
In doing so, we identified the SGAP module as the primary source of common artifacts, including body-scene penetrations, hand-object penetrations, and unnatural poses for out-of-distribution objects.
SGAP works in two main stages, the GNet cVAE pose generation phase followed up by an iterative contact based loss optimisation on hand and object contact.
Improving the base GNet [stage 1]
We retrained the SGAP model with a the new dataset and refined losses to address this. The results below show a clear improvement in grasping quality, specially for the penetration metrics.
Improving pose refinement [stage 2]
We replace the original single-step, full-body optimization with a 3-stage approach that prioritizes hand/arm refinement before body pose, removes overpowering losses like gaze, and applies a lower learning rate for body adjustments
Video gen based HOI
One of the core challenges in HOI generation is generalizing to novel objects and diverse motion patterns. We explore using video generation as a prior to address this — leveraging WAN[6] to synthesize videos from captions and a starting frame, using the first frame’s depth and scene geometry as a conditioning signal. Object trajectories and hand poses are then recovered in parallel via FoundationPose[6] and HaMeR[7], producing a 3D HOI output.
We are also exploring how to integrate video-based HOI generation with full-body motion, toward a unified pipeline.
Proposed framework
Ultimately, these components will be integrated into a cohesive pipeline, enhanced with a VLM planner to reason over visual inputs and determine appropriate interactions, and a human-in-the-loop framework for controllability
