Introduction
Animatable 3D human avatars are increasingly important across gaming, telepresence, VR, and AR. Traditional avatar production depends on multi-camera capture rigs or expert 3D artists, both costly enough to keep avatar creation out of reach for most users. Lowering this barrier — making avatar creation possible from inputs as casual as a single photo or monocular video — would expand access dramatically and unlock new applications for individual creators and small businesses.
Existing accessible-avatar methods take one of two paths. Per-subject optimization approaches fit a representation (typically SMPL-bound 3D Gaussians) to monocular video, taking hours of GPU time per subject; single-image variants like TeCH lean on pretrained diffusion models for supervision, pushing costs higher. Feed-forward methods avoid per-subject optimization by predicting avatars directly. Early work like PIFu produced implicit surfaces but struggled with animation and geometric consistency. The recent Large Human Model (LHM), trained on over 300K videos, marks the current state of the art in this direction.
We identify three failure modes in LHM. (1) Side and back views are blurry, reflecting the limits of single-view input. (2) Reconstructions are spatially misaligned with the input image — overlaying predictions onto inputs reveals systematic discrepancies, which we trace to the use of full attention between 3D points and image pixels. (3) Observed regions show texture over-smoothing, which we attribute to the model’s reliance on Linear Blend Skinning (LBS); LBS cannot capture non-rigid deformation, so the network compensates by blurring textures during training on real video.
To address these, we introduce Disentangled Gaussian Transformers (DGT), a multi-stream point transformer built on PTv3. A canonical T-stream maintains the subject’s T-pose representation, while a variable number of P-streams (one per input image) produce image-specific deformations as residuals on top of the LBS-deformed canonical avatar. To recover the spatial precision lost under full attention, P-streams use a Foreground Point Projection (FPP) module that tightens correspondence between the 3D point cloud and 2D image features.
Related Works
Creating animatable 3D human avatars has been a long-standing problem in graphics and vision, with early parametric body models giving way to more expressive reconstruction methods over time. Existing approaches fall broadly into two paradigms — subject-specific optimization and feed-forward reconstruction — and the former further divides by the density of input supervision available at training time.
Subject-specific optimization fits a separate model per individual, achieving high fidelity at the cost of generalization. Multiview-video methods produce strong results using implicit, explicit (3DGS-based), or hybrid representations, but require expensive calibrated capture rigs that are out of reach for most users. Monocular-video methods relax the input requirement but typically depend on near-360° coverage and degrade sharply under sparse views, where occluded regions cannot be reliably inferred.
A third line of subject-specific work compensates for sparse inputs by leveraging generative priors. Early GAN-based approaches generated avatars from limited views, while recent diffusion-based methods use Score Distillation Sampling or personalization on top of 2D or multi-view diffusion priors. While these methods generate plausible reconstructions from very sparse inputs, their reliance on iterative diffusion supervision makes them computationally expensive, limiting practical use for real-time AR/VR applications.
Method
We aim to reconstruct an animatable 3D Gaussian avatar from a flexible set of visual inputs — anywhere from a single image to multiple frames of a monocular video — paired with their corresponding SMPLX poses. The reconstructed avatar must align with each input under its given pose while remaining drivable by unseen poses for downstream animation. Our representation builds on standard 3D Gaussian Splatting anchored to SMPLX vertices, with a fixed-size canonical T-pose Gaussian set that is animated to arbitrary poses via Linear Blend Skinning. The challenge is that LBS alone cannot account for non-rigid deformations or pixel-level alignment with input images, and naive solutions either over-smooth textures or are computationally infeasible at scale.
We address this with Disentangled Gaussian Transformers (DGT), a multi-stream point transformer in which a canonical T-stream refines a shared T-pose avatar while one P-stream per input image produces image-specific residual deformations defined directly in the animated space. The streams communicate through a lightweight cross-stream mechanism that exploits the one-to-one correspondence between Gaussians across streams, avoiding the quadratic cost of full multi-view attention. A separate alignment pathway injects localized image evidence into each P-stream so that the per-image residuals correct fine-grained misalignment without disturbing the shared canonical geometry.
DGT is trained with a composite objective combining Huber RGB loss, silhouette mask loss, and LPIPS perceptual loss, applied across input views, novel views at the input poses (for both T- and P-stream outputs), and novel poses (for the T-stream only). A scale regularizer suppresses degenerate needle-shaped Gaussians. This training scheme cleanly separates what each stream is responsible for: the T-stream learns a coherent canonical avatar from all available evidence, while the P-streams absorb the per-image deformations needed for tight alignment, leaving the canonical representation clean and animatable.
Experiments
We report comparisons on Human4DiT and 4D-Dress in Table 1. DGT consistently outperforms LHM across all three metrics, showing that the projection-enhancement design and residual composition scheme effectively improve alignment and animation quality.
First, both DGT/T and DGT/P show significant alignment improvements over LHM. On IPS-IV, they achieve +3.8 dB and +5.0 dB PSNR gains, confirming that projection-enhancement improves alignment between the reconstructed avatar and the input image. DGT/P further improves on DGT/T, demonstrating the effectiveness of the residual composition scheme.
Second, both variants outperform LHM on novel view synthesis, with +1.8 dB and +2.3 dB PSNR gains on IPS-NV, indicating that the reconstructed 3D avatar is more consistent with the underlying geometry.
Third, animation quality is better than or comparable to LHM. On Human4DiT, DGT/T gains +0.6 dB over LHM; on 4D-Dress, results are similar.
For novel pose synthesis (NPS), DGT/T consistently outperforms LHM across all metrics on both datasets — improving PSNR by +0.6 dB and reducing LPIPS by 0.019 on Human4DiT, and surpassing LHM by +0.11 dB PSNR on 4D-Dress. This demonstrates stronger generalization to unseen poses and better reconstruction fidelity.
In summary, DGT clearly beats LHM on input alignment and novel view synthesis, and marginally surpasses it on animation.
Resources
Team
Soham Dasgupta , MSCV
Jianjin Xu, PhD CMU
Prof Fernando Del La Torre