Method<\/p>\n\n\n\n
As shown in Figure 1, we propose repurposing a video diffusion model, Stable Video Diffusion (SVD), to achieve highly accurate and generalizable video amodal segmentation. One key insight is that foundational diffusion models trained to generate pixels also bake-in strong priors on object shape.<\/p>\n\n\n\n
The first stage of our pipeline generates amodal masks {At<\/sub>} for an object, given its modal masks {Mt<\/sub>}and pseudo-depth of the scene {Dt<\/sub>} (which is obtained by running a monocular depth estimator on RGB video sequence {It<\/sub>}). The predicted amodal masks from the first stage are then sent as input to the second stage, along with the modal RGB content of the occluded object in consideration. The second stage then inpaints the occluded region and outputs the amodal RGB content {Ct<\/sub>} for the occluded object. Both stages employ a conditional latent diffusion framework with a 3D UNet backbone. Conditionings are encoded via a VAE encoder into latent space, concatenated, and processed by a 3D UNet with interleaved spatial and temporal blocks. CLIP embeddings of {Mt<\/sub>} and the modal RGB content provide cross-attention cues for the first and second stage respectively. Finally, the VAE decoder translates outputs back to pixel space.<\/p>\n\n\n\n Figure 2: Model pipeline for amodal segmentation and content completion<\/p>\n\n\n\n A key challenge with the stage 2 in our method (amodal content completion) is the lack of ground-truth RGB content in occluded regions, even in synthetic datasets like SAIL-VOS. Inspired by self-supervised training-pair construction used extensively in image amodal tasks, we extend this approach to video sequences. Figure 3 illustrates an example of a modal-amodal RGB content training pair. To construct such a pair, we first select an object from the dataset with near-complete visibility (above 95%). We then sequentially overlay random amodal mask sequences onto this fully visible object until its visibility falls below a set threshold, thereby simulating occlusion. This effectively generates ground-truth RGB data for the occluded regions.<\/p>\n\n\n\n Figure 3: Modal-amodal RGB training pair for content completion<\/p>\n\n\n\n Quantitative Results<\/p>\n\n\n\n Table 1 shows the quantitative comparisons on SAIL-VOS and TAO-Amodal, where our method surpasses all baselines. Notably, it achieves nearly 13% improvement over the second-best method, PCNet-M, in terms of mIoUocc<\/sub>, highlighting effective completion of occluded object regions. Despite being trained exclusively on synthetic SAIL-VOS, a zero-shot evaluation on TAO-Amodal highlights the strong generalization of our model. We posit that, in addition to leveraging foundational knowledge and rich priors from the large-scale pretraining of SVD, our model is able to learn temporal cues that help it amodally complete any unseen object classes from neighboring frames.<\/p>\n\n\n Table 1: Quantitative comparison on SAIL-VOS and TAO-Amodal<\/p>\n\n\n\n Table 2 provides quantitative comparisons on the MOVi-B\/D datasets, where our method beats the prior state-of-the-art. Despite strong camera motion in MOVi-B\/D, our model adapts well without<\/em> access to camera extrinsics or optical flow (unlike some baselines). We posit that our method is able to use the 3D priors from Stable Video Diffusion and is therefore, successfully able to maintain consistent object shapes from different view-points.<\/p>\n\n\n Table 2: Quantitative Comparison on MOVi-B\/D<\/p>\n\n\n\n Qualitative Results<\/p>\n\n\n\n In Figure 4, we show the qualitative comparison across diverse dataset. Our method leverages strong shape priors, such as for humans, chairs, and teapots, to generate clean and realistic object shapes. It also excels in handling heavy occlusions; even when objects are nearly fully occluded (e.g., “chair” in the second row of SAIL-VOS), our method achieves high-fidelity shape completion by utilizing temporal priors. Note that TAO-Amodal contains out-of-frame occlusions which none of the methods are trained for, but our method is able to handle such cases.<\/p>\n\n\n\n Figure 4: Qualitative comparison of amodal segmentation methods across diverse datasets<\/p>\n\n\n\n In Figure 5, we show the qualitative results for content completion. Although our content completion module, initialized from pretrained SVD weights, is finetuned solely on synthetic SAIL-VOS, it achieves photorealistic, high-fidelity object inpainting even in real-world scenarios. Furthermore, our method can complete unseen<\/em> categories, such as giraffes and plastic bottle, likely due to its ability to transfer styles and patterns from the visible parts of objects to occluded areas in the current or neighboring frames. We show examples from TAO-Amodal (top) and in-the-wild YouTube videos (bottom).<\/p>\n\n\n\n Figure 5: Qualitative results for content completion<\/p>\n\n\n\n In Figure 6, we highlight the lack of temporal coherence<\/em> in a single-frame diffusion based method, pix2gestalt, for both the predicted amodal segmentation mask and the RGB content for the occluded person in the example shown. By leveraging temporal priors, our approach achieves significantly higher temporal consistency across occlusions.<\/p>\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-model-1024x461.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-data_pair-1024x318.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/image.png\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/image-1.png\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-Comparison.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-Content-Comp-1024x186.png\")
<\/figure>\n\n\n\n
![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-temporal_consistency-1024x567.png\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/mscvprojects.ri.cmu.edu\/2024team7\/wp-content\/uploads\/sites\/105\/2024\/12\/Figure-multimodal-1024x211.png\")
<\/figure>\n\n\n\n