{"id":38,"date":"2025-05-08T17:06:19","date_gmt":"2025-05-08T17:06:19","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/?page_id=38"},"modified":"2025-12-13T23:44:53","modified_gmt":"2025-12-13T23:44:53","slug":"experiments","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/","title":{"rendered":"Experiments"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\"><strong>Dataset<\/strong><\/h2>\n\n\n\n<p>The primary dataset we use for training is <strong>Mixkit<\/strong> from <strong>Open-Sora-Plan<\/strong><sup data-fn=\"80930205-a5bf-4ca5-b4af-6dfb06f97ddf\" class=\"fn\"><a id=\"80930205-a5bf-4ca5-b4af-6dfb06f97ddf-link\" href=\"#80930205-a5bf-4ca5-b4af-6dfb06f97ddf\">1<\/a><\/sup> dataset. It is a high-quality open-source video dataset&nbsp;, created as part of open-sora-plan project to recreate OpenAI&#8217;s Sora model. It consists of <strong>1234<\/strong> videos, the total duration is about&nbsp;<strong>6h 19m 32s<\/strong>, and the total number of frames is&nbsp;<strong>570815<\/strong>.<\/p>\n\n\n\n<p>Input data consists of sequences of:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>RGB video frames<\/strong> of resolution 1920&#215;1080 and 1080&#215;1920.<\/li>\n\n\n\n<li><strong>Text Captions<\/strong> generated as descriptions of the videos.<\/li>\n<\/ul>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"384\" height=\"216\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif\" alt=\"\" class=\"wp-image-205\" \/><figcaption class=\"wp-element-caption\">&#8220;A man traveling through the desert on his motorcycle&#8221;<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"384\" height=\"216\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-four-international-friends-watching-the-world-cup-44601-ezgif.com-resize.gif\" alt=\"\" class=\"wp-image-206\" \/><figcaption class=\"wp-element-caption\">&#8220;Four international friends watching the world cup&#8221;<\/figcaption><\/figure>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Inference<\/strong><\/h2>\n\n\n\n<p>To validate the correctness of our Megatron-based implementation of the VACE-Wan model, we first perform inference without any parallelism using both the <strong>Hugging Face VACE checkpoint<\/strong> and the <strong>Megatron Core\u2013compatible checkpoint<\/strong>. Using the same random seed, both implementations produce identical output videos. We then conduct inference using tensor parallelism alone and context parallelism alone to evaluate and compare the memory usage and runtime characteristics of these different parallelization strategies.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Finetuning<\/strong><\/h2>\n\n\n\n<p>After converting the <strong>Hugging Face VACE checkpoint<\/strong> from its original PyTorch format into a <strong>Megatron Core\u2013compatible checkpoint<\/strong>, we fine-tune the model using our custom Megatron training recipe. We enable efficient distributed training across multiple GPUs by leveraging Megatron Core\u2019s support for tensor, pipeline, context, and data parallelism. It allows us to scale sequence length and model size without exceeding memory limits. We also use mixed-precision computation and optimized attention kernels to maximize throughput and stability. During fine-tuning, we freeze the pretrained DiT backbone while training the context-adapter layer, adapting the model to support per-frame control signals and task-specific objectives. Using this recipe, we finetune for 2 tasks primarily:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Reference-Image2Video Generation<\/strong><\/h3>\n\n\n\n<p>For image-to-video (I2V) generation, we condition the model on a <strong>single reference image provided as the first frame<\/strong>, with the remaining frames initialized as noise. The reference image is encoded into the latent space and injected through the context-adapter layer into the DiT backbone. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Video Inpainting<\/strong><\/h3>\n\n\n\n<p>For video inpainting, we train the model to reconstruct missing or corrupted regions using <strong>spatio-temporal masks provided at each frame<\/strong>. The masked video latents, along with their corresponding binary masks, are used as per-frame control inputs through the context-adapter. The model learns to jointly reason over spatial structure and temporal continuity, filling in occluded regions while maintaining consistency across frames, and the loss computed selectively on masked regions, encouraging reconstruction of missing content without altering unmasked areas.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Results<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Qualitative Results<\/strong><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Finetuning<\/h4>\n\n\n\n<p>The source video used for demonstration is shown below, depicting two cats boxing on a stage. The text prompt used for demonstration is &#8220;Two dogs fit each other during boxing&#8221;.<\/p>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"720\" style=\"aspect-ratio: 1280 \/ 720;\" width=\"1280\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/sample.mp4\"><\/video><figcaption class=\"wp-element-caption\">Source video<\/figcaption><\/figure>\n\n\n\n<p>When conditioned on the depth video, as shown below, the VACE model performs decently, not only aligning with the text description but also maintaining strong temporal consistency and fluid motion.<\/p>\n\n\n\n<div class=\"wp-block-group is-content-justification-center is-nowrap is-layout-flex wp-container-core-group-is-layout-94bc23d7 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_depth-1.mp4\"><\/video><figcaption class=\"wp-element-caption\">Depth video<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_Two_dogs_hit_each_other_during_boxing_depth_scale_1.mp4\"><\/video><figcaption class=\"wp-element-caption\">Output video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<p>When conditioned on the flow video, as shown below, the VACE model achieves the best performance, producing a clearer background and reduced distortion of the gloves.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_flow.mp4\"><\/video><figcaption class=\"wp-element-caption\">Flow video<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_Two_dogs_hit_each_other_during_boxing_flow_scale_1.mp4\"><\/video><figcaption class=\"wp-element-caption\">Output video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<p>However, when conditioned on the pose video, as shown below, the VACE model fails to achieve comparable quality to the depth and flow conditions, exhibiting deformed or missing gloves and inconsistent motion. Upon closer inspection, we observe that the pose video deviates significantly from the actual poses in the source video, indicating the need for a more accurate off-the-shelf pose estimation model. This also suggests that the VACE model is sensitive to the quality of the conditioning video.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_pose.mp4\"><\/video><figcaption class=\"wp-element-caption\">Pose video<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_Two_dogs_hit_each_other_during_boxing_pose_scale_1.mp4\"><\/video><figcaption class=\"wp-element-caption\">Output video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<p>When additionally conditioned on a reference image of a golden retriever, the VACE model successfully replaces the dog on the right with one exhibiting golden fur.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-8cf370e7 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"857\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1024x857.jpg\" alt=\"\" class=\"wp-image-294\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1024x857.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-300x251.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-768x642.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1536x1285.jpg 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-2048x1713.jpg 2048w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Reference image<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_depth-3.mp4\"><\/video><figcaption class=\"wp-element-caption\">Depth video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_depth_ref_Two_dogs_hit_each_other_during_boxing-2.mp4\"><\/video><figcaption class=\"wp-element-caption\">Output video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<p>When additionally conditioned on a reference image of a golden retriever, the VACE model successfully replaces the dog on the right with one exhibiting golden fur.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-8cf370e7 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"857\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1024x857.jpg\" alt=\"\" class=\"wp-image-294\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1024x857.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-300x251.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-768x642.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1536x1285.jpg 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-2048x1713.jpg 2048w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Reference image<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_flow-1.mp4\"><\/video><figcaption class=\"wp-element-caption\">Flow video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_flow_ref_Two_dogs_hit_each_other_during_boxing.mp4\"><\/video><figcaption class=\"wp-element-caption\">Output video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<p>When additionally conditioned on a reference image of a golden retriever, the VACE model successfully replaces the dog on the right with one exhibiting golden fur.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-8cf370e7 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"857\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1024x857.jpg\" alt=\"\" class=\"wp-image-294\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1024x857.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-300x251.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-768x642.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-1536x1285.jpg 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/Golden_Retriever_1-2048x1713.jpg 2048w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Reference image<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_pose-1.mp4\"><\/video><figcaption class=\"wp-element-caption\">Pose video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"480\" style=\"aspect-ratio: 832 \/ 480;\" width=\"832\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_pose_ref_Two_dogs_hit_each_other_during_boxing.mp4\"><\/video><figcaption class=\"wp-element-caption\">Output video<\/figcaption><\/figure>\n<\/div>\n\n\n\n<h4 class=\"wp-block-heading\">Finetuning<\/h4>\n\n\n\n<p>Reference Image to video generation results<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-video\"><video height=\"360\" style=\"aspect-ratio: 240 \/ 360;\" width=\"240\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video-frameref-online-video-cutter.com_.mp4\"><\/video><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"360\" style=\"aspect-ratio: 240 \/ 360;\" width=\"240\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-MB_1000_Cat_jumps_from_the_cabinet._20251202_204617-online-video-cutter.com_.mp4\"><\/video><\/figure>\n<\/div>\n\n\n\n<p>Inpainting results<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-ad2f72ca wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-video\"><video height=\"360\" style=\"aspect-ratio: 640 \/ 360;\" width=\"640\" controls src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/src_video_obj_1-online-video-cutter.com_.mp4\"><\/video><\/figure>\n\n\n\n<figure class=\"wp-block-video\"><video height=\"360\" style=\"aspect-ratio: 624 \/ 360;\" width=\"624\" controls src=\"http:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/vace-1.3B_test_videoindex0_size832_480-online-video-cutter.com_.mp4\"><\/video><\/figure>\n<\/div>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Quantitative Results<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-table has-small-font-size\"><table><thead><tr><th>  Mode<\/th><th>Model<\/th><th>Setup<\/th><th>GPU Memory (per GPU)<\/th><th>Runtime<\/th><\/tr><\/thead><tbody><tr><td rowspan=\"4\">  Inference<\/td><td>VACE-Base<\/td><td>1\u00d7 GPU (HuggingFace)<\/td><td>28 GB<\/td><td>3m 13s<\/td><\/tr><tr><td>VACE-Base<\/td><td>1\u00d7 GPU (Megatron)<\/td><td><strong>24.8 GB<\/strong><\/td><td><strong>2m 19s<\/strong><\/td><\/tr><tr><td>   VACE-TP<\/td><td>2\u00d7 GPUs (Megatron)<\/td><td><strong>31.5 GB<\/strong><\/td><td><strong>1m 41s<\/strong><\/td><\/tr><tr><td>   VACE-CP<\/td><td>2\u00d7 GPUs (Megatron)<\/td><td><strong>34.4 GB<\/strong><\/td><td><strong>1m 19s<\/strong><\/td><\/tr><tr><td rowspan=\"4\">  Training<\/td><td>VACE-Base<\/td><td>1\u00d7 GPU (Megatron)<\/td><td>28.31 GB<\/td><td>56.68s<\/td><\/tr><tr><td>   VACE-Base<\/td><td>2\u00d7 GPUs (Megatron)<\/td><td><strong>23.95 GB<\/strong><\/td><td><strong>27.15s<\/strong><\/td><\/tr><tr><td>   VACE-TP<\/td><td>2\u00d7 GPUs (Megatron)<\/td><td><strong>18.15 GB<\/strong><\/td><td><strong>55.9s<\/strong><\/td><\/tr><tr><td>   VACE-CP<\/td><td>2\u00d7 GPUs (Megatron)<\/td><td><strong>18.26 GB<\/strong><\/td><td><strong>56.02s<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The results demonstrate that converting VACE to Megatron Core yields <strong>substantial efficiency gains in both inference and training<\/strong>. For inference on a single GPU, the Megatron implementation reduces peak memory usage (28 GB \u2192 24.8 GB) while also improving runtime by over <strong>1.4\u00d7<\/strong> compared to the Hugging Face baseline. Scaling to two GPUs further highlights the benefits of distributed execution: tensor parallelism (TP) and context parallelism (CP) significantly reduce end-to-end inference time, with CP achieving the fastest runtime (1m 19s) by effectively sharding long video sequences across GPUs, albeit with slightly higher per-GPU memory usage.<\/p>\n\n\n\n<p>During training, Megatron enables efficient multi-GPU scaling even when fine-tuning only the context-adapter layers. Moving from one to two GPUs nearly halves the runtime while reducing per-GPU memory consumption, demonstrating effective data and model parallelism. TP and CP configurations further lower memory usage by distributing activations and sequence context, trading off longer runtimes due to increased communication overhead. Overall, these results show that Megatron Core provides a flexible set of parallelism strategies that allow VACE to balance <strong>memory efficiency, throughput, and scalability<\/strong> across different training and inference regimes.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n<ol class=\"wp-block-footnotes\"><li id=\"80930205-a5bf-4ca5-b4af-6dfb06f97ddf\">OpenSoraPlan A dataset for robot learning at scale, 2023. https:\/\/huggingface.co\/datasets\/LanguageBind\/Open-Sora-Plan-v1.0.0 <a href=\"#80930205-a5bf-4ca5-b4af-6dfb06f97ddf-link\" aria-label=\"Jump to footnote reference 1\">\u21a9\ufe0e<\/a><\/li><\/ol>","protected":false},"excerpt":{"rendered":"<p>Dataset The primary dataset we use for training is Mixkit from Open-Sora-Plan1 dataset. It is a high-quality open-source video dataset&nbsp;, created as part of open-sora-plan project to recreate OpenAI&#8217;s Sora model. It consists of 1234 videos, the total duration is about&nbsp;6h 19m 32s, and the total number of frames is&nbsp;570815. Input data consists of sequences &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Experiments&#8221;<\/span><\/a><\/p>\n","protected":false},"author":254,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":"[{\"id\":\"80930205-a5bf-4ca5-b4af-6dfb06f97ddf\",\"content\":\"OpenSoraPlan A dataset for robot learning at scale, 2023. https:\\\/\\\/huggingface.co\\\/datasets\\\/LanguageBind\\\/Open-Sora-Plan-v1.0.0\"}]"},"class_list":["post-38","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Experiments - Scalable Video Creating and Editing<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Experiments - Scalable Video Creating and Editing\" \/>\n<meta property=\"og:description\" content=\"Dataset The primary dataset we use for training is Mixkit from Open-Sora-Plan1 dataset. It is a high-quality open-source video dataset&nbsp;, created as part of open-sora-plan project to recreate OpenAI&#8217;s Sora model. It consists of 1234 videos, the total duration is about&nbsp;6h 19m 32s, and the total number of frames is&nbsp;570815. Input data consists of sequences &hellip; Continue reading &quot;Experiments&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/\" \/>\n<meta property=\"og:site_name\" content=\"Scalable Video Creating and Editing\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-13T23:44:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif\" \/>\n\t<meta property=\"og:image:width\" content=\"384\" \/>\n\t<meta property=\"og:image:height\" content=\"216\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/gif\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/\",\"name\":\"Experiments - Scalable Video Creating and Editing\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/wp-content\\\/uploads\\\/sites\\\/135\\\/2025\\\/12\\\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif\",\"datePublished\":\"2025-05-08T17:06:19+00:00\",\"dateModified\":\"2025-12-13T23:44:53+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/wp-content\\\/uploads\\\/sites\\\/135\\\/2025\\\/12\\\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/wp-content\\\/uploads\\\/sites\\\/135\\\/2025\\\/12\\\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif\",\"width\":384,\"height\":216},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/experiments\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Experiments\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/\",\"name\":\"Scalable Video Creating and Editing\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team2-1\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Experiments - Scalable Video Creating and Editing","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/","og_locale":"en_US","og_type":"article","og_title":"Experiments - Scalable Video Creating and Editing","og_description":"Dataset The primary dataset we use for training is Mixkit from Open-Sora-Plan1 dataset. It is a high-quality open-source video dataset&nbsp;, created as part of open-sora-plan project to recreate OpenAI&#8217;s Sora model. It consists of 1234 videos, the total duration is about&nbsp;6h 19m 32s, and the total number of frames is&nbsp;570815. Input data consists of sequences &hellip; Continue reading \"Experiments\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/","og_site_name":"Scalable Video Creating and Editing","article_modified_time":"2025-12-13T23:44:53+00:00","og_image":[{"width":384,"height":216,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif","type":"image\/gif"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/","name":"Experiments - Scalable Video Creating and Editing","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif","datePublished":"2025-05-08T17:06:19+00:00","dateModified":"2025-12-13T23:44:53+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-content\/uploads\/sites\/135\/2025\/12\/mixkit-man-traveling-through-the-desert-on-his-motorcycle-43114-ezgif.com-cut.gif","width":384,"height":216},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/experiments\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/"},{"@type":"ListItem","position":2,"name":"Experiments"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/","name":"Scalable Video Creating and Editing","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/pages\/38","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/users\/254"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/comments?post=38"}],"version-history":[{"count":43,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/pages\/38\/revisions"}],"predecessor-version":[{"id":332,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/pages\/38\/revisions\/332"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team2-1\/wp-json\/wp\/v2\/media?parent=38"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}