{"id":8,"date":"2025-05-07T18:31:51","date_gmt":"2025-05-07T18:31:51","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team1\/?page_id=8"},"modified":"2025-12-12T17:34:53","modified_gmt":"2025-12-12T17:34:53","slug":"training-data","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team1\/training-data\/","title":{"rendered":"Control via Prompts"},"content":{"rendered":"\n

Data Availability<\/h2>\n\n\n\n

The first roadblock to determining the architecture for this pipeline was the available data. We need to consider 4 key modalities for this:

1. Natural Language Prompts
2. Camera Trajectories
3. RGB Video
4. 3D Scene<\/p>\n\n\n\n

Based on these considerations, we found 2 main datasets:<\/p>\n\n\n\n

The Exceptional Trajectories<\/strong> – A dataset of 115000 samples with natural language prompts and camera trajectories, along with pose data on human actors in the scene [1]. <\/p>\n\n\n\n

\"\"
Fig. 1 – Exceptional Trajectories Data Sample<\/figcaption><\/figure>\n\n\n\n

GenDoP<\/strong> – A better curated dataset of 27000 samples with natural language prompts, camera trajectories, and RGB video [2]. <\/p>\n\n\n\n

With both the datasets, there was no 3D scene data available, necessitating an approach that would separate the grounding and the text-to-trajectory aspects of the pipeline.<\/p>\n\n\n\n

System Architecture<\/h2>\n\n\n\n
\"\"
Fig. 2 – Control via Natural-Language Cinematographic Prompts Architecture Diagram<\/figcaption><\/figure>\n\n\n\n

The key observation behind the architecture is that the user’s prompt can be separated into a series of anchors and trajectories. Anchors <\/strong>are physical keypoints that the camera should focus on, such as actors or objects of interest. Trajectories<\/strong> are the discrete camera movements that need to be made to go from one anchor to another. <\/p>\n\n\n\n

Therefore, an LLM is used to decompose the prompt, as seen in Figure 2, and then smaller prompts for anchors and trajectories are passed into the Anchor Detector<\/strong> and Text-To-Trajectory Model<\/strong> respectively.<\/p>\n\n\n\n

These are then combined through affine transformations to generate the final, grounded output trajectory, which is then passed into FLAIR Classic to control the robot. A small set of post-processing is done on the trajectory to ensure that the robot can physically execute the trajectory.<\/p>\n\n\n\n

More details on the Anchor Detection and Text-to-Trajectory models can be seen on their respective pages.<\/strong><\/p>\n\n\n\n

Results<\/h2>\n\n\n\n

We can demonstrate this pipeline by showing results from an input prompt and scene.<\/p>\n\n\n\n

Input Prompt:<\/strong> Slow dolly left from the actor to the laptop, then to the water bottle.<\/p>\n\n\n\n

Input Scene: <\/strong>Figure 3 below shows the input scene, with Shaurye sitting on his chair and a laptop and water bottle on the table in front of him.<\/p>\n\n\n\n

\"\"
Fig. 3 – Input Scene<\/figcaption><\/figure>\n\n\n\n

The anchors are then detected, seen through the detected objects then a screenshot of the 3D anchor points on the pointcloud below. The points are in order red-green-blue, indicating the camera should start far away then move closer to the laptop and water bottle in succession.<\/p>\n\n\n\n

\n
\"\"
Fig. 4 – Object Detections<\/figcaption><\/figure>\n\n\n\n
\"\"

Fig. 5 – Anchor Points<\/figcaption><\/figure>\n<\/figure>\n\n\n\n

The next step is to generate trajectories from the text prompts. Here is a visualization of one of the trajectories:<\/p>\n\n\n\n