{"id":3,"date":"2025-05-09T20:59:44","date_gmt":"2025-05-09T20:59:44","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/?p=3"},"modified":"2025-05-10T00:41:02","modified_gmt":"2025-05-10T00:41:02","slug":"temporally-hierarchical-scene-graph-generation-for-video-question-answering","status":"publish","type":"post","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/2025\/05\/09\/temporally-hierarchical-scene-graph-generation-for-video-question-answering\/","title":{"rendered":"Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b"},"content":{"rendered":"\n

2025 MSCV Capstone project of Daniel Yang, Tianzhi Li
Supervised by Katia Sycara<\/a>, Yaqi Xie<\/p>\n\n\n\n

Motivation<\/h2>\n\n\n\n

Video Question Answering (VideoQA)<\/strong> requires reasoning over actions that occur at drastically different timescales \u2014 from short, atomic gestures like picking up an apple to long-term goals like shopping in the supermarket. Traditional vision-language models struggle to capture such temporal diversity, particularly in long videos where long, overarching intentions and fine-grained actions coexist.
We propose a temporally hierarchical scene graph<\/strong> representation to address this. Scene graphs encode video content as compact <Subject – Relation – Object> triplets, enabling structured and explainable reasoning. By building scene graphs at multiple temporal resolutions \u2014 from frame-level interactions to high-level goals \u2014 we can preserve both fine-grained actions and overarching intentions.
This approach offers a scalable, interpretable, and resource-efficient method for long-form video understanding, making VideoQA more tractable across diverse tasks and durations.<\/p>\n\n\n\n

\"\"
An example of an image scene graph that captures the semantics of a scene.<\/figcaption><\/figure>\n\n\n\n

Datasets<\/h2>\n\n\n\n

EgoSchema<\/a><\/p>\n\n\n\n

The EgoSchema benchmark contains over 5000 very long-form video language understanding questions spanning over 250 hours of real, diverse, and high-quality egocentric video data. Many videos feature complex scenes with cluttered rooms, where we could take full potential of scene graphs.<\/p>\n\n\n\n

\"\"
EgoSchema dataset QA example<\/figcaption><\/figure>\n\n\n\n

NExT-QA<\/a><\/p>\n\n\n\n

NExT-QA is a VideoQA benchmark targeting the explanation of video content. It challenges QA models to reason about the causal and temporal actions and understand the rich object interactions in daily activities. NExT-QA contains 5,440 videos and about 52K manually annotated question-answer pairs, grouped into causal, temporal, and descriptive questions. The videos are about 35 seconds long and focus on interactions between people and sometimes pets. <\/p>\n\n\n\n

\"\"
NExT-QA examples and question category statistics<\/figcaption><\/figure>\n\n\n\n

Proposed Method<\/h2>\n\n\n\n

We plan to simultaneously generate a hierarchical structure of scene graphs and utilize them to tackle VideoQA tasks. <\/p>\n\n\n\n

Since we don’t have access to higher-level scene graphs beyond frame-by-frame image scene graphs, we have to construct the higher-level scene graphs in a self-supervised manner.<\/p>\n\n\n\n

\"\"
The key source of supervision for scene graph encoding: Variational AutoEncoder (VAE)<\/figcaption><\/figure>\n\n\n\n

We outlined the steps to generate high-level scene graphs that express short clips from image-level scene graphs.<\/p>\n\n\n\n

\"\"
Architecture overview of our proposed temporally hierarchical scene graph generator.<\/figcaption><\/figure>\n\n\n\n