{"id":21,"date":"2025-05-09T21:25:52","date_gmt":"2025-05-09T21:25:52","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/?page_id=21"},"modified":"2025-12-13T01:23:51","modified_gmt":"2025-12-13T01:23:51","slug":"proposed-method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/","title":{"rendered":"Method"},"content":{"rendered":"\n<p>Our method comprises two tightly coupled components: a video indexing stage that incrementally constructs temporally evolving scene graphs from video frames, and a video question answering (VQA) stage that performs graph-based retrieval and reasoning over the resulting indexed scene graphs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. Video Indexing: Frame-Level Scene Graphs with Temporal Deltas<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Scene Graph Definition<\/h3>\n\n\n\n<p>We adopt a scene graph representation tailored for temporal indexing.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Entity format: (entity_name | entity_type | description)<\/li>\n\n\n\n<li>Relationship format: (source | target | description)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Video Indexing Pipeline<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"661\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing-1024x661.png\" alt=\"\" class=\"wp-image-80\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing-1024x661.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing-300x194.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing-768x496.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing.png 1354w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">Overview of our video indexing pipeline. For each frame, the LLM takes the current image together with the previous frame\u2019s scene graph as input and predicts the scene graph differences. These differences are then accumulated with the previous scene graph to produce the current frame\u2019s scene graph.<\/figcaption><\/figure>\n\n\n\n<p><strong>Frame-wise graph generation<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sample video frames at 1 FPS.<\/li>\n\n\n\n<li>Use GPT-4o-mini to extract scene graph information for each frame.<\/li>\n\n\n\n<li>Condition extraction on both the current frame image and the previous frame\u2019s scene graph.<\/li>\n<\/ul>\n\n\n\n<p><strong>Temporal delta extraction<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Output only differences from the previous frame:\n<ul class=\"wp-block-list\">\n<li>newly appeared or disappeared entities and relationships<\/li>\n\n\n\n<li>updated descriptions of existing entities or relationships<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Avoid redundant storage by capturing only meaningful changes.<\/li>\n<\/ul>\n\n\n\n<p><strong>Scene graph accumulation<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Merge predicted deltas with the previous frame\u2019s scene graph.<\/li>\n\n\n\n<li>Produce the current frame\u2019s complete scene graph while maintaining temporal continuity.<\/li>\n<\/ul>\n\n\n\n<p><strong>Background object handling<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>List visible objects whose states remain unchanged as background objects.<\/li>\n\n\n\n<li>Include only object names without descriptions or relationships to reduce redundancy.<\/li>\n<\/ul>\n\n\n\n<p><strong>Prompt design for video indexing<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>List all other visible but inactive objects as background.<\/li>\n\n\n\n<li>Instruct the model to output only entities and relationships whose presence or description differs from the previous frame.<\/li>\n\n\n\n<li>Include concise evidence from the current frame for changed elements.<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"392\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/prompt-1024x392.png\" alt=\"\" class=\"wp-image-82\" style=\"width:468px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/prompt-1024x392.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/prompt-300x115.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/prompt-768x294.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/prompt.png 1030w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">The prompt used in Video Indexing.<\/figcaption><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\">2. Video Question Answering: GraphRAG-Inspired Community Summaries and DRIFT Search<\/h2>\n\n\n\n<p><strong>Temporal community construction<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cluster frame-level scene graphs into temporal communities to support long-range reasoning.<\/li>\n\n\n\n<li>Apply change-point detection with <em>ruptures<\/em> on node-level temporal sequences.<\/li>\n\n\n\n<li>Identify shifts in entity activity or graph structure.<\/li>\n\n\n\n<li>Use detected change points to segment the video into coherent temporal intervals.<\/li>\n<\/ul>\n\n\n\n<p><strong>Multi-scale community summaries<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Generate hierarchical summaries for each temporal community inspired by GraphRAG.<\/li>\n\n\n\n<li>Capture salient entities and key interactions within each community.<\/li>\n\n\n\n<li>Preserve higher-level contextual information across longer temporal spans.<\/li>\n\n\n\n<li>Enable efficient retrieval at multiple levels of abstraction.<\/li>\n<\/ul>\n\n\n\n<p><strong>Question answering with DRIFT search<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Match the input question against community-level summaries.<\/li>\n\n\n\n<li>Retrieve the top-<em>K<\/em> most relevant temporal communities.<\/li>\n\n\n\n<li>Generate an initial answer and follow-up queries conditioned on retrieved summaries.<\/li>\n\n\n\n<li>Perform localized search within corresponding frame-level scene graphs and temporal deltas.<\/li>\n\n\n\n<li>Synthesize the final answer from evidence gathered across multiple temporal scales.<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"550\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/graphrag-1-1024x550.png\" alt=\"\" class=\"wp-image-76\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/graphrag-1-1024x550.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/graphrag-1-300x161.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/graphrag-1-768x413.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/graphrag-1.png 1180w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">GraphRAG organizes graphs without explicit temporal structure, relying on top-down community segmentation and bottom-up summaries<\/figcaption><\/figure>\n<\/div>\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"534\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/thsg-1024x534.png\" alt=\"\" class=\"wp-image-77\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/thsg-1024x534.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/thsg-300x156.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/thsg-768x401.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/thsg.png 1204w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><figcaption class=\"wp-element-caption\">THSG explicitly models temporal structure, performs bottom-up segmentation based on temporal changes, and generates bottom-up temporal summaries that reflect the evolution of scene graphs over time.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Our method comprises two tightly coupled components: a video indexing stage that incrementally constructs temporally evolving scene graphs from video frames, and a video question answering (VQA) stage that performs graph-based retrieval and reasoning over the resulting indexed scene graphs. 1. Video Indexing: Frame-Level Scene Graphs with Temporal Deltas Scene Graph Definition We adopt a &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Method&#8221;<\/span><\/a><\/p>\n","protected":false},"author":239,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-21","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Method - Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b\" \/>\n<meta property=\"og:description\" content=\"Our method comprises two tightly coupled components: a video indexing stage that incrementally constructs temporally evolving scene graphs from video frames, and a video question answering (VQA) stage that performs graph-based retrieval and reasoning over the resulting indexed scene graphs. 1. Video Indexing: Frame-Level Scene Graphs with Temporal Deltas Scene Graph Definition We adopt a &hellip; Continue reading &quot;Method&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/\" \/>\n<meta property=\"og:site_name\" content=\"Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-13T01:23:51+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1354\" \/>\n\t<meta property=\"og:image:height\" content=\"874\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/\",\"name\":\"Method - Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/wp-content\\\/uploads\\\/sites\\\/124\\\/2025\\\/12\\\/indexing-1024x661.png\",\"datePublished\":\"2025-05-09T21:25:52+00:00\",\"dateModified\":\"2025-12-13T01:23:51+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/wp-content\\\/uploads\\\/sites\\\/124\\\/2025\\\/12\\\/indexing.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/wp-content\\\/uploads\\\/sites\\\/124\\\/2025\\\/12\\\/indexing.png\",\"width\":1354,\"height\":874},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/proposed-method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/\",\"name\":\"Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team17\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b","og_description":"Our method comprises two tightly coupled components: a video indexing stage that incrementally constructs temporally evolving scene graphs from video frames, and a video question answering (VQA) stage that performs graph-based retrieval and reasoning over the resulting indexed scene graphs. 1. Video Indexing: Frame-Level Scene Graphs with Temporal Deltas Scene Graph Definition We adopt a &hellip; Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/","og_site_name":"Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b","article_modified_time":"2025-12-13T01:23:51+00:00","og_image":[{"width":1354,"height":874,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/","name":"Method - Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing-1024x661.png","datePublished":"2025-05-09T21:25:52+00:00","dateModified":"2025-12-13T01:23:51+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-content\/uploads\/sites\/124\/2025\/12\/indexing.png","width":1354,"height":874},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/proposed-method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/","name":"Temporally Hierarchical Scene Graph Generation for Video Question Answering\u200b","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/pages\/21","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/users\/239"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/comments?post=21"}],"version-history":[{"count":15,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/pages\/21\/revisions"}],"predecessor-version":[{"id":99,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/pages\/21\/revisions\/99"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team17\/wp-json\/wp\/v2\/media?parent=21"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}