{"id":38,"date":"2026-05-07T00:00:34","date_gmt":"2026-05-07T00:00:34","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/?page_id=38"},"modified":"2026-05-08T04:02:00","modified_gmt":"2026-05-08T04:02:00","slug":"2d-vlms-for-3d-vqa","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/","title":{"rendered":"2D VLMs for 3D VQA"},"content":{"rendered":"\n<h2 class=\"wp-block-heading has-x-large-font-size\"><strong>Motivation<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Humans effortlessly infer rich 3D structure<\/strong> from sparse 2D observations, yet replicating this in machines remains an open challenge.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>While 3D scene understanding is essential for embodied AI, <strong>3D-language datasets remain orders of magnitude smaller<\/strong> than 2D counterparts.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>This leaves models data-starved and prone to <strong>poor generalization<\/strong>, while powerful <strong>2D vision-language models<\/strong> go <strong>underutilized for 3D tasks<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading has-x-large-font-size\"><strong>Problem Statement<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We exploit this asymmetry by leveraging <strong>frozen 2D VLMs<\/strong> for <strong>3D Visual Question Answering<\/strong>, bypassing the need for 3D-specific training entirely.&nbsp;<\/li>\n\n\n\n<li>Through <strong>principled view selection<\/strong>, we project the 3D scene into a small set of informative 2D frames and query the frozen model directly,&nbsp;<\/li>\n\n\n\n<li>Our aim is to achieve <strong>competitive zero-shot performance<\/strong> on ScanQA without any parameter updates.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading has-x-large-font-size\"><strong>Dataset<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>ScanNet<\/strong> (CVPR 2017) : RGB-D dataset of 1,513 indoor scenes with 2.5M views, annotated with 3D poses, reconstructions, and semantic segmentations.<\/li>\n\n\n\n<li><strong>ScanQA<\/strong> (CVPR 2022) : 41K free-form QA pairs grounded to 3D objects across 800 ScanNet scenes, requiring spatial scene reasoning.<\/li>\n\n\n\n<li><strong>SQA3D<\/strong> (ICLR 2023) :&nbsp; 33.4K situated reasoning questions across 650 scenes, where agents must understand their own position and orientation before answering.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Prior Work 1: cd Views<\/strong><\/h2>\n\n\n\n<p>Key Idea: Use frozen VLMs for 3D Question Answering<\/p>\n\n\n\n<p><strong>The 3D Gap:<\/strong> 3D-language datasets are significantly smaller than 2D datasets. <\/p>\n\n\n\n<p><strong>Training Bottleneck:<\/strong> Training 3D models from scratch leads to poor performance due to this scarcity.<\/p>\n\n\n\n<p><strong>Problem Setting<\/strong><\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"750\" height=\"660\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png\" alt=\"\" class=\"wp-image-12\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png 750w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2-300x264.png 300w\" sizes=\"auto, (max-width: 750px) 100vw, 750px\" \/><\/figure>\n<\/div>\n\n\n\n<p>Question: \u201cWhich direction should I go if I want to open the window?\u201d<br><br>Situation: \u201cI am sitting in an armchair with a lamp behind me.\u201d<\/p>\n\n\n\n<p>Answer: &#8220;Left&#8221;<\/p>\n\n\n\n<p>The frozen VLM should now take these multi-view images as input and answer the question as per the given situation.<\/p>\n\n\n\n<p><strong>cdViews (critical and diverse view selection framework ICML 2025)<\/strong><\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"549\" data-id=\"14\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.20.59-PM-1024x549.png\" alt=\"\" class=\"wp-image-14\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.20.59-PM-1024x549.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.20.59-PM-300x161.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.20.59-PM-768x411.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.20.59-PM-1536x823.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.20.59-PM.png 1598w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/figure>\n<\/div>\n\n\n\n<p>Views Selected by viewSelector module have lots of redundancy, this redundancy is eliminated through the viewNMS module, which suppresses nearby views using camera parameters. <\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-2 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"534\" data-id=\"15\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.18-PM-1024x534.png\" alt=\"\" class=\"wp-image-15\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.18-PM-1024x534.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.18-PM-300x156.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.18-PM-768x401.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.18-PM-1536x801.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.18-PM.png 1756w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-3 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"542\" data-id=\"16\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.23-PM-1024x542.png\" alt=\"\" class=\"wp-image-16\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.23-PM-1024x542.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.23-PM-300x159.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.23-PM-768x406.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.23-PM-1536x813.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.23.23-PM.png 1742w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/figure>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading has-x-large-font-size\"><strong>Prior Work 2: SplatTalk<\/strong><\/h2>\n\n\n\n<p>A framework that bridges the gap between 3D scene representation and Large Language Models (LLMs) for 3D Visual Question Answering (VQA):<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>It uses 3D Gaussian Splatting in a self-supervised way.&nbsp;<\/li>\n\n\n\n<li>It learns what objects are by looking at 2D images and &#8220;lifting&#8221; that knowledge into a 3D field.&nbsp;<\/li>\n\n\n\n<li>It is feed-forward, meaning it works on new rooms instantly.<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading has-large-font-size\"><strong>FreeSplat: Foundational Framework<\/strong><\/h4>\n\n\n\n<p>FreeSplat provides the backbone for constructing a high-quality 3D representation from sparse inputs. Starting from a sequence of images, it builds cost volumes between nearby views to estimate correspondences, and from there predicts depth maps and feature maps.<\/p>\n\n\n\n<p>These are then unprojected into 3D space as Gaussian triplets, giving us a structured representation of the scene. The key innovation in FreeSplat is the Pixel-aligned Triplet Fusion module, which progressively fuses information across views to create a consistent global 3D representation.<\/p>\n\n\n\n<p>So in short, FreeSplat gives us a clean, efficient, and real-time capable 3D Gaussian scene representation.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"460\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1-1024x460.png\" alt=\"\" class=\"wp-image-117\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1-1024x460.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1-300x135.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1-768x345.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1-1536x689.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading has-large-font-size\"><strong>SplatTalk: 3D VQA with Gaussian Splatting<\/strong><\/h4>\n\n\n\n<p>Building on top of this, SplatTalk extends the idea from geometry to language.<\/p>\n\n\n\n<p>Instead of just encoding spatial information, SplatTalk integrates visual-language features into the same 3D Gaussian representation. This means each part of the scene doesn\u2019t just know where it is, but also what it represents.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"305\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-2-1024x305.png\" alt=\"\" class=\"wp-image-122\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-2-1024x305.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-2-300x89.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-2-768x229.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-2-1536x457.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-2.png 1666w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Our Approach and Results<\/strong><\/h2>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. 3D Scene Object Maps<\/strong><\/h4>\n\n\n\n<p>Given a set of posed RGB-D images, our goal is to construct a semantic object map that localizes every object instance in 3D world coordinates.&nbsp;<br><\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"764\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1024x764.png\" alt=\"\" class=\"wp-image-17\" title=\"Best-Summary.png\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-1024x764.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-300x224.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-768x573.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image.png 1056w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>Method<\/td><td><strong>ScanQA (EM@1)<\/strong><\/td><td><strong>SQA (EM@1)<\/strong><\/td><\/tr><tr><td>BridgeQA<\/td><td>26.98<\/td><td>52.9<\/td><\/tr><tr><td>cdViews<\/td><td>30.1<\/td><td><strong>53.1<\/strong><\/td><\/tr><tr><td>ObjectMap3D (Ours)<\/td><td><strong>30.3<\/strong><\/td><td>51.2<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Leveraging Foundational Novel View Synthesis<\/strong><\/h4>\n\n\n\n<p>We aim to synthesize a target view as per the situation and the question, taking into considering the available views. For this, we leverage <strong>Qwen Camera Control<\/strong> that allow NVS via natural language prompts, clarifying spatial ambiguities a VLM might miss. The new synthesized images clearly reveal that the \u201cdesk\u201d is behind the chair.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\">\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-4 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"298\" data-id=\"18\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.40.04-PM-1024x298.png\" alt=\"\" class=\"wp-image-18\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.40.04-PM-1024x298.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.40.04-PM-300x87.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.40.04-PM-768x223.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.40.04-PM-1536x447.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.40.04-PM.png 1602w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/figure>\n<\/div>\n\n\n\n<p>Synthesizing target views surpasses key works on SQA3D but it is limited by 1) frequent hallucinations 2) input view selection (which single view to input ?)<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"352\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.43.56-PM-1024x352.png\" alt=\"\" class=\"wp-image-19\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.43.56-PM-1024x352.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.43.56-PM-300x103.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.43.56-PM-768x264.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.43.56-PM-1536x528.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.43.56-PM.png 1774w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Leveraging Plucker Raymaps<\/strong><\/h4>\n\n\n\n<p>So the core problem we&#8217;re tackling is that current 2D VLMs don&#8217;t really understand camera geometry: they see multiple images but have no sense of how those images relate to each other in 3D space. To fix this, we introduce Pl\u00fccker raymaps. The pipeline first uses an LLM to extract the two objects from the question: the reference and the target. It then detects each object&#8217;s 2D center in its image, back-projects it into a 3D ray using the camera intrinsics, and computes a Pl\u00fccker line for each: a direction vector and a moment. Finally, the spatial relation: distance and angle; is derived from those two lines. This gives the VLM explicit geometric grounding that it otherwise completely lacks.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"752\" height=\"768\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-8.png\" alt=\"\" class=\"wp-image-129\" style=\"aspect-ratio:0.9791735850846814;width:341px;height:auto\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-8.png 752w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-8-294x300.png 294w\" sizes=\"auto, (max-width: 752px) 100vw, 752px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"524\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-5-1024x524.png\" alt=\"\" class=\"wp-image-125\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-5-1024x524.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-5-300x153.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-5-768x393.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-5-1536x785.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-5.png 1932w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4.<\/strong> <strong>Situation-aware 3D VQA: Plan Forward<\/strong><\/h4>\n\n\n\n<p>Our plan forward is to make the localization step more robust before we ever compute Pl\u00fccker rays. <\/p>\n\n\n\n<p>We use SQA3D&#8217;s ground-truth agent positions to supervise a lightweight position prediction model:  it takes a Bird&#8217;s-Eye View map of the scene plus a text situation description, and predicts where the agent is. Once we have that predicted location, we anchor a wide field-of-view camera there and render four egocentric views at 0, 90, 180, and 270 degrees: full panoramic coverage. Those four views go directly into a frozen 2D VLM to answer the question. No 3D fine-tuning, no heavy reconstruction. The key insight is that by grounding the agent&#8217;s position first, we ensure the views fed to the VLM are spatially correct: directly addressing the failure mode we just saw.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"399\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-7-1024x399.png\" alt=\"\" class=\"wp-image-128\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-7-1024x399.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-7-300x117.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-7-768x299.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-7-1536x599.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/image-7.png 1554w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><a data-type=\"page\" data-id=\"121\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/team-members\/\">Team Members<\/a><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Motivation Problem Statement Dataset Prior Work 1: cd Views Key Idea: Use frozen VLMs for 3D Question Answering The 3D Gap: 3D-language datasets are significantly smaller than 2D datasets. Training Bottleneck: Training 3D models from scratch leads to poor performance due to this scarcity. Problem Setting Question: \u201cWhich direction should I go if I want [&hellip;]<\/p>\n","protected":false},"author":303,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-38","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>2D VLMs for 3D VQA - 3D Scene Understanding<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"2D VLMs for 3D VQA - 3D Scene Understanding\" \/>\n<meta property=\"og:description\" content=\"Motivation Problem Statement Dataset Prior Work 1: cd Views Key Idea: Use frozen VLMs for 3D Question Answering The 3D Gap: 3D-language datasets are significantly smaller than 2D datasets. Training Bottleneck: Training 3D models from scratch leads to poor performance due to this scarcity. Problem Setting Question: \u201cWhich direction should I go if I want [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/\" \/>\n<meta property=\"og:site_name\" content=\"3D Scene Understanding\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-08T04:02:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png\" \/>\n\t<meta property=\"og:image:width\" content=\"750\" \/>\n\t<meta property=\"og:image:height\" content=\"660\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/\",\"name\":\"2D VLMs for 3D VQA - 3D Scene Understanding\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/wp-content\\\/uploads\\\/sites\\\/159\\\/2026\\\/05\\\/Screenshot-2026-05-05-at-6.12.48-PM-2.png\",\"datePublished\":\"2026-05-07T00:00:34+00:00\",\"dateModified\":\"2026-05-08T04:02:00+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/wp-content\\\/uploads\\\/sites\\\/159\\\/2026\\\/05\\\/Screenshot-2026-05-05-at-6.12.48-PM-2.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/wp-content\\\/uploads\\\/sites\\\/159\\\/2026\\\/05\\\/Screenshot-2026-05-05-at-6.12.48-PM-2.png\",\"width\":750,\"height\":660},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/2d-vlms-for-3d-vqa\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"2D VLMs for 3D VQA\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/\",\"name\":\"3D Scene Understanding\",\"description\":\"Resources\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf17\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"2D VLMs for 3D VQA - 3D Scene Understanding","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/","og_locale":"en_US","og_type":"article","og_title":"2D VLMs for 3D VQA - 3D Scene Understanding","og_description":"Motivation Problem Statement Dataset Prior Work 1: cd Views Key Idea: Use frozen VLMs for 3D Question Answering The 3D Gap: 3D-language datasets are significantly smaller than 2D datasets. Training Bottleneck: Training 3D models from scratch leads to poor performance due to this scarcity. Problem Setting Question: \u201cWhich direction should I go if I want [&hellip;]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/","og_site_name":"3D Scene Understanding","article_modified_time":"2026-05-08T04:02:00+00:00","og_image":[{"width":750,"height":660,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/","name":"2D VLMs for 3D VQA - 3D Scene Understanding","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png","datePublished":"2026-05-07T00:00:34+00:00","dateModified":"2026-05-08T04:02:00+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-content\/uploads\/sites\/159\/2026\/05\/Screenshot-2026-05-05-at-6.12.48-PM-2.png","width":750,"height":660},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/2d-vlms-for-3d-vqa\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/"},{"@type":"ListItem","position":2,"name":"2D VLMs for 3D VQA"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/","name":"3D Scene Understanding","description":"Resources","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/pages\/38","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/users\/303"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/comments?post=38"}],"version-history":[{"count":6,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/pages\/38\/revisions"}],"predecessor-version":[{"id":240,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/pages\/38\/revisions\/240"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf17\/wp-json\/wp\/v2\/media?parent=38"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}