Motivation
- Humans effortlessly infer rich 3D structure from sparse 2D observations, yet replicating this in machines remains an open challenge.
- While 3D scene understanding is essential for embodied AI, 3D-language datasets remain orders of magnitude smaller than 2D counterparts.
- This leaves models data-starved and prone to poor generalization, while powerful 2D vision-language models go underutilized for 3D tasks.
Problem Statement
- We exploit this asymmetry by leveraging frozen 2D VLMs for 3D Visual Question Answering, bypassing the need for 3D-specific training entirely.
- Through principled view selection, we project the 3D scene into a small set of informative 2D frames and query the frozen model directly,
- Our aim is to achieve competitive zero-shot performance on ScanQA without any parameter updates.
Dataset
- ScanNet (CVPR 2017) : RGB-D dataset of 1,513 indoor scenes with 2.5M views, annotated with 3D poses, reconstructions, and semantic segmentations.
- ScanQA (CVPR 2022) : 41K free-form QA pairs grounded to 3D objects across 800 ScanNet scenes, requiring spatial scene reasoning.
- SQA3D (ICLR 2023) : 33.4K situated reasoning questions across 650 scenes, where agents must understand their own position and orientation before answering.
Prior Work 1: cd Views
Key Idea: Use frozen VLMs for 3D Question Answering
The 3D Gap: 3D-language datasets are significantly smaller than 2D datasets.
Training Bottleneck: Training 3D models from scratch leads to poor performance due to this scarcity.
Problem Setting
Question: “Which direction should I go if I want to open the window?”
Situation: “I am sitting in an armchair with a lamp behind me.”
Answer: “Left”
The frozen VLM should now take these multi-view images as input and answer the question as per the given situation.
cdViews (critical and diverse view selection framework ICML 2025)
Views Selected by viewSelector module have lots of redundancy, this redundancy is eliminated through the viewNMS module, which suppresses nearby views using camera parameters.
Prior Work 2: SplatTalk
A framework that bridges the gap between 3D scene representation and Large Language Models (LLMs) for 3D Visual Question Answering (VQA):
- It uses 3D Gaussian Splatting in a self-supervised way.
- It learns what objects are by looking at 2D images and “lifting” that knowledge into a 3D field.
- It is feed-forward, meaning it works on new rooms instantly.
FreeSplat: Foundational Framework
FreeSplat provides the backbone for constructing a high-quality 3D representation from sparse inputs. Starting from a sequence of images, it builds cost volumes between nearby views to estimate correspondences, and from there predicts depth maps and feature maps.
These are then unprojected into 3D space as Gaussian triplets, giving us a structured representation of the scene. The key innovation in FreeSplat is the Pixel-aligned Triplet Fusion module, which progressively fuses information across views to create a consistent global 3D representation.
So in short, FreeSplat gives us a clean, efficient, and real-time capable 3D Gaussian scene representation.
SplatTalk: 3D VQA with Gaussian Splatting
Building on top of this, SplatTalk extends the idea from geometry to language.
Instead of just encoding spatial information, SplatTalk integrates visual-language features into the same 3D Gaussian representation. This means each part of the scene doesn’t just know where it is, but also what it represents.
Our Approach and Results
1. 3D Scene Object Maps
Given a set of posed RGB-D images, our goal is to construct a semantic object map that localizes every object instance in 3D world coordinates.
| Method | ScanQA (EM@1) | SQA (EM@1) |
| BridgeQA | 26.98 | 52.9 |
| cdViews | 30.1 | 53.1 |
| ObjectMap3D (Ours) | 30.3 | 51.2 |
2. Leveraging Foundational Novel View Synthesis
We aim to synthesize a target view as per the situation and the question, taking into considering the available views. For this, we leverage Qwen Camera Control that allow NVS via natural language prompts, clarifying spatial ambiguities a VLM might miss. The new synthesized images clearly reveal that the “desk” is behind the chair.
Synthesizing target views surpasses key works on SQA3D but it is limited by 1) frequent hallucinations 2) input view selection (which single view to input ?)
3. Leveraging Plucker Raymaps
So the core problem we’re tackling is that current 2D VLMs don’t really understand camera geometry: they see multiple images but have no sense of how those images relate to each other in 3D space. To fix this, we introduce Plücker raymaps. The pipeline first uses an LLM to extract the two objects from the question: the reference and the target. It then detects each object’s 2D center in its image, back-projects it into a 3D ray using the camera intrinsics, and computes a Plücker line for each: a direction vector and a moment. Finally, the spatial relation: distance and angle; is derived from those two lines. This gives the VLM explicit geometric grounding that it otherwise completely lacks.
4. Situation-aware 3D VQA: Plan Forward
Our plan forward is to make the localization step more robust before we ever compute Plücker rays.
We use SQA3D’s ground-truth agent positions to supervise a lightweight position prediction model: it takes a Bird’s-Eye View map of the scene plus a text situation description, and predicts where the agent is. Once we have that predicted location, we anchor a wide field-of-view camera there and render four egocentric views at 0, 90, 180, and 270 degrees: full panoramic coverage. Those four views go directly into a frozen 2D VLM to answer the question. No 3D fine-tuning, no heavy reconstruction. The key insight is that by grounding the agent’s position first, we ensure the views fed to the VLM are spatially correct: directly addressing the failure mode we just saw.
