Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data

Method

3D scene representation

Given an RGB-D observation, a frozen ViT encoder extracts a sequence of N patch tokens $\{\mathbf{z}_i\}_{i=1}^N \in \mathbb{R}^d$ . Each patch center is unprojected through the depth map and camera intrinsics K to yield a 3D point $\mathbf{p}_i = (x_i, y_i, z_i)^\top$ in camera coordinates. These coordinates serve as positional inputs to the denoising transformer via 3D RoPE rather than being concatenated to the token features, preserving the geometry in the attention structure rather than the embedding space.

3D Rotary Position Embeddings

Standard 1D RoPE encodes position m into query/key features by rotating pairs of dimensions by angle $m\theta_j$ , where $\theta_j = b^{-2j/d}$ is a base-dependent frequency. We extend this to three independent spatial axes. For a token at position $(m_x, m_y, m_z)$ , the rotation matrix applied to a d-dimensional feature vector is:

$\mathcal{R}_{m_x,m_y,m_z} = \mathrm{blockdiag}\!\left(\mathcal{R}^x_{m_x},\, \mathcal{R}^y_{m_y},\, \mathcal{R}^z_{m_z}\right) \in SO(d)$

where each axis block $\mathcal{R}^\alpha_{m_\alpha} \in \mathbb{R}^{(d/3)\times(d/3)}$ is itself block diagonal over frequency pairs $j = 1, \ldots, d/6$ :

$\left[\mathcal{R}^\alpha_{m_\alpha}\right]^{(j)} = \begin{pmatrix} \cos m_\alpha \theta_j & -\sin m_\alpha \theta_j \\ \sin m_\alpha \theta_j & \cos m_\alpha \theta_j \end{pmatrix}$

The rotated query and key for token i are $\tilde{\mathbf{q}}_i = \mathcal{R}_{m_x^i, m_y^i, m_z^i}\, \mathbf{q}_i$ and analogously for $\tilde{\mathbf{k}}_i$ . The attention score between tokens i and j then depends only on their relative 3D displacement:

$\tilde{\mathbf{q}}_i^\top \tilde{\mathbf{k}}_j = \mathbf{q}_i^\top \mathcal{R}_{m_x^i - m_x^j,\, m_y^i - m_y^j,\, m_z^i - m_z^j}\, \mathbf{k}_j$

which encodes translation equivariance in 3D space directly into attention.

Flow matching objective

The policy models the conditional distribution over action trajectories $\tau = (a_t, \ldots, a_{t+H-1})$ via a learned vector field. Given a scene encoding $\mathbf{c}$ (3D scene tokens, proprioception, and language), we define a time-conditional flow from a noise distribution $p_0 = \mathcal{N}(0, I)$ to the data distribution $p_1$ via the ODE:

$\frac{d\tau^s}{ds} = v_\psi(\tau^s, s, \mathbf{c}), \quad s \in [0,1]$

The vector field $v_\psi$ is parameterized by the denoising transformer and trained with the conditional flow matching loss:

$\mathcal{L}_\mathrm{FM} = \mathbb{E}_{s,\,\tau^0,\,\tau^1}\!\left[\left\|v_\psi(\tau^s, s, \mathbf{c}) - (\tau^1 - \tau^0)\right\|^2\right]$

where $\tau^s = (1-s)\tau^0 + s\tau^1$ is the linear interpolant, $\tau^0 \sim \mathcal{N}(0,I)$ , and $\tau^1 \sim p_\mathrm{data}$ .

Learned calibration correction

Problem setup.

Let $R^* \in SO(3)$ denote the unknown camera-to-robot rotation for a given data source. Points reconstructed from the camera are expressed as $\mathbf{p}^\mathrm{cam} = R^{*\top} \mathbf{p}^\mathrm{robot}$ , so feeding $\mathbf{p}^\mathrm{cam}$ into 3D RoPE implicitly applies an erroneous coordinate frame. We seek to learn a correction that absorbs $R^*$ from data alone.

Camera token.

For a frame with N patch tokens $\{\mathbf{z}_i\}$ , we form a camera token by mean pooling:

$\mathbf{c}_\mathrm{cam} = \frac{1}{N}\sum_{i=1}^N \mathbf{z}_i \in \mathbb{R}^d$

Parameterization via the matrix exponential.

We require the correction $\delta M \in SO(d)$ to be a valid rotation matrix so that the modified RoPE $\delta M_\mathrm{full} \cdot \mathcal{R}$ remains an element of $SO(d)$ . We enforce this by construction: an MLP $f_\phi$ maps $\mathbf{c}_\mathrm{cam}$ to the entries of a skew-symmetric matrix $A \in \mathfrak{so}(6)$ , and $\delta M_6 = \exp(A)$ is obtained via the matrix exponential. Skew-symmetry $A^\top = -A$ guarantees:

$\delta M_6^\top \delta M_6 = \exp(A^\top)\exp(A) = \exp(-A)\exp(A) = I$

so $\delta M_6 \in SO(6)$ by construction, without any projection step or penalty term.

Lifting to full dimension.

The RoPE frequency vector for a token at position $(m_x, m_y, m_z)$ is a stacked 6-vector of cosines and sines across the three axes. To act on the full d-dimensional feature space, $\delta M_6$ is tiled as:

$\delta M_\mathrm{full} = \mathrm{blockdiag}(\underbrace{\delta M_6, \ldots, \delta M_6}_{d/6}) \in SO(d)$

Since $\delta M_\mathrm{full}$ is block diagonal with identical blocks, $\delta M_\mathrm{full}^\top \delta M_\mathrm{full} = I$ if and only if $\delta M_6^\top \delta M_6 = I$ . Orthogonality of the tile is therefore necessary and sufficient for orthogonality of the full matrix.

Modified RoPE.

The corrected position encoding for token i replaces the raw cosine-sine vector $\boldsymbol{\phi}_i = [\cos\theta^x_i,\, \cos\theta^y_i,\, \cos\theta^z_i,\, \sin\theta^x_i,\, \sin\theta^y_i,\, \sin\theta^z_i]^\top$ with:

$\tilde{\boldsymbol{\phi}}_i = \delta M_6\, \boldsymbol{\phi}_i$

The relative-position property of RoPE is preserved under this transformation: for any two tokens $i, j$ ,

$\tilde{\boldsymbol{\phi}}_i^\top \tilde{\boldsymbol{\phi}}_j = \boldsymbol{\phi}_i^\top \delta M_6^\top \delta M_6\, \boldsymbol{\phi}_j = \boldsymbol{\phi}_i^\top \boldsymbol{\phi}_j$

so the correction acts as a global re-orientation of the coordinate frame rather than a distortion of pairwise geometry. The entire model including $f_\phi$ is trained end-to-end with $\mathcal{L}_\mathrm{FM}$ , with no auxiliary supervision on $\delta M$ .

Extension to GR00T N1.5

GR00T N1.5 follows a DiT-based architecture with alternating cross-attention (vision-to-action) and self-attention (action) blocks. We replace the 2D sinusoidal positional encodings on visual tokens with 3D RoPE frequencies derived from unprojected depth, and optionally prepend a $\delta M$ head as described above. The vision encoder and language backbone remain frozen; only the DiT blocks and the $\delta M$ MLP $f_\phi$ are updated during finetuning.