{"id":8,"date":"2026-05-08T00:29:51","date_gmt":"2026-05-08T00:29:51","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/?page_id=8"},"modified":"2026-05-08T04:29:42","modified_gmt":"2026-05-08T04:29:42","slug":"our_contribution","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/","title":{"rendered":"Our Contribution"},"content":{"rendered":"\n<figure class=\"wp-block-image alignwide size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"497\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-1024x497.png\" alt=\"\" class=\"wp-image-55\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-1024x497.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-300x146.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-768x373.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-1536x746.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-2048x995.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<!-- Paste this into a WordPress \"Custom HTML\" block -->\n\n<div style=\"max-width:760px;margin:0 auto\">\n\n<h2 style=\"margin-top:0\">Method<\/h2>\n\n<h3 style=\"margin-top:2rem;margin-bottom:0.5rem\">3D scene representation<\/h3>\n<p style=\"margin-top:0\">\nGiven an RGB-D observation, a frozen ViT encoder extracts a sequence of <em>N<\/em> patch tokens <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5C%7B%5Cmathbf%7Bz%7D_i%5C%7D_%7Bi%3D1%7D%5EN%20%5Cin%20%5Cmathbb%7BR%7D%5Ed\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\{\\mathbf{z}_i\\}_{i=1}^N \\in \\mathbb{R}^d\">. Each patch center is unprojected through the depth map and camera intrinsics <em>K<\/em> to yield a 3D point <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathbf%7Bp%7D_i%20%3D%20(x_i%2C%20y_i%2C%20z_i)%5E%5Ctop\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathbf{p}_i = (x_i, y_i, z_i)^\\top\"> in camera coordinates. These coordinates serve as positional inputs to the denoising transformer via 3D RoPE rather than being concatenated to the token features, preserving the geometry in the attention structure rather than the embedding space.\n<\/p>\n\n<h3 style=\"margin-top:2rem;margin-bottom:0.5rem\">3D Rotary Position Embeddings<\/h3>\n<p style=\"margin-top:0\">\nStandard 1D RoPE encodes position <em>m<\/em> into query\/key features by rotating pairs of dimensions by angle <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?m%5Ctheta_j\" style=\"height:1.1em;vertical-align:middle\" alt=\"m\\theta_j\">, where <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctheta_j%20%3D%20b%5E%7B-2j%2Fd%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\theta_j = b^{-2j\/d}\"> is a base-dependent frequency. We extend this to three independent spatial axes. For a token at position <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?(m_x%2C%20m_y%2C%20m_z)\" style=\"height:1.1em;vertical-align:middle\" alt=\"(m_x, m_y, m_z)\">, the rotation matrix applied to a <em>d<\/em>-dimensional feature vector is:\n<\/p>\n\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathcal%7BR%7D_%7Bm_x%2Cm_y%2Cm_z%7D%20%3D%20%5Cmathrm%7Bblockdiag%7D%5C!%5Cleft(%5Cmathcal%7BR%7D%5Ex_%7Bm_x%7D%2C%5C%2C%20%5Cmathcal%7BR%7D%5Ey_%7Bm_y%7D%2C%5C%2C%20%5Cmathcal%7BR%7D%5Ez_%7Bm_z%7D%5Cright)%20%5Cin%20SO(d)\" style=\"max-width:100%;height:auto\" alt=\"\\mathcal{R}_{m_x,m_y,m_z} = \\mathrm{blockdiag}\\!\\left(\\mathcal{R}^x_{m_x},\\, \\mathcal{R}^y_{m_y},\\, \\mathcal{R}^z_{m_z}\\right) \\in SO(d)\"><\/div>\n\n<p>where each axis block <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathcal%7BR%7D%5E%5Calpha_%7Bm_%5Calpha%7D%20%5Cin%20%5Cmathbb%7BR%7D%5E%7B(d%2F3)%5Ctimes(d%2F3)%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathcal{R}^\\alpha_{m_\\alpha} \\in \\mathbb{R}^{(d\/3)\\times(d\/3)}\"> is itself block diagonal over frequency pairs <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?j%20%3D%201%2C%20%5Cldots%2C%20d%2F6\" style=\"height:1.1em;vertical-align:middle\" alt=\"j = 1, \\ldots, d\/6\">:<\/p>\n\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cleft%5B%5Cmathcal%7BR%7D%5E%5Calpha_%7Bm_%5Calpha%7D%5Cright%5D%5E%7B(j)%7D%20%3D%20%5Cbegin%7Bpmatrix%7D%20%5Ccos%20m_%5Calpha%20%5Ctheta_j%20%26%20-%5Csin%20m_%5Calpha%20%5Ctheta_j%20%5C%5C%20%5Csin%20m_%5Calpha%20%5Ctheta_j%20%26%20%5Ccos%20m_%5Calpha%20%5Ctheta_j%20%5Cend%7Bpmatrix%7D\" style=\"max-width:100%;height:auto\" alt=\"\\left[\\mathcal{R}^\\alpha_{m_\\alpha}\\right]^{(j)} = \\begin{pmatrix} \\cos m_\\alpha \\theta_j &amp; -\\sin m_\\alpha \\theta_j \\\\ \\sin m_\\alpha \\theta_j &amp; \\cos m_\\alpha \\theta_j \\end{pmatrix}\"><\/div>\n\n<p>The rotated query and key for token <em>i<\/em> are <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctilde%7B%5Cmathbf%7Bq%7D%7D_i%20%3D%20%5Cmathcal%7BR%7D_%7Bm_x%5Ei%2C%20m_y%5Ei%2C%20m_z%5Ei%7D%5C%2C%20%5Cmathbf%7Bq%7D_i\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\tilde{\\mathbf{q}}_i = \\mathcal{R}_{m_x^i, m_y^i, m_z^i}\\, \\mathbf{q}_i\"> and analogously for <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctilde%7B%5Cmathbf%7Bk%7D%7D_i\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\tilde{\\mathbf{k}}_i\">. The attention score between tokens <em>i<\/em> and <em>j<\/em> then depends only on their relative 3D displacement:<\/p>\n\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctilde%7B%5Cmathbf%7Bq%7D%7D_i%5E%5Ctop%20%5Ctilde%7B%5Cmathbf%7Bk%7D%7D_j%20%3D%20%5Cmathbf%7Bq%7D_i%5E%5Ctop%20%5Cmathcal%7BR%7D_%7Bm_x%5Ei%20-%20m_x%5Ej%2C%5C%2C%20m_y%5Ei%20-%20m_y%5Ej%2C%5C%2C%20m_z%5Ei%20-%20m_z%5Ej%7D%5C%2C%20%5Cmathbf%7Bk%7D_j\" style=\"max-width:100%;height:auto\" alt=\"\\tilde{\\mathbf{q}}_i^\\top \\tilde{\\mathbf{k}}_j = \\mathbf{q}_i^\\top \\mathcal{R}_{m_x^i - m_x^j,\\, m_y^i - m_y^j,\\, m_z^i - m_z^j}\\, \\mathbf{k}_j\"><\/div>\n\n<p>which encodes translation equivariance in 3D space directly into attention.<\/p>\n\n<h3 style=\"margin-top:2rem;margin-bottom:0.5rem\">Flow matching objective<\/h3>\n<p style=\"margin-top:0\">\nThe policy models the conditional distribution over action trajectories <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctau%20%3D%20(a_t%2C%20%5Cldots%2C%20a_%7Bt%2BH-1%7D)\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\tau = (a_t, \\ldots, a_{t+H-1})\"> via a learned vector field. Given a scene encoding <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathbf%7Bc%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathbf{c}\"> (3D scene tokens, proprioception, and language), we define a time-conditional flow from a noise distribution <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?p_0%20%3D%20%5Cmathcal%7BN%7D(0%2C%20I)\" style=\"height:1.1em;vertical-align:middle\" alt=\"p_0 = \\mathcal{N}(0, I)\"> to the data distribution <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?p_1\" style=\"height:1.1em;vertical-align:middle\" alt=\"p_1\"> via the ODE:\n<\/p>\n\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cfrac%7Bd%5Ctau%5Es%7D%7Bds%7D%20%3D%20v_%5Cpsi(%5Ctau%5Es%2C%20s%2C%20%5Cmathbf%7Bc%7D)%2C%20%5Cquad%20s%20%5Cin%20%5B0%2C1%5D\" style=\"max-width:100%;height:auto\" alt=\"\\frac{d\\tau^s}{ds} = v_\\psi(\\tau^s, s, \\mathbf{c}), \\quad s \\in [0,1]\"><\/div>\n\n<p>The vector field <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?v_%5Cpsi\" style=\"height:1.1em;vertical-align:middle\" alt=\"v_\\psi\"> is parameterized by the denoising transformer and trained with the conditional flow matching loss:<\/p>\n\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathcal%7BL%7D_%5Cmathrm%7BFM%7D%20%3D%20%5Cmathbb%7BE%7D_%7Bs%2C%5C%2C%5Ctau%5E0%2C%5C%2C%5Ctau%5E1%7D%5C!%5Cleft%5B%5Cleft%5C%7Cv_%5Cpsi(%5Ctau%5Es%2C%20s%2C%20%5Cmathbf%7Bc%7D)%20-%20(%5Ctau%5E1%20-%20%5Ctau%5E0)%5Cright%5C%7C%5E2%5Cright%5D\" style=\"max-width:100%;height:auto\" alt=\"\\mathcal{L}_\\mathrm{FM} = \\mathbb{E}_{s,\\,\\tau^0,\\,\\tau^1}\\!\\left[\\left\\|v_\\psi(\\tau^s, s, \\mathbf{c}) - (\\tau^1 - \\tau^0)\\right\\|^2\\right]\"><\/div>\n\n<p>where <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctau%5Es%20%3D%20(1-s)%5Ctau%5E0%20%2B%20s%5Ctau%5E1\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\tau^s = (1-s)\\tau^0 + s\\tau^1\"> is the linear interpolant, <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctau%5E0%20%5Csim%20%5Cmathcal%7BN%7D(0%2CI)\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\tau^0 \\sim \\mathcal{N}(0,I)\">, and <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctau%5E1%20%5Csim%20p_%5Cmathrm%7Bdata%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\tau^1 \\sim p_\\mathrm{data}\">.<\/p>\n\n<h3 style=\"margin-top:2rem;margin-bottom:0.5rem\">Learned calibration correction<\/h3>\n\n<div style=\"margin-bottom:1.25rem\">\n<p style=\"margin-bottom:0.4rem\"><strong>Problem setup.<\/strong><\/p>\n<p style=\"margin-top:0\">Let <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?R%5E*%20%5Cin%20SO(3)\" style=\"height:1.1em;vertical-align:middle\" alt=\"R^* \\in SO(3)\"> denote the unknown camera-to-robot rotation for a given data source. Points reconstructed from the camera are expressed as <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathbf%7Bp%7D%5E%5Cmathrm%7Bcam%7D%20%3D%20R%5E%7B*%5Ctop%7D%20%5Cmathbf%7Bp%7D%5E%5Cmathrm%7Brobot%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathbf{p}^\\mathrm{cam} = R^{*\\top} \\mathbf{p}^\\mathrm{robot}\">, so feeding <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathbf%7Bp%7D%5E%5Cmathrm%7Bcam%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathbf{p}^\\mathrm{cam}\"> into 3D RoPE implicitly applies an erroneous coordinate frame. We seek to learn a correction that absorbs <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?R%5E*\" style=\"height:1.1em;vertical-align:middle\" alt=\"R^*\"> from data alone.<\/p>\n<\/div>\n\n<div style=\"margin-bottom:1.25rem\">\n<p style=\"margin-bottom:0.4rem\"><strong>Camera token.<\/strong><\/p>\n<p style=\"margin-top:0\">For a frame with <em>N<\/em> patch tokens <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5C%7B%5Cmathbf%7Bz%7D_i%5C%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\{\\mathbf{z}_i\\}\">, we form a camera token by mean pooling:<\/p>\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathbf%7Bc%7D_%5Cmathrm%7Bcam%7D%20%3D%20%5Cfrac%7B1%7D%7BN%7D%5Csum_%7Bi%3D1%7D%5EN%20%5Cmathbf%7Bz%7D_i%20%5Cin%20%5Cmathbb%7BR%7D%5Ed\" style=\"max-width:100%;height:auto\" alt=\"\\mathbf{c}_\\mathrm{cam} = \\frac{1}{N}\\sum_{i=1}^N \\mathbf{z}_i \\in \\mathbb{R}^d\"><\/div>\n<\/div>\n\n<div style=\"margin-bottom:1.25rem\">\n<p style=\"margin-bottom:0.4rem\"><strong>Parameterization via the matrix exponential.<\/strong><\/p>\n<p style=\"margin-top:0\">We require the correction <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M%20%5Cin%20SO(d)\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M \\in SO(d)\"> to be a valid rotation matrix so that the modified RoPE <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_%5Cmathrm%7Bfull%7D%20%5Ccdot%20%5Cmathcal%7BR%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_\\mathrm{full} \\cdot \\mathcal{R}\"> remains an element of <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?SO(d)\" style=\"height:1.1em;vertical-align:middle\" alt=\"SO(d)\">. We enforce this by construction: an MLP <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?f_%5Cphi\" style=\"height:1.1em;vertical-align:middle\" alt=\"f_\\phi\"> maps <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathbf%7Bc%7D_%5Cmathrm%7Bcam%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathbf{c}_\\mathrm{cam}\"> to the entries of a skew-symmetric matrix <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?A%20%5Cin%20%5Cmathfrak%7Bso%7D(6)\" style=\"height:1.1em;vertical-align:middle\" alt=\"A \\in \\mathfrak{so}(6)\">, and <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_6%20%3D%20%5Cexp(A)\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_6 = \\exp(A)\"> is obtained via the matrix exponential. Skew-symmetry <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?A%5E%5Ctop%20%3D%20-A\" style=\"height:1.1em;vertical-align:middle\" alt=\"A^\\top = -A\"> guarantees:<\/p>\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_6%5E%5Ctop%20%5Cdelta%20M_6%20%3D%20%5Cexp(A%5E%5Ctop)%5Cexp(A)%20%3D%20%5Cexp(-A)%5Cexp(A)%20%3D%20I\" style=\"max-width:100%;height:auto\" alt=\"\\delta M_6^\\top \\delta M_6 = \\exp(A^\\top)\\exp(A) = \\exp(-A)\\exp(A) = I\"><\/div>\n<p>so <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_6%20%5Cin%20SO(6)\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_6 \\in SO(6)\"> by construction, without any projection step or penalty term.<\/p>\n<\/div>\n\n<div style=\"margin-bottom:1.25rem\">\n<p style=\"margin-bottom:0.4rem\"><strong>Lifting to full dimension.<\/strong><\/p>\n<p style=\"margin-top:0\">The RoPE frequency vector for a token at position <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?(m_x%2C%20m_y%2C%20m_z)\" style=\"height:1.1em;vertical-align:middle\" alt=\"(m_x, m_y, m_z)\"> is a stacked 6-vector of cosines and sines across the three axes. To act on the full <em>d<\/em>-dimensional feature space, <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_6\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_6\"> is tiled as:<\/p>\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_%5Cmathrm%7Bfull%7D%20%3D%20%5Cmathrm%7Bblockdiag%7D(%5Cunderbrace%7B%5Cdelta%20M_6%2C%20%5Cldots%2C%20%5Cdelta%20M_6%7D_%7Bd%2F6%7D)%20%5Cin%20SO(d)\" style=\"max-width:100%;height:auto\" alt=\"\\delta M_\\mathrm{full} = \\mathrm{blockdiag}(\\underbrace{\\delta M_6, \\ldots, \\delta M_6}_{d\/6}) \\in SO(d)\"><\/div>\n<p>Since <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_%5Cmathrm%7Bfull%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_\\mathrm{full}\"> is block diagonal with identical blocks, <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_%5Cmathrm%7Bfull%7D%5E%5Ctop%20%5Cdelta%20M_%5Cmathrm%7Bfull%7D%20%3D%20I\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_\\mathrm{full}^\\top \\delta M_\\mathrm{full} = I\"> if and only if <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M_6%5E%5Ctop%20%5Cdelta%20M_6%20%3D%20I\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M_6^\\top \\delta M_6 = I\">. Orthogonality of the tile is therefore necessary and sufficient for orthogonality of the full matrix.<\/p>\n<\/div>\n\n<div style=\"margin-bottom:1.25rem\">\n<p style=\"margin-bottom:0.4rem\"><strong>Modified RoPE.<\/strong><\/p>\n<p style=\"margin-top:0\">The corrected position encoding for token <em>i<\/em> replaces the raw cosine-sine vector <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cboldsymbol%7B%5Cphi%7D_i%20%3D%20%5B%5Ccos%5Ctheta%5Ex_i%2C%5C%2C%20%5Ccos%5Ctheta%5Ey_i%2C%5C%2C%20%5Ccos%5Ctheta%5Ez_i%2C%5C%2C%20%5Csin%5Ctheta%5Ex_i%2C%5C%2C%20%5Csin%5Ctheta%5Ey_i%2C%5C%2C%20%5Csin%5Ctheta%5Ez_i%5D%5E%5Ctop\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\boldsymbol{\\phi}_i = [\\cos\\theta^x_i,\\, \\cos\\theta^y_i,\\, \\cos\\theta^z_i,\\, \\sin\\theta^x_i,\\, \\sin\\theta^y_i,\\, \\sin\\theta^z_i]^\\top\"> with:<\/p>\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctilde%7B%5Cboldsymbol%7B%5Cphi%7D%7D_i%20%3D%20%5Cdelta%20M_6%5C%2C%20%5Cboldsymbol%7B%5Cphi%7D_i\" style=\"max-width:100%;height:auto\" alt=\"\\tilde{\\boldsymbol{\\phi}}_i = \\delta M_6\\, \\boldsymbol{\\phi}_i\"><\/div>\n<p>The relative-position property of RoPE is preserved under this transformation: for any two tokens <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?i%2C%20j\" style=\"height:1.1em;vertical-align:middle\" alt=\"i, j\">,<\/p>\n<div style=\"margin:1.25rem 0;text-align:center\"><img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Ctilde%7B%5Cboldsymbol%7B%5Cphi%7D%7D_i%5E%5Ctop%20%5Ctilde%7B%5Cboldsymbol%7B%5Cphi%7D%7D_j%20%3D%20%5Cboldsymbol%7B%5Cphi%7D_i%5E%5Ctop%20%5Cdelta%20M_6%5E%5Ctop%20%5Cdelta%20M_6%5C%2C%20%5Cboldsymbol%7B%5Cphi%7D_j%20%3D%20%5Cboldsymbol%7B%5Cphi%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cphi%7D_j\" style=\"max-width:100%;height:auto\" alt=\"\\tilde{\\boldsymbol{\\phi}}_i^\\top \\tilde{\\boldsymbol{\\phi}}_j = \\boldsymbol{\\phi}_i^\\top \\delta M_6^\\top \\delta M_6\\, \\boldsymbol{\\phi}_j = \\boldsymbol{\\phi}_i^\\top \\boldsymbol{\\phi}_j\"><\/div>\n<p>so the correction acts as a global re-orientation of the coordinate frame rather than a distortion of pairwise geometry. The entire model including <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?f_%5Cphi\" style=\"height:1.1em;vertical-align:middle\" alt=\"f_\\phi\"> is trained end-to-end with <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cmathcal%7BL%7D_%5Cmathrm%7BFM%7D\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\mathcal{L}_\\mathrm{FM}\">, with no auxiliary supervision on <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M\">.<\/p>\n<\/div>\n\n<h3 style=\"margin-top:2rem;margin-bottom:0.5rem\">Extension to GR00T N1.5<\/h3>\n<p style=\"margin-top:0\">\nGR00T N1.5 follows a DiT-based architecture with alternating cross-attention (vision-to-action) and self-attention (action) blocks. We replace the 2D sinusoidal positional encodings on visual tokens with 3D RoPE frequencies derived from unprojected depth, and optionally prepend a <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M\"> head as described above. The vision encoder and language backbone remain frozen; only the DiT blocks and the <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?%5Cdelta%20M\" style=\"height:1.1em;vertical-align:middle\" alt=\"\\delta M\"> MLP <img decoding=\"async\" src=\"https:\/\/latex.codecogs.com\/svg.image?f_%5Cphi\" style=\"height:1.1em;vertical-align:middle\" alt=\"f_\\phi\"> are updated during finetuning.\n<\/p>\n\n<\/div>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Method 3D scene representation Given an RGB-D observation, a frozen ViT encoder extracts a sequence of N patch tokens . Each patch center is unprojected through the depth map and camera intrinsics K to yield a 3D point in camera coordinates. These coordinates serve as positional inputs to the denoising transformer via 3D RoPE rather [&hellip;]<\/p>\n","protected":false},"author":283,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-8","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data\" \/>\n<meta property=\"og:description\" content=\"Method 3D scene representation Given an RGB-D observation, a frozen ViT encoder extracts a sequence of N patch tokens . Each patch center is unprojected through the depth map and camera intrinsics K to yield a 3D point in camera coordinates. These coordinates serve as positional inputs to the denoising transformer via 3D RoPE rather [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/\" \/>\n<meta property=\"og:site_name\" content=\"Learning 3D-VLAs with Noisy Miscalibrated Data\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-08T04:29:42+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-scaled.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1244\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/\",\"name\":\"Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/wp-content\\\/uploads\\\/sites\\\/150\\\/2026\\\/05\\\/Screenshot-2026-05-07-at-11.55.33-PM-1024x497.png\",\"datePublished\":\"2026-05-08T00:29:51+00:00\",\"dateModified\":\"2026-05-08T04:29:42+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/wp-content\\\/uploads\\\/sites\\\/150\\\/2026\\\/05\\\/Screenshot-2026-05-07-at-11.55.33-PM-scaled.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/wp-content\\\/uploads\\\/sites\\\/150\\\/2026\\\/05\\\/Screenshot-2026-05-07-at-11.55.33-PM-scaled.png\",\"width\":2560,\"height\":1244},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/our_contribution\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Our Contribution\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/\",\"name\":\"Learning 3D-VLAs with Noisy Miscalibrated Data\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf8\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/","og_locale":"en_US","og_type":"article","og_title":"Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data","og_description":"Method 3D scene representation Given an RGB-D observation, a frozen ViT encoder extracts a sequence of N patch tokens . Each patch center is unprojected through the depth map and camera intrinsics K to yield a 3D point in camera coordinates. These coordinates serve as positional inputs to the denoising transformer via 3D RoPE rather [&hellip;]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/","og_site_name":"Learning 3D-VLAs with Noisy Miscalibrated Data","article_modified_time":"2026-05-08T04:29:42+00:00","og_image":[{"width":2560,"height":1244,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-scaled.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/","name":"Our Contribution - Learning 3D-VLAs with Noisy Miscalibrated Data","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-1024x497.png","datePublished":"2026-05-08T00:29:51+00:00","dateModified":"2026-05-08T04:29:42+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-scaled.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-content\/uploads\/sites\/150\/2026\/05\/Screenshot-2026-05-07-at-11.55.33-PM-scaled.png","width":2560,"height":1244},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/our_contribution\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/"},{"@type":"ListItem","position":2,"name":"Our Contribution"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/","name":"Learning 3D-VLAs with Noisy Miscalibrated Data","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/pages\/8","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/users\/283"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/comments?post=8"}],"version-history":[{"count":20,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/pages\/8\/revisions"}],"predecessor-version":[{"id":73,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/pages\/8\/revisions\/73"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf8\/wp-json\/wp\/v2\/media?parent=8"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}