Method

Figure 5. BEV Encoder

Figure 6. BEV-JEPA Self-Supervised Pretraining

Figure 7. BEV-JEPA Downstream Fine-Tuning Example on 3D Object Detection (Optional)

BEV Encoder and Fusion with Agents

We construct a unified BEV feature map by fusing multi-view camera and LiDAR signals in BEV space. For the camera branch, we adopt a pull-based BEV fusion strategy: for each BEV grid cell, we back-project the cell center into each camera view, sample the corresponding image feature from the multi-scale feature maps, and then aggregate features across views to obtain a BEV-aligned camera representation. In parallel, LiDAR point clouds are voxelized/encoded and projected into BEV to provide geometry-accurate features. We then fuse the BEV camera features and LiDAR BEV features (e.g., via concatenation + MLP/conv fusion) to capture both rich semantics (lanes, signals, markings) and 3D structure (free space, object geometry). For motion prediction, we extract agent-aligned BEV embeddings by sampling (or locally pooling) BEV features at each agent’s BEV location, and fuse them with the agent’s history and interaction representations to form the final per-agent token used by the trajectory decoder.

BEV-JEPA Self-Supervised Pretraining

We pretrain the BEV encoder using a JEPA-style objective to learn scene structure and dynamics directly in BEV space. Given a BEV feature map $Z \in \mathbb{R}^{B\times C\times H\times W}$ from the context encoder, we apply a random mask to a subset of BEV cells and use a predictor to infer the missing content (spatial JEPA) and/or future BEV features (temporal JEPA). A target encoder (EMA of the context encoder) produces the target BEV features $Z^{\star}$with stop-gradient, preventing representation collapse. Training combines a reconstruction loss with variance/covariance regularization to encourage informative, diverse embeddings.

Loss Definition:

Let:

• $Z_{\text{pred}} \in \mathbb{R}^{B\times C\times H\times W}$: predicted BEV feature map (predictor output)
• $Z_{\text{tgt}} \in \mathbb{R}^{B\times C\times H\times W}$: target BEV feature map (EMA encoder output, stop-grad)
• $\Omega$: set of supervised BEV cells (e.g., masked cells for spatial JEPA, or all cells for temporal JEPA)
• $M \in \{0,1\}^{B\times 1\times H\times W}$: mask indicator for $\Omega$Ω (1 if supervised)

1) Reconstruction MSE loss:$$\mathcal{L}_{\text{mse}} = \frac{1}{|\Omega|}\sum_{(b,h,w)\in\Omega} \left\| Z_{\text{pred}}[b,:,h,w]-Z_{\text{tgt}}[b,:,h,w] \right\|_2^2$$(Equivalently, implement with a mask $M$: average MSE only over masked/supervised cells.)

2) Variance regularization loss (avoid collapse)

First flatten spatial dims and compute per-channel std over samples:$$\tilde{Z} \in \mathbb{R}^{N \times C},\quad N = B\cdot H\cdot W \;(\text{or only } \Omega)$$$$\sigma_c = \sqrt{\mathrm{Var}(\tilde{Z}_{:,c})+\epsilon}$$Then enforce a minimum target standard deviation $\gamma$ (e.g., $\gamma=1.0$):$$\mathcal{L}_{\text{var}} = \frac{1}{C}\sum_{c=1}^{C}\max(0,\gamma-\sigma_c)$$(You can apply this to $Z_{\text{pred}}$​, $Z_{\text{tgt}}$​, or both; commonly applied to the predictor output.)

3) Covariance off-diagonal regularization loss (decorrelate channels)

Center features and compute covariance:$$\bar{Z}_{:,c}=\tilde{Z}_{:,c}-\frac{1}{N}\sum_{n=1}^{N}\tilde{Z}_{n,c}$$$$\Sigma = \frac{1}{N-1}\bar{Z}^\top \bar{Z} \in\mathbb{R}^{C\times C}$$

Penalize off-diagonal entries:$$\mathcal{L}_{\text{cov}} = \frac{1}{C}\sum_{i\neq j}\Sigma_{ij}^{2}$$This encourages different channels to encode complementary information rather than collapsing to redundant dimensions.

Combined Objective

$$\mathcal{L} = \lambda_{\text{mse}}\mathcal{L}_{\text{mse}} + \lambda_{\text{var}}\mathcal{L}_{\text{var}} + \lambda_{\text{cov}}\mathcal{L}_{\text{cov}}$$