{"id":9,"date":"2025-05-08T02:20:22","date_gmt":"2025-05-08T02:20:22","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/?page_id=9"},"modified":"2025-12-12T17:50:40","modified_gmt":"2025-12-12T17:50:40","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n
\n

Proposed Model Architecture<\/h2>\n\n\n\n
<\/div>\n\n\n\n

To address the challenges of multi-agent motion prediction under occlusion, complex interactions, and rare traffic-rule scenarios, we propose a vision\u2013language\u2013augmented, multi-modal architecture that jointly reasons over scene dynamics and traffic semantics. The model integrates agent history, inter-agent relationships, camera and LiDAR perception, and HD-map context, while explicitly incorporating traffic-rule knowledge distilled from a vision\u2013language model (VLM). Unlike conventional approaches that rely solely on trajectory patterns or visual cues, our design introduces rule-aware reasoning as a first-class signal, enabling safer, more interpretable, and legally consistent trajectory prediction in ambiguous driving situations.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
Figure1. Model Architecture<\/h5>\n\n\n\n
<\/div>\n\n\n\n

As illustrated in the figure, the architecture consists of three main components. First, a Scene Encoder processes multi-modal inputs\u2014including agent states, agent interactions, camera images, LiDAR point clouds, and map features\u2014using modality-specific encoders followed by cross-modal attention to produce unified scene embeddings for all agents. In parallel, a Traffic-Rule-Aware Encoder renders the transportation context into BEV maps and leverages a teacher VLM to extract high-level semantic signals (intentions, affordances, scenarios, and traffic rules), which are distilled into a lightweight student model for efficient inference. These scene and rule embeddings are then fused via dual attention and passed to an autoregressive Transformer decoder, which jointly predicts future trajectories for all agents. This modular design enables the model to capture interaction-aware motion dynamics while enforcing rule-compliant behavior, even when critical cues are visually occluded or underrepresented in the training data.<\/p>\n<\/div>\n\n\n\n

\n
\n

Input Data and Scene Encoder<\/h2>\n\n\n\n
<\/div>\n\n\n\n

The Scene Encoder takes multi-modal input data\u2014including agent state history, lane centerlines from HD maps, traffic light signals, agent\u2013agent interactions, and ego-centric camera and LiDAR observations\u2014and projects all modalities into a unified embedding space. Each modality is encoded independently using modality-specific attention encoders, after which cross-modal self-attention is applied to fuse spatial and semantic information across agents and sensors. This fusion produces a unified scene representation that captures both global scene context and local inter-agent dynamics. The resulting scene embedding has shape (R,N,L,H), where R denotes the number of rollouts, N the number of agents, L the temporal sequence length, and H the hidden dimension.<\/p>\n\n\n\n

<\/div>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n
Figure 2. Input Data and Scene Encoder<\/h5>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
\n

Agent State History<\/h3>\n\n\n\n

Encodes past positions, velocities, and headings of each agent, providing temporal cues about motion trends, maneuver intent (e.g., turning or braking), and short-term dynamics.<\/p>\n<\/div>\n\n\n\n

\n

Lane Centerlines (HD Maps)<\/h3>\n\n\n\n

Provide high-definition lane geometry and topology, helping align predicted trajectories with drivable space and enforcing structural constraints of the road layout.<\/p>\n<\/div>\n\n\n\n

\n

Traffic Light Signals<\/h3>\n\n\n\n

Encode the spatial location of nearby traffic lights, supplying critical information about motion legality and right-of-way at intersections.<\/p>\n<\/div>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n
\n
\n

Agent Interactions<\/h3>\n\n\n\n

Capture relative positions, velocities, and headings between nearby agents, enabling the model to reason about interactions such as yielding, following, merging, and cut-ins.<\/p>\n<\/div>\n\n\n\n

\n

Camera Images<\/h3>\n\n\n\n

Offer rich semantic information\u2014including lanes, traffic signs, pedestrians, and road markings\u2014that is essential for understanding scene context and visual affordances.<\/p>\n<\/div>\n\n\n\n

\n

LiDAR Point Clouds<\/h3>\n\n\n\n

Provide precise 3D geometry and distance measurements, complementing camera perception and remaining robust under challenging lighting or weather conditions.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

Traffic-Rule-Aware Encoder<\/h2>\n\n\n\n
<\/div>\n\n\n\n

The Traffic-Rule-Aware Encoder is designed to explicitly inject traffic-rule semantics into multi-agent motion prediction. Given the input scene, we extract rule-relevant spatial context\u2014including agent positions, lane geometry, traffic lights, crosswalks, and intersection layouts\u2014and rasterize it into a structured bird\u2019s-eye-view (BEV) representation. This BEV map is paired with a carefully constructed prompt that includes auto-generated scene descriptions, predefined semantic categories, traffic-rule definitions, and a strict output format, augmented with few-shot examples to ensure consistent reasoning. The prompt\u2013BEV pair is fed into a teacher vision\u2013language model (Qwen2.5-VL), which performs high-level reasoning to infer traffic semantics that are not always directly observable from raw sensor inputs.<\/p>\n\n\n\n

To enable efficient inference, the teacher VLM\u2019s outputs are distilled into a lightweight student model that predicts structured traffic-rule embeddings without querying the VLM at test time. The student is trained using Binary Cross-Entropy (BCE) losses against predefined ground-truth labels, producing interpretable and temporally aligned rule embeddings with shape (R,N,L,D), where R<\/mi><\/mrow>R<\/annotation><\/semantics><\/math>R is the number of rollouts, N the number of agents, L the sequence length, and D the total embedding dimension. These traffic-rule-aware embeddings are later fused with scene embeddings via dual attention, enabling the decoder to generate trajectories that are both interaction-consistent and legally compliant, even under occlusion or rare traffic scenarios.<\/p>\n\n\n\n

IASR Embedding Definition<\/p>\n\n\n\n

    \n
  • Intentions (I)<\/strong>: Encode the agent\u2019s intended maneuver (e.g., go straight, turn left\/right, stop, lane change) as a weighted one-hot vector in R<\/mi>8<\/mn><\/msup><\/mrow>\\mathbb{R}^8<\/annotation><\/semantics><\/math>, capturing high-level motion intent beyond short-term kinematics.<\/li>\n\n\n\n
  • Affordances (A)<\/strong>: Represent the legality or feasibility of possible actions (e.g., can turn left, must stop, must yield to oncoming traffic) as a binary vector inR<\/mi>10<\/mn><\/msup><\/mrow>\\mathbb{R}^{10}<\/annotation><\/semantics><\/math>, reflecting actionable constraints imposed by traffic rules and scene context.<\/li>\n\n\n\n
  • Scenario Types (S)<\/strong>: Describe the high-level traffic scenario (e.g., straight road, signalized intersection, stop-controlled intersection, pedestrian crossing, merge area) as a binary vector inR<\/mi>9<\/mn><\/msup><\/mrow>\\mathbb{R}^9<\/annotation><\/semantics><\/math>, providing global contextual grounding.<\/li>\n\n\n\n
  • Rules (R)<\/strong>: Explicitly encode applicable traffic rules (e.g., stop line ahead, crosswalk ahead, through-lane priority, no stopping in lane) as a binary vector inR<\/mi>9<\/mn><\/msup><\/mrow>\\mathbb{R}^9<\/annotation><\/semantics><\/math>, enabling interpretable rule enforcement during prediction.<\/li>\n<\/ul>\n\n\n\n

    The overall IASR loss is defined as: <\/p>\n\n\n\n

    \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n

    This enables the model to align predicted traffic-rule embeddings with structured, interpretable ground truth labels.<\/p>\n\n\n\n

    <\/div>\n\n\n\n
    \n
    \n

    Vision-Language Model (VLM)<\/h4>\n\n\n\n
      \n
    • Receives prompt and BEV rendered maps.<\/li>\n\n\n\n
    • Trained using Binary Cross-Entropy Loss to match GT-IAS labels from teacher VLM Qwen2.5VL model.<\/li>\n\n\n\n
    • Outputs legal-aware traffic embeddings.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
      \n
      \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
      Figure 3. Traffic-Rule-Aware Encoder Architecture<\/h5>\n<\/div>\n<\/div>\n\n\n\n
      <\/div>\n<\/div>\n\n\n\n
      \n
      \n
      \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
      Figure 4. Prompt Engineering Schema<\/h5>\n<\/div>\n\n\n\n
      \n

      Prompt Engineering<\/h3>\n\n\n\n
        \n
      • Transportation Context Map Rasterization converts dynamic and static traffic elements into BEV maps.\n
          \n
        • Captures stop signs, lights, lanes, and agent context to structure the prompt input.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
        • Text Prompt is autogenerated with captions describing the scene and actions<\/li>\n\n\n\n
        • Provides rule definitions and few-shot examples to guide the teacher VLM output.<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
          \n
          \n

          BEV Encoder and JEPA Self-Supervised Enhancement<\/h2>\n\n\n\n
          <\/div>\n\n\n\n
          <\/div>\n\n\n\n
          <\/div>\n\n\n\n

          A BEV encoder combines camera semantics and LiDAR geometry into BEV features; JEPA self-supervised pretraining improves robustness and generalization by learning scene structure and dynamics directly in BEV space.<\/p>\n<\/div>\n\n\n\n

          <\/div>\n\n\n\n
          \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n

          Figure 5. BEV Encoder<\/p>\n\n\n\n

          \"\"<\/figure>\n\n\n\n

          Figure 6. BEV-JEPA Self-Supervised Pretraining<\/p>\n\n\n\n

          \"\"<\/figure>\n\n\n\n

          Figure 7. BEV-JEPA Downstream Fine-Tuning Example on 3D Object Detection (Optional)<\/p>\n\n\n\n

          \n
          <\/div>\n\n\n\n
          <\/div>\n\n\n\n
          <\/div>\n<\/div>\n\n\n\n

          BEV Encoder and Fusion with Agents<\/h3>\n\n\n\n

          We construct a unified BEV feature map by fusing multi-view camera and LiDAR signals in BEV space. For the camera branch, we adopt a pull-based BEV fusion strategy: for each BEV grid cell, we back-project the cell center into each camera view, sample the corresponding image feature from the multi-scale feature maps, and then aggregate features across views to obtain a BEV-aligned camera representation. In parallel, LiDAR point clouds are voxelized\/encoded and projected into BEV to provide geometry-accurate features. We then fuse the BEV camera features and LiDAR BEV features (e.g., via concatenation + MLP\/conv fusion) to capture both rich semantics (lanes, signals, markings) and 3D structure (free space, object geometry). For motion prediction, we extract agent-aligned BEV embeddings by sampling (or locally pooling) BEV features at each agent\u2019s BEV location, and fuse them with the agent\u2019s history and interaction representations to form the final per-agent token used by the trajectory decoder.<\/p>\n\n\n\n

          BEV-JEPA Self-Supervised Pretraining<\/h3>\n\n\n\n

          We pretrain the BEV encoder using a JEPA-style objective to learn scene structure and dynamics directly in BEV space. Given a BEV feature map Z<\/mi>\u2208<\/mo>R<\/mi>B<\/mi>\u00d7<\/mo>C<\/mi>\u00d7<\/mo>H<\/mi>\u00d7<\/mo>W<\/mi><\/mrow><\/msup><\/mrow>Z \\in \\mathbb{R}^{B\\times C\\times H\\times W}<\/annotation><\/semantics><\/math> from the context encoder, we apply a random mask to a subset of BEV cells and use a predictor to infer the missing content (spatial JEPA) and\/or future BEV features (temporal JEPA). A target encoder (EMA of the context encoder) produces the target BEV features Z<\/mi>\u22c6<\/mo><\/msup><\/mrow>Z^{\\star}<\/annotation><\/semantics><\/math>with stop-gradient, preventing representation collapse. Training combines a reconstruction loss with variance\/covariance regularization to encourage informative, diverse embeddings.<\/p>\n\n\n\n

          Loss Definition:<\/p>\n\n\n\n

          Let:<\/p>\n\n\n\n

            \n
          • Z<\/mi>pred<\/mtext><\/msub>\u2208<\/mo>R<\/mi>B<\/mi>\u00d7<\/mo>C<\/mi>\u00d7<\/mo>H<\/mi>\u00d7<\/mo>W<\/mi><\/mrow><\/msup><\/mrow>Z_{\\text{pred}} \\in \\mathbb{R}^{B\\times C\\times H\\times W}<\/annotation><\/semantics><\/math>: predicted BEV feature map (predictor output)<\/li>\n\n\n\n
          • Z<\/mi>tgt<\/mtext><\/msub>\u2208<\/mo>R<\/mi>B<\/mi>\u00d7<\/mo>C<\/mi>\u00d7<\/mo>H<\/mi>\u00d7<\/mo>W<\/mi><\/mrow><\/msup><\/mrow>Z_{\\text{tgt}} \\in \\mathbb{R}^{B\\times C\\times H\\times W}<\/annotation><\/semantics><\/math>: target BEV feature map (EMA encoder output, stop-grad)<\/li>\n\n\n\n
          • \u03a9<\/mi><\/mrow>\\Omega<\/annotation><\/semantics><\/math>: set of supervised BEV cells (e.g., masked cells for spatial JEPA, or all cells for temporal JEPA)<\/li>\n\n\n\n
          • M<\/mi>\u2208<\/mo>{<\/mo>0<\/mn>,<\/mo>1<\/mn>}<\/mo>B<\/mi>\u00d7<\/mo>1<\/mn>\u00d7<\/mo>H<\/mi>\u00d7<\/mo>W<\/mi><\/mrow><\/msup><\/mrow>M \\in \\{0,1\\}^{B\\times 1\\times H\\times W}<\/annotation><\/semantics><\/math>: mask indicator for \u03a9<\/mi><\/mrow>\\Omega<\/annotation><\/semantics><\/math>\u03a9 (1 if supervised)<\/li>\n<\/ul>\n\n\n\n

            1) Reconstruction MSE loss<\/bdo>:<\/strong>L<\/mi>mse<\/mtext><\/msub>=<\/mo>1<\/mn>\u2223<\/mi>\u03a9<\/mi>\u2223<\/mi><\/mrow><\/mfrac>\u2211<\/mo>(<\/mo>b<\/mi>,<\/mo>h<\/mi>,<\/mo>w<\/mi>)<\/mo>\u2208<\/mo>\u03a9<\/mi><\/mrow><\/munder>\u2225<\/mo>Z<\/mi>pred<\/mtext><\/msub>[<\/mo>b<\/mi>,<\/mo>:<\/mo>,<\/mo>h<\/mi>,<\/mo>w<\/mi>]<\/mo>\u2212<\/mo>Z<\/mi>tgt<\/mtext><\/msub>[<\/mo>b<\/mi>,<\/mo>:<\/mo>,<\/mo>h<\/mi>,<\/mo>w<\/mi>]<\/mo>\u2225<\/mo><\/mrow>2<\/mn>2<\/mn><\/msubsup><\/mrow>\\mathcal{L}_{\\text{mse}} = \\frac{1}{|\\Omega|}\\sum_{(b,h,w)\\in\\Omega} \\left\\| Z_{\\text{pred}}[b,:,h,w]-Z_{\\text{tgt}}[b,:,h,w] \\right\\|_2^2<\/annotation><\/semantics><\/math>(Equivalently, implement with a mask M<\/mi><\/mrow>M<\/annotation><\/semantics><\/math>: average MSE only over masked\/supervised cells.)<\/p>\n\n\n\n
            <\/div>\n\n\n\n

            2) Variance regularization loss (avoid collapse)<\/strong><\/p>\n\n\n\n

            First flatten spatial dims and compute per-channel std over samples:Z<\/mi>~<\/mo><\/mover>\u2208<\/mo>R<\/mi>N<\/mi>\u00d7<\/mo>C<\/mi><\/mrow><\/msup>,<\/mo><\/mspace>N<\/mi>=<\/mo>B<\/mi>\u22c5<\/mo>H<\/mi>\u22c5<\/mo>W<\/mi>\u2005\u200a<\/mtext>(<\/mo>or only <\/mtext>\u03a9<\/mi>)<\/mo><\/mrow>\\tilde{Z} \\in \\mathbb{R}^{N \\times C},\\quad N = B\\cdot H\\cdot W \\;(\\text{or only } \\Omega)<\/annotation><\/semantics><\/math>\u03c3<\/mi>c<\/mi><\/msub>=<\/mo>V<\/mi>a<\/mi>r<\/mi><\/mrow>(<\/mo>Z<\/mi>~<\/mo><\/mover>:<\/mo>,<\/mo>c<\/mi><\/mrow><\/msub>)<\/mo>+<\/mo>\u03f5<\/mi><\/mrow><\/msqrt><\/mrow>\\sigma_c = \\sqrt{\\mathrm{Var}(\\tilde{Z}_{:,c})+\\epsilon}<\/annotation><\/semantics><\/math>Then enforce a minimum target standard deviation \u03b3<\/mi><\/mrow>\\gamma<\/annotation><\/semantics><\/math> (e.g., \u03b3<\/mi>=<\/mo>1.0<\/mn><\/mrow>\\gamma=1.0<\/annotation><\/semantics><\/math>):L<\/mi>var<\/mtext><\/msub>=<\/mo>1<\/mn>C<\/mi><\/mfrac>\u2211<\/mo>c<\/mi>=<\/mo>1<\/mn><\/mrow>C<\/mi><\/munderover>max<\/mi>\u2061<\/mo>(<\/mo>0<\/mn>,<\/mo>\u03b3<\/mi>\u2212<\/mo>\u03c3<\/mi>c<\/mi><\/msub>)<\/mo><\/mrow>\\mathcal{L}_{\\text{var}} = \\frac{1}{C}\\sum_{c=1}^{C}\\max(0,\\gamma-\\sigma_c)<\/annotation><\/semantics><\/math>(You can apply this to Z<\/mi>pred<\/mtext><\/msub><\/mrow>Z_{\\text{pred}}<\/annotation><\/semantics><\/math>\u200b, Z<\/mi>tgt<\/mtext><\/msub><\/mrow>Z_{\\text{tgt}}<\/annotation><\/semantics><\/math>\u200b, or both; commonly applied to the predictor output.)<\/p>\n\n\n\n
            <\/div>\n\n\n\n

            3) Covariance off-diagonal regularization loss (decorrelate channels)<\/strong><\/p>\n\n\n\n

            Center features and compute covariance:Z<\/mi>\u02c9<\/mo><\/mover>:<\/mo>,<\/mo>c<\/mi><\/mrow><\/msub>=<\/mo>Z<\/mi>~<\/mo><\/mover>:<\/mo>,<\/mo>c<\/mi><\/mrow><\/msub>\u2212<\/mo>1<\/mn>N<\/mi><\/mfrac>\u2211<\/mo>n<\/mi>=<\/mo>1<\/mn><\/mrow>N<\/mi><\/munderover>Z<\/mi>~<\/mo><\/mover>n<\/mi>,<\/mo>c<\/mi><\/mrow><\/msub><\/mrow>\\bar{Z}_{:,c}=\\tilde{Z}_{:,c}-\\frac{1}{N}\\sum_{n=1}^{N}\\tilde{Z}_{n,c}<\/annotation><\/semantics><\/math>\u03a3<\/mi>=<\/mo>1<\/mn>N<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/mfrac>Z<\/mi>\u02c9<\/mo><\/mover>\u22a4<\/mi><\/msup>Z<\/mi>\u02c9<\/mo><\/mover>\u2208<\/mo>R<\/mi>C<\/mi>\u00d7<\/mo>C<\/mi><\/mrow><\/msup><\/mrow>\\Sigma = \\frac{1}{N-1}\\bar{Z}^\\top \\bar{Z} \\in\\mathbb{R}^{C\\times C}<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

            Penalize off-diagonal entries:L<\/mi>cov<\/mtext><\/msub>=<\/mo>1<\/mn>C<\/mi><\/mfrac>\u2211<\/mo>i<\/mi>\u2260<\/mo>j<\/mi><\/mrow><\/munder>\u03a3<\/mi>i<\/mi>j<\/mi><\/mrow>2<\/mn><\/msubsup><\/mrow>\\mathcal{L}_{\\text{cov}} = \\frac{1}{C}\\sum_{i\\neq j}\\Sigma_{ij}^{2}<\/annotation><\/semantics><\/math>This encourages different channels to encode complementary information rather than collapsing to redundant dimensions.<\/p>\n\n\n\n

            Combined Objective<\/strong><\/h4>\n\n\n\n

            L<\/mi>=<\/mo>\u03bb<\/mi>mse<\/mtext><\/msub>L<\/mi>mse<\/mtext><\/msub>+<\/mo>\u03bb<\/mi>var<\/mtext><\/msub>L<\/mi>var<\/mtext><\/msub>+<\/mo>\u03bb<\/mi>cov<\/mtext><\/msub>L<\/mi>cov<\/mtext><\/msub><\/mrow>\\mathcal{L} = \\lambda_{\\text{mse}}\\mathcal{L}_{\\text{mse}} + \\lambda_{\\text{var}}\\mathcal{L}_{\\text{var}} + \\lambda_{\\text{cov}}\\mathcal{L}_{\\text{cov}}<\/annotation><\/semantics><\/math><\/p>\n<\/div>\n\n\n\n
            \n
            \n
            \n

            Decoder<\/h2>\n\n\n\n
            <\/div>\n\n\n\n

            The decoder generates future trajectories for each agent by integrating scene and traffic-rule-aware embeddings information into an autoregressive generation process. It leverages dual cross-attention to fuse two modalities\u2014scene context and legal reasoning\u2014and produces sequential outputs step-by-step. The decoder is trained using teacher forcing and evaluated autoregressively at test time.<\/p>\n\n\n\n

              \n
            • Dual Attention Fusion<\/strong>: Combines scene embeddings (R,N,L,H) and traffic-rule-aware embeddings (R,N,L,D) via bi-directional cross-attention for comprehensive context modeling.<\/li>\n\n\n\n
            • Autoregressive Generation<\/strong>: Predicts future coordinates step-by-step, using prior outputs, self-attention, and cross-attention to the input embeddings.<\/li>\n\n\n\n
            • Training & Inference<\/strong>: During training, teacher forcing is used (ground truth fed into next step). At inference, previous predictions are recursively used as inputs.<\/li>\n\n\n\n
            • Output Format<\/strong>: The decoder outputs a motion sequence of shape (R,N,T,2), where 2 corresponds to predicted (x, y) positions at each timestep.<\/li>\n<\/ul>\n\n\n\n
              \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n

              This motion loss maximizes the likelihood of predicted future positions given the agent history A before t\u200b and scene context S, averaged over all agents and time steps.<\/p>\n\n\n\n

              The following shows the decoder details. The decoder takes two inputs\u2014scene embeddings (R,N,L,H) from the Scene Encoder and traffic-rule-aware embeddings (R,N,L,D) from the traffic-rule-aware encoder\u2014each passed through an MLP to align dimensions. These are further fused using a Dual Attention module, which applies bi-directional cross-attention between the two modalities. The resulting fused embeddings are fed into a multi-layer autoregressive decoder composed of self-attention and cross-attention layers, allowing the model to generate future coordinates step-by-step while attending to both past predictions and contextual inputs. During training, teacher forcing is used by feeding in ground-truth positions, while inference is conducted autoregressively, relying on previously predicted outputs.<\/p>\n\n\n\n

              <\/div>\n\n\n\n
              \"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n<\/div>\n<\/div>\n\n\n\n
              Figure 8. Decoder Architecture<\/h5>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

              Proposed Model Architecture To address the challenges of multi-agent motion prediction under occlusion, complex interactions, and rare traffic-rule scenarios, we propose a vision\u2013language\u2013augmented, multi-modal architecture that jointly reasons over scene dynamics and traffic semantics. The model integrates agent history, inter-agent relationships, camera and LiDAR perception, and HD-map context, while explicitly incorporating traffic-rule knowledge distilled from […]<\/p>\n","protected":false},"author":244,"featured_media":0,"parent":0,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-9","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - 2025 Group 13 Project (P13)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - 2025 Group 13 Project (P13)\" \/>\n<meta property=\"og:description\" content=\"Proposed Model Architecture To address the challenges of multi-agent motion prediction under occlusion, complex interactions, and rare traffic-rule scenarios, we propose a vision\u2013language\u2013augmented, multi-modal architecture that jointly reasons over scene dynamics and traffic semantics. The model integrates agent history, inter-agent relationships, camera and LiDAR perception, and HD-map context, while explicitly incorporating traffic-rule knowledge distilled from […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"2025 Group 13 Project (P13)\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-12T17:50:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-content\/uploads\/sites\/129\/2025\/12\/model_arch.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1877\" \/>\n\t<meta property=\"og:image:height\" content=\"1042\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/\",\"name\":\"Method - 2025 Group 13 Project (P13)\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/wp-content\\\/uploads\\\/sites\\\/129\\\/2025\\\/12\\\/model_arch.png\",\"datePublished\":\"2025-05-08T02:20:22+00:00\",\"dateModified\":\"2025-12-12T17:50:40+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/wp-content\\\/uploads\\\/sites\\\/129\\\/2025\\\/12\\\/model_arch.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/wp-content\\\/uploads\\\/sites\\\/129\\\/2025\\\/12\\\/model_arch.png\",\"width\":1877,\"height\":1042},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/\",\"name\":\"2025 Group 13 Project (P13)\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025team6\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - 2025 Group 13 Project (P13)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - 2025 Group 13 Project (P13)","og_description":"Proposed Model Architecture To address the challenges of multi-agent motion prediction under occlusion, complex interactions, and rare traffic-rule scenarios, we propose a vision\u2013language\u2013augmented, multi-modal architecture that jointly reasons over scene dynamics and traffic semantics. The model integrates agent history, inter-agent relationships, camera and LiDAR perception, and HD-map context, while explicitly incorporating traffic-rule knowledge distilled from […]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/","og_site_name":"2025 Group 13 Project (P13)","article_modified_time":"2025-12-12T17:50:40+00:00","og_image":[{"width":1877,"height":1042,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-content\/uploads\/sites\/129\/2025\/12\/model_arch.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/","name":"Method - 2025 Group 13 Project (P13)","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-content\/uploads\/sites\/129\/2025\/12\/model_arch.png","datePublished":"2025-05-08T02:20:22+00:00","dateModified":"2025-12-12T17:50:40+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-content\/uploads\/sites\/129\/2025\/12\/model_arch.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-content\/uploads\/sites\/129\/2025\/12\/model_arch.png","width":1877,"height":1042},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/","name":"2025 Group 13 Project (P13)","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/pages\/9","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/comments?post=9"}],"version-history":[{"count":48,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/pages\/9\/revisions"}],"predecessor-version":[{"id":259,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/pages\/9\/revisions\/259"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025team6\/wp-json\/wp\/v2\/media?parent=9"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}