Background
Autonomous vehicle planning has traditionally relied on RGB camera images, which are sensitive to lighting, shadows, and visual texture that have little to do with whether a vehicle should turn, brake, or yield. RAP (Rasterized Augmented Planning) by Feng et al. challenges this assumption. It hypothesizes that planning fundamentally depends on geometry and agent dynamics, not visual appearance.
RAP converts driving scenes into a rasterized view representation — a clean, structured encoding of road geometry, ego state, and surrounding agents as shown in the figure above. This representation enables scalable data augmentations that would be impossible with raw RGB. To bridge the gap between training and test time, RAP aligns raster features with RGB in a shared latent space using a domain-aligned backbone, so the model trains on raster augmentations but can run inference on either modality.
Motivation
RAP’s rasterized representation is powerful but it makes deliberate simplifications about what information matters for planning. Our work asks: what does RAP miss, and does it matter?
We identify two critical gaps in RAP’s current design that motivate this project:
These gaps define the scope of our capstone. We investigate RAP’s failure modes through systematic off-policy analysis and simulate safety-critical long-tail scenarios using InterPlan on the nuPlan dataset, evaluating whether rasterized representations can serve as a foundation for more robust AV planning.