{"id":22,"date":"2026-05-05T18:06:13","date_gmt":"2026-05-05T18:06:13","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/?page_id=22"},"modified":"2026-05-05T18:58:55","modified_gmt":"2026-05-05T18:58:55","slug":"backgroundmotivation","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","title":{"rendered":"Background & Motivation"},"content":{"rendered":"\n

Background<\/strong><\/h2>\n\n\n\n

Autonomous vehicle planning has traditionally relied on RGB camera images, which are sensitive to lighting, shadows, and visual texture that have little to do with whether a vehicle should turn, brake, or yield. RAP (Rasterized Augmented Planning)<\/strong> by Feng et al. challenges this assumption. It hypothesizes that planning fundamentally depends on geometry and agent dynamics<\/strong>, not visual appearance.<\/p>\n\n\n\n

\"\"
RAP pipeline<\/strong>: annotated driving logs are converted to rasterized views and passed through a domain-aligned backbone for perception, trajectory, and scoring.<\/figcaption><\/figure>\n\n\n\n

RAP converts driving scenes into a rasterized view representation<\/strong> \u2014 a clean, structured encoding of road geometry, ego state, and surrounding agents as shown in the figure above. This representation enables scalable data augmentations that would be impossible with raw RGB. To bridge the gap between training and test time, RAP aligns raster features with RGB in a shared latent space<\/strong> using a domain-aligned backbone, so the model trains on raster augmentations but can run inference on either modality.<\/p>\n\n\n\n

Motivation<\/strong><\/h2>\n\n\n\n

RAP’s rasterized representation is powerful but it makes deliberate simplifications about what information matters for planning. Our work asks: what does RAP miss, and does it matter?<\/strong><\/p>\n\n\n\n

We identify two critical gaps in RAP’s current design that motivate this project:<\/p>\n\n\n\n

\"\"
Two core gaps in RAP’s raster representation drive our research questions.<\/figcaption><\/figure>\n\n\n\n

These gaps define the scope of our capstone. We investigate RAP’s failure modes through systematic off-policy analysis and simulate safety-critical long-tail scenarios using InterPlan on the nuPlan dataset, evaluating whether rasterized representations can serve as a foundation for more robust AV planning.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Background Autonomous vehicle planning has traditionally relied on RGB camera images, which are sensitive to lighting, shadows, and visual texture that have little to do with whether a vehicle should turn, brake, or yield. RAP (Rasterized Augmented Planning) by Feng et al. challenges this assumption. It hypothesizes that planning fundamentally depends on geometry and agent … <\/p>\n

Continue reading “Background & Motivation”<\/span><\/a><\/p>\n","protected":false},"author":286,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-22","page","type-page","status-publish","hentry"],"yoast_head":"\nBackground & Motivation - Rasterized Representations for Autonomous Driving<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Background & Motivation - Rasterized Representations for Autonomous Driving\" \/>\n<meta property=\"og:description\" content=\"Background Autonomous vehicle planning has traditionally relied on RGB camera images, which are sensitive to lighting, shadows, and visual texture that have little to do with whether a vehicle should turn, brake, or yield. RAP (Rasterized Augmented Planning) by Feng et al. challenges this assumption. It hypothesizes that planning fundamentally depends on geometry and agent … Continue reading "Background & Motivation"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/\" \/>\n<meta property=\"og:site_name\" content=\"Rasterized Representations for Autonomous Driving\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-05T18:58:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/Screenshot-2026-05-05-at-2.00.30-PM.png\" \/>\n\t<meta property=\"og:image:width\" content=\"686\" \/>\n\t<meta property=\"og:image:height\" content=\"491\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\",\"name\":\"Background & Motivation - Rasterized Representations for Autonomous Driving\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/wp-content\\\/uploads\\\/sites\\\/152\\\/2026\\\/05\\\/Screenshot-2026-05-05-at-2.00.30-PM.png\",\"datePublished\":\"2026-05-05T18:06:13+00:00\",\"dateModified\":\"2026-05-05T18:58:55+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/wp-content\\\/uploads\\\/sites\\\/152\\\/2026\\\/05\\\/Screenshot-2026-05-05-at-2.00.30-PM.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/wp-content\\\/uploads\\\/sites\\\/152\\\/2026\\\/05\\\/Screenshot-2026-05-05-at-2.00.30-PM.png\",\"width\":686,\"height\":491},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Background & Motivation\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\",\"name\":\"Rasterized Representations for Autonomous Driving\",\"description\":\"Nikita Malik, Karthik Pullalarevu, Kris Kitani, Nicolas Ugrinovic, Varun Ramakrishna, Micol Marchetti-Bowick \",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Background & Motivation - Rasterized Representations for Autonomous Driving","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","og_locale":"en_US","og_type":"article","og_title":"Background & Motivation - Rasterized Representations for Autonomous Driving","og_description":"Background Autonomous vehicle planning has traditionally relied on RGB camera images, which are sensitive to lighting, shadows, and visual texture that have little to do with whether a vehicle should turn, brake, or yield. RAP (Rasterized Augmented Planning) by Feng et al. challenges this assumption. It hypothesizes that planning fundamentally depends on geometry and agent … Continue reading \"Background & Motivation\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","og_site_name":"Rasterized Representations for Autonomous Driving","article_modified_time":"2026-05-05T18:58:55+00:00","og_image":[{"width":686,"height":491,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/Screenshot-2026-05-05-at-2.00.30-PM.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","name":"Background & Motivation - Rasterized Representations for Autonomous Driving","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/Screenshot-2026-05-05-at-2.00.30-PM.png","datePublished":"2026-05-05T18:06:13+00:00","dateModified":"2026-05-05T18:58:55+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/Screenshot-2026-05-05-at-2.00.30-PM.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/Screenshot-2026-05-05-at-2.00.30-PM.png","width":686,"height":491},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/"},{"@type":"ListItem","position":2,"name":"Background & Motivation"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","name":"Rasterized Representations for Autonomous Driving","description":"Nikita Malik, Karthik Pullalarevu, Kris Kitani, Nicolas Ugrinovic, Varun Ramakrishna, Micol Marchetti-Bowick ","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages\/22","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/users\/286"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/comments?post=22"}],"version-history":[{"count":8,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages\/22\/revisions"}],"predecessor-version":[{"id":47,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages\/22\/revisions\/47"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/media?parent=22"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}