{"id":160,"date":"2026-05-09T02:52:02","date_gmt":"2026-05-09T02:52:02","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/?page_id=160"},"modified":"2026-05-09T03:44:19","modified_gmt":"2026-05-09T03:44:19","slug":"analysis","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/","title":{"rendered":"Analysis"},"content":{"rendered":"\n

Pedestrian Analysis<\/strong><\/h2>\n\n\n\n

Setup<\/strong><\/h3>\n\n\n\n

We evaluated RAP on 144 edited scenes per speed condition, where synthetic jaywalking pedestrians were inserted into front camera RGB. All analysis is conducted off-policy: RAP’s predicted trajectory is compared against the edited scene without closed-loop replanning. A scene is counted as a correct response if the model predicts a trajectory that stops before the pedestrian.<\/p>\n\n\n\n

RAP Stopping Behavior<\/strong><\/h3>\n\n\n\n

RAP stopped in fewer than half of all scenes at best. At 3.0 m\/s under original log acceleration, the model correctly stopped in only 48.61%<\/strong> of scenes despite all scenes being physically stoppable. Performance deteriorates sharply at higher speeds: at 7.5 m\/s, only 7.64%<\/strong> of scenes resulted in a correct stop. At highway speeds (13.1 m\/s), the model almost never stopped regardless of input acceleration.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

A critical insight emerges when varying the input acceleration. Explicitly providing a harder braking signal of -2 m\/s\u00b2 raises the stopping rate from 48.61% to 96.53%<\/strong> at 3.0 m\/s. This demonstrates that the model is physically capable of stopping but does not generate sufficient braking from visual input alone. The failure is not perceptual: it is a supervision gap rooted in the training data.<\/p>\n\n\n\n

\"\"
Qualitative trajectories across speeds and input acceleration values. Red boxes mark scenes where the model correctly stopped. Correct responses increase dramatically with harder input acceleration.<\/figcaption><\/figure>\n\n\n\n

Effect of Pedestrian Density<\/strong><\/h3>\n\n\n\n

We further evaluated whether pedestrian density affects RAP’s response across three conditions: single pedestrian, groups of 2 to 3, and larger groups. Reaction rate improves consistently with density. Group pedestrian scenes achieved 29.5%<\/strong> correct responses under original acceleration, compared to 12.8%<\/strong> for single pedestrian scenes. This suggests RAP’s response is partially driven by visual salience rather than semantic reasoning about individual pedestrians.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Why Does RAP Fail to Brake?<\/strong><\/h3>\n\n\n\n

Analysis of 103,173 nuPlan training scenes reveals the root cause. 75.7%<\/strong> of all scenes involve only comfortable braking under 3 m\/s\u00b2, and the maximum deceleration recorded in the entire dataset is 4.84 m\/s\u00b2. There are zero hard braking and zero emergency braking cases. The model has simply never seen a hard stop during training and cannot generate one from visual input alone.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Obstacle Analysis<\/strong><\/h2>\n\n\n\n

Setup<\/strong><\/h3>\n\n\n\n

We tested RAP on visual interventions covering four physical obstacle categories: broken down cars, road blockages, debris on road, and lane closures. Trajectory deviation was measured using ADE\/v, the average displacement error normalized by ego velocity, comparing predictions on original versus edited RGB scenes.<\/p>\n\n\n\n

\"\"
ADE\/v across all visual interventions. Physical obstacles cause measurable trajectory deviation while traffic sign interventions produce near-zero response.<\/figcaption><\/figure>\n\n\n\n

Key Findings<\/strong><\/h3>\n\n\n\n

Three consistent patterns emerge across all obstacle types. <\/p>\n\n\n\n

1. RAP responds better to larger obstacles. Large debris produces significantly higher ADE\/v than small debris, suggesting the model’s response is driven by the visual footprint of the object rather than its semantic identity. <\/p>\n\n\n\n

2. Reaction degrades sharply at higher velocities, with the model failing to respond to road blockages above 7.5 m\/s. <\/p>\n\n\n\n

3. The model occasionally steers around obstacles at low speeds but does not generate a complete stop, indicating partial but insufficient avoidance behavior.<\/p>\n\n\n\n

\"\"
Large vs small debris comparison. RAP reacts to large obstacles but ignores small debris.<\/figcaption><\/figure>\n\n\n\n
\"\"
Road blockage results across speeds. The model avoids at low speeds but fails to react above 7.5 m\/s.<\/figcaption><\/figure>\n\n\n\n

Traffic Sign Analysis<\/strong><\/h2>\n\n\n\n

No Reaction to Signs<\/strong><\/h3>\n\n\n\n

Inserting speed limit signs, school zone signs, and stop signs into RGB scenes produced near-zero trajectory deviation across all conditions. ADE\/v values for all three sign types are statistically indistinguishable from the non-traffic sign control condition at every tested ego speed. RAP plans as if traffic signs do not exist in the scene.<\/p>\n\n\n\n

\"\"
Traffic sign interventions produce no change in RAP’s predicted trajectory regardless of sign type or ego speed.<\/figcaption><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Encoder Analysis<\/strong><\/h2>\n\n\n\n

The absence of response raises a fundamental question: is the encoder failing to perceive the signs, or is the planner failing to act on them? <\/strong>To answer this, we analyzed the encoder’s projection layer using PCA. Comparing feature maps with and without a traffic sign present, the embeddings are visibly different, confirming the encoder does capture sign-related information. The failure lies entirely in the downstream planner, which does not route these features into trajectory prediction. Signs are seen but not acted upon.<\/p>\n\n\n\n

\"\"
PCA of encoder features with and without a traffic sign. Distinct token patterns confirm sign features are encoded but not utilized by the planner.<\/figcaption><\/figure>\n\n\n\n

Summary of Findings<\/strong><\/h2>\n\n\n\n

Our off-policy analysis across three tracks reveals a consistent picture. RAP’s failures trace back to two structural causes. <\/p>\n\n\n\n

First, the rasterized representation omits traffic sign information entirely, and while the encoder perceives signs in RGB, the planner is never trained to use them. <\/p>\n\n\n\n

Second, the nuPlan training distribution is heavily skewed toward comfortable, low-stakes driving scenarios, leaving the model without the supervision signal needed to respond to pedestrians, obstacles, or emergency situations at higher speeds. <\/p>\n\n\n\n

These findings directly motivate simulation-based data augmentation as the path forward.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Pedestrian Analysis Setup We evaluated RAP on 144 edited scenes per speed condition, where synthetic jaywalking pedestrians were inserted into front camera RGB. All analysis is conducted off-policy: RAP’s predicted trajectory is compared against the edited scene without closed-loop replanning. A scene is counted as a correct response if the model predicts a trajectory that … <\/p>\n

Continue reading “Analysis”<\/span><\/a><\/p>\n","protected":false},"author":286,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-160","page","type-page","status-publish","hentry"],"yoast_head":"\nAnalysis - Rasterized Representations for Autonomous Driving<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Analysis - Rasterized Representations for Autonomous Driving\" \/>\n<meta property=\"og:description\" content=\"Pedestrian Analysis Setup We evaluated RAP on 144 edited scenes per speed condition, where synthetic jaywalking pedestrians were inserted into front camera RGB. All analysis is conducted off-policy: RAP’s predicted trajectory is compared against the edited scene without closed-loop replanning. A scene is counted as a correct response if the model predicts a trajectory that … Continue reading "Analysis"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/\" \/>\n<meta property=\"og:site_name\" content=\"Rasterized Representations for Autonomous Driving\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-09T03:44:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/image-25.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1664\" \/>\n\t<meta property=\"og:image:height\" content=\"688\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/\",\"name\":\"Analysis - Rasterized Representations for Autonomous Driving\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/wp-content\\\/uploads\\\/sites\\\/152\\\/2026\\\/05\\\/image-25-1024x423.png\",\"datePublished\":\"2026-05-09T02:52:02+00:00\",\"dateModified\":\"2026-05-09T03:44:19+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/wp-content\\\/uploads\\\/sites\\\/152\\\/2026\\\/05\\\/image-25.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/wp-content\\\/uploads\\\/sites\\\/152\\\/2026\\\/05\\\/image-25.png\",\"width\":1664,\"height\":688},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/analysis\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Analysis\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/\",\"name\":\"Rasterized Representations for Autonomous Driving\",\"description\":\"Nikita Malik, Karthik Pullalarevu, Kris Kitani, Nicolas Ugrinovic, Varun Ramakrishna, Micol Marchetti-Bowick \",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf10\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Analysis - Rasterized Representations for Autonomous Driving","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/","og_locale":"en_US","og_type":"article","og_title":"Analysis - Rasterized Representations for Autonomous Driving","og_description":"Pedestrian Analysis Setup We evaluated RAP on 144 edited scenes per speed condition, where synthetic jaywalking pedestrians were inserted into front camera RGB. All analysis is conducted off-policy: RAP’s predicted trajectory is compared against the edited scene without closed-loop replanning. A scene is counted as a correct response if the model predicts a trajectory that … Continue reading \"Analysis\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/","og_site_name":"Rasterized Representations for Autonomous Driving","article_modified_time":"2026-05-09T03:44:19+00:00","og_image":[{"width":1664,"height":688,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/image-25.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/","name":"Analysis - Rasterized Representations for Autonomous Driving","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/image-25-1024x423.png","datePublished":"2026-05-09T02:52:02+00:00","dateModified":"2026-05-09T03:44:19+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/image-25.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-content\/uploads\/sites\/152\/2026\/05\/image-25.png","width":1664,"height":688},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/analysis\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/"},{"@type":"ListItem","position":2,"name":"Analysis"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/","name":"Rasterized Representations for Autonomous Driving","description":"Nikita Malik, Karthik Pullalarevu, Kris Kitani, Nicolas Ugrinovic, Varun Ramakrishna, Micol Marchetti-Bowick ","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages\/160","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/users\/286"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/comments?post=160"}],"version-history":[{"count":3,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages\/160\/revisions"}],"predecessor-version":[{"id":191,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/pages\/160\/revisions\/191"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf10\/wp-json\/wp\/v2\/media?parent=160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}