{"id":12,"date":"2025-12-11T06:36:23","date_gmt":"2025-12-11T06:36:23","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/"},"modified":"2025-12-12T01:29:50","modified_gmt":"2025-12-12T01:29:50","slug":"overview","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/","title":{"rendered":"Introduction"},"content":{"rendered":"\n

Diffusion Models for Action Generation<\/strong><\/h3>\n\n\n\n
\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Diffusion models have recently achieved impressive results in action generation, spanning both robotic manipulation and autonomous driving.<\/p>\n\n\n\n

On the left, Diffusion Policy<\/strong> demonstrates one of the earliest successes: it reformulates control as a denoising process, generating smooth and consistent action sequences directly from image observations. By iteratively refining noisy action samples, the policy predicts future motion over a time horizon in a stable and expressive way.<\/p>\n\n\n\n

On the right, we see how diffusion models naturally capture multi-modal behaviors<\/strong> in driving. The learned latent distribution becomes multi-peaked, allowing the model to represent diverse possible futures\u2014such as turning left, going straight, or turning right. This inherent multi-modality makes diffusion particularly well-suited for complex, decision-rich driving scenarios.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Drivers and road environments differ widey around the world.<\/strong><\/h3>\n\n\n\n
\"\"
Different drivers have different driving styles.<\/figcaption><\/figure>\n\n\n\n
\"\"
Typical roads in China and the US.<\/figcaption><\/figure>\n\n\n\n

Even within the same city, people exhibit diverse driving styles \u2014 from calm and conservative to aggressive and hurried \u2014 leading to very different trajectory preferences.<\/p>\n\n\n\n

Across countries, the contrast becomes even larger:<\/p>\n\n\n\n

cities like those in China feature dense, multimodal traffic with frequent interactions, while the U.S. often has wider roads, clearer lanes, and more structured flow.<\/p>\n\n\n\n

These variations in driving style, traffic culture, and visual appearance<\/strong> create a significant distribution shift. As a result, a driving model trained in one style or region may struggle when deployed in another.<\/p>\n\n\n\n

This motivates our work: enabling diffusion-based driving policies to adapt flexibly across different driver behaviors and road environments<\/strong> without retraining the entire model.<\/p>\n\n\n\n


Comparison Between Different Finetuning Methods<\/strong>
<\/strong><\/h3>\n\n\n\n
Method<\/strong><\/td>Core Idea<\/strong><\/td>Modified Part<\/strong><\/td>Cons<\/strong><\/td><\/tr>
Direct Optimization<\/strong><\/td>Update diffusion weights to maximize reward<\/td>Model params<\/td>Expensive in diffusion (deep nested backprop), unstable, may break imitation<\/td><\/tr>
Rejection Sampling<\/strong><\/td>Sample multiple actions \u2192 pick best via value<\/td>No params changed<\/td>High compute, slow for real-time<\/td><\/tr>
Residual Policy<\/strong><\/td>Train small residual on top of frozen base<\/td>Add-on module<\/td>Limited improvement<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Reinforcement learning has been used in several ways to finetune diffusion-based policies, but each existing approach has notable limitations:<\/p>\n\n\n\n

1. Direct Optimization<\/strong><\/p>\n\n\n\n

This method directly updates the diffusion model\u2019s parameters to maximize reward. While expressive, it is extremely expensive \u2014 diffusion requires deep, nested backpropagation and often suffers from unstable gradients, which can break the learned imitation behavior.<\/p>\n\n\n\n

2. Rejection Sampling<\/strong><\/p>\n\n\n\n

This approach keeps the diffusion model frozen and instead samples many candidate actions, selecting the one with the highest value. The downside is high computational cost, making it too slow for real-time autonomous driving.<\/p>\n\n\n\n

3. Residual Policy<\/strong><\/p>\n\n\n\n

A small residual network is trained on top of the frozen diffusion model to adjust the actions slightly. It is lightweight and easy to apply, but typically provides limited performance gains.<\/p>\n\n\n\n

Overall<\/strong>, existing methods are either computationally expensive<\/em>, too slow<\/em>, or only mildly effective<\/em>.<\/p>\n\n\n\n

This gap motivates our approach \u2014 aiming for a simple, stable, and sample-efficient RL finetuning method<\/strong> for diffusion-based driving models.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Diffusion Models for Action Generation Diffusion models have recently achieved impressive results in action generation, spanning both robotic manipulation and autonomous driving. On the left, Diffusion Policy demonstrates one of the earliest successes: it reformulates control as a denoising process, generating smooth and consistent action sequences directly from image observations. By iteratively refining noisy action […]<\/p>\n","protected":false},"author":245,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-12","page","type-page","status-publish","hentry"],"yoast_head":"\nIntroduction - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\" \/>\n<meta property=\"og:description\" content=\"Diffusion Models for Action Generation Diffusion models have recently achieved impressive results in action generation, spanning both robotic manipulation and autonomous driving. On the left, Diffusion Policy demonstrates one of the earliest successes: it reformulates control as a denoising process, generating smooth and consistent action sequences directly from image observations. By iteratively refining noisy action […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/\" \/>\n<meta property=\"og:site_name\" content=\"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-12T01:29:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i.imgur.com\/ZX9b7Y2.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/\",\"name\":\"Introduction - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i.imgur.com\\\/ZX9b7Y2.png\",\"datePublished\":\"2025-12-11T06:36:23+00:00\",\"dateModified\":\"2025-12-12T01:29:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i.imgur.com\\\/ZX9b7Y2.png\",\"contentUrl\":\"https:\\\/\\\/i.imgur.com\\\/ZX9b7Y2.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/overview\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introduction\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/\",\"name\":\"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/","og_locale":"en_US","og_type":"article","og_title":"Introduction - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","og_description":"Diffusion Models for Action Generation Diffusion models have recently achieved impressive results in action generation, spanning both robotic manipulation and autonomous driving. On the left, Diffusion Policy demonstrates one of the earliest successes: it reformulates control as a denoising process, generating smooth and consistent action sequences directly from image observations. By iteratively refining noisy action […]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/","og_site_name":"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","article_modified_time":"2025-12-12T01:29:50+00:00","og_image":[{"url":"https:\/\/i.imgur.com\/ZX9b7Y2.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/","name":"Introduction - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/#primaryimage"},"thumbnailUrl":"https:\/\/i.imgur.com\/ZX9b7Y2.png","datePublished":"2025-12-11T06:36:23+00:00","dateModified":"2025-12-12T01:29:50+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/#primaryimage","url":"https:\/\/i.imgur.com\/ZX9b7Y2.png","contentUrl":"https:\/\/i.imgur.com\/ZX9b7Y2.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/overview\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/"},{"@type":"ListItem","position":2,"name":"Introduction"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/","name":"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/users\/245"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":21,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages\/12\/revisions"}],"predecessor-version":[{"id":136,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages\/12\/revisions\/136"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/media?parent=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}