{"id":70,"date":"2025-12-11T19:06:04","date_gmt":"2025-12-11T19:06:04","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/?page_id=70"},"modified":"2025-12-12T01:26:50","modified_gmt":"2025-12-12T01:26:50","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\">Learning to Choose noise = Control the driving style<\/h3>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/3GPXO6s.png\" alt=\"\" \/><figcaption class=\"wp-element-caption\">DSRL[1]<\/figcaption><\/figure>\n\n\n\n<p>Diffusion models are strong generative policies, but directly fine-tuning them is expensive and unstable. Instead of modifying diffusion weights, we take advantage of a key observation:<\/p>\n\n\n\n<p><strong>the latent noise space is much larger and more expressive than the action space.<\/strong><\/p>\n\n\n\n<p>Each noise sample <em>w<\/em> represents a different latent behavior mode\u2014some leading to smoother, conservative trajectories, others producing sharper or more aggressive turns.<\/p>\n\n\n\n<p>This means that <strong>choosing the noise is equivalent to choosing the driving style.<\/strong><\/p>\n\n\n\n<p>Our method learns a lightweight policy \u03c0_w that selects the noise before the diffusion model generates actions. By operating entirely in this latent-noise space:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>No nested backprop through the diffusion process<\/strong><\/li>\n\n\n\n<li><strong>No modification to the diffusion model weights<\/strong><\/li>\n\n\n\n<li><strong>Stable and efficient RL training<\/strong><\/li>\n\n\n\n<li><strong>Full control over driving behavior through noise selection<\/strong><\/li>\n<\/ul>\n\n\n\n<p>In other words, we reformulate reinforcement learning from <em>action-space optimization<\/em> to <em>latent-noise steering<\/em>. This allows us to adapt driving styles flexibly\u2014conservative, neutral, or aggressive\u2014while keeping the diffusion model completely frozen.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Diffusion Policy Finetuning via Latent Noise RL Steering<\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/i.imgur.com\/8aqsqGB.png\" alt=\"\" \/><\/figure>\n\n\n\n<p>Our method builds on <strong>DiffusionDrive<\/strong>[4] and applies <strong>DSRL<\/strong>[1] that <em>steers the latent noise<\/em> instead of modifying the diffusion model itself. This design enables fine-grained control over driving behavior while keeping training stable, efficient, and entirely offline.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>1. Observation Encoding<\/strong><\/h3>\n\n\n\n<p>The system first extracts <strong>Bird\u2019s-Eye-View (BEV) features<\/strong> from multi-camera images, along with <strong>3 seconds of history information<\/strong>.<\/p>\n\n\n\n<p>These features summarize the scene geometry, surrounding agents, and past motion, and are used as input to both the diffusion policy and the latent-noise actor.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>2. Latent Noise Steering via DSRL<\/strong><\/h3>\n\n\n\n<p>Instead of optimizing actions directly, we introduce a policy \u03c0_w that learns to select <strong>latent noise samples<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Each noise vector w corresponds to a different latent behavior mode (e.g., conservative, neutral, aggressive).<\/li>\n\n\n\n<li>The diffusion model interprets this noise and generates an action trajectory accordingly.<\/li>\n<\/ul>\n\n\n\n<p>During training:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The <strong>latent-noise actor<\/strong> maximizes the value of the selected noise:<\/li>\n<\/ul>\n\n\n\n<div class=\"wp-block-math\"><math display=\"block\"><semantics><mrow><mi class=\"mathcal\">\u2112<\/mi><msup><mi>\u03c0<\/mi><mi>w<\/mi><\/msup><mo>=<\/mo><mi>\ud835\udd3c<\/mi><mrow><mi>s<\/mi><mo>\u223c<\/mo><mi class=\"mathcal\">\u212c<\/mi><\/mrow><mrow><mo fence=\"true\" form=\"prefix\">[<\/mo><mi>\u03b1<\/mi><mrow><mspace width=\"0.1667em\"><\/mspace><mi>log<\/mi><mo>\u2061<\/mo><mspace width=\"0.1667em\"><\/mspace><\/mrow><msup><mi>\u03c0<\/mi><mi>w<\/mi><\/msup><mo form=\"prefix\" stretchy=\"false\">(<\/mo><mi>w<\/mi><mi>|<\/mi><mi>s<\/mi><mo form=\"postfix\" stretchy=\"false\">)<\/mo><mo>\u2212<\/mo><msup><mi>Q<\/mi><mi>w<\/mi><\/msup><mo form=\"prefix\" stretchy=\"false\">(<\/mo><mi>s<\/mi><mo separator=\"true\">,<\/mo><mi>w<\/mi><mo form=\"postfix\" stretchy=\"false\">)<\/mo><mo fence=\"true\" form=\"postfix\">]<\/mo><\/mrow><\/mrow><annotation encoding=\"application\/x-tex\">\\mathcal{L}{\\pi^w} = \\mathbb{E}{s\\sim\\mathcal{B}}\\left[\\alpha \\log \\pi^w(w|s) &#8211; Q^w(s,w)\\right]<\/annotation><\/semantics><\/math><\/div>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The <strong>action critic <\/strong>Q^A learns the value of diffusion-generated actions: <\/li>\n<\/ul>\n\n\n\n<div class=\"wp-block-math\"><math display=\"block\"><semantics><mrow><msub><mi class=\"mathcal\">\u2112<\/mi><msup><mi>Q<\/mi><mi>A<\/mi><\/msup><\/msub><mo>=<\/mo><mfrac><mn>1<\/mn><mn>2<\/mn><\/mfrac><mrow><munderover><mo movablelimits=\"false\">\u2211<\/mo><mrow><mi>i<\/mi><mo>=<\/mo><mn>1<\/mn><\/mrow><msub><mi>N<\/mi><mi>c<\/mi><\/msub><\/munderover><\/mrow><msub><mi>\ud835\udd3c<\/mi><mi>B<\/mi><\/msub><mrow><mo fence=\"true\" form=\"prefix\">[<\/mo><msup><mrow><mo fence=\"true\" form=\"prefix\">(<\/mo><msubsup><mi>Q<\/mi><msub><mi>\u03b8<\/mi><mi>i<\/mi><\/msub><mi>A<\/mi><\/msubsup><mo form=\"prefix\" stretchy=\"false\">(<\/mo><mi>s<\/mi><mo separator=\"true\">,<\/mo><mi>a<\/mi><mo form=\"postfix\" stretchy=\"false\">)<\/mo><mo>\u2212<\/mo><mi>y<\/mi><mo fence=\"true\" form=\"postfix\">)<\/mo><\/mrow><mn>2<\/mn><\/msup><mo fence=\"true\" form=\"postfix\">]<\/mo><\/mrow><\/mrow><annotation encoding=\"application\/x-tex\">\\mathcal{L}_{Q^A}\n= \\frac{1}{2} \\sum_{i=1}^{N_c} \\mathbb{E}_{B} \\left[ \\left( Q^{A}_{\\theta_i}(s,a) &#8211; y \\right)^2 \\right]<\/annotation><\/semantics><\/math><\/div>\n\n\n\n<p>By performing RL in the <strong>noise space<\/strong>, we avoid backpropagating through the diffusion model and do not modify its weights.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>3. Diffusion Policy Model<\/strong><\/h3>\n\n\n\n<p>A frozen DiffusionDrive model receives:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Observation features<\/li>\n\n\n\n<li>Latent noise selected by \u03c0_w<\/li>\n<\/ul>\n\n\n\n<p>It then generates a full trajectory through its denoising process. Since the diffusion model remains unchanged, our method is:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Compatible with any pretrained diffusion policy<\/strong><\/li>\n\n\n\n<li><strong>Efficient (no nested gradient computation)<\/strong><\/li>\n\n\n\n<li><strong>Stable<\/strong> compared to action-space RL<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>4. Optional VLM Guidance<\/strong><\/h3>\n\n\n\n<p>We are exploring integrating a Vision-Language Model (VLM) to provide <strong>high-level planning cues<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The VLM generates natural-language instructions (via chain-of-thought reasoning).<\/li>\n\n\n\n<li>These instructions may guide the latent-noise policy toward human-preferred behaviors.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>5. Improved Trajectory Quality<\/strong><\/h3>\n\n\n\n<p>Because the noise space contains diverse behavioral modes, steering noise enables:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Smoother turns<\/li>\n\n\n\n<li>Safer lane changes<\/li>\n\n\n\n<li>More consistent trajectories<\/li>\n\n\n\n<li>Better generalization under different driving styles and environments<\/li>\n<\/ul>\n\n\n\n<p>Our experiments confirm that <strong>controlling noise leads to better driving trajectories<\/strong> without retraining or altering the diffusion model.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learning to Choose noise = Control the driving style Diffusion models are strong generative policies, but directly fine-tuning them is expensive and unstable. Instead of modifying diffusion weights, we take advantage of a key observation: the latent noise space is much larger and more expressive than the action space. Each noise sample w represents a [&hellip;]<\/p>\n","protected":false},"author":245,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-70","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Method - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\" \/>\n<meta property=\"og:description\" content=\"Learning to Choose noise = Control the driving style Diffusion models are strong generative policies, but directly fine-tuning them is expensive and unstable. Instead of modifying diffusion weights, we take advantage of a key observation: the latent noise space is much larger and more expressive than the action space. Each noise sample w represents a [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-12T01:26:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i.imgur.com\/3GPXO6s.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/\",\"name\":\"Method - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i.imgur.com\\\/3GPXO6s.png\",\"datePublished\":\"2025-12-11T19:06:04+00:00\",\"dateModified\":\"2025-12-12T01:26:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i.imgur.com\\\/3GPXO6s.png\",\"contentUrl\":\"https:\\\/\\\/i.imgur.com\\\/3GPXO6s.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/\",\"name\":\"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2025fteam24\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","og_description":"Learning to Choose noise = Control the driving style Diffusion models are strong generative policies, but directly fine-tuning them is expensive and unstable. Instead of modifying diffusion weights, we take advantage of a key observation: the latent noise space is much larger and more expressive than the action space. Each noise sample w represents a [&hellip;]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/","og_site_name":"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","article_modified_time":"2025-12-12T01:26:50+00:00","og_image":[{"url":"https:\/\/i.imgur.com\/3GPXO6s.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/","name":"Method - Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/i.imgur.com\/3GPXO6s.png","datePublished":"2025-12-11T19:06:04+00:00","dateModified":"2025-12-12T01:26:50+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/#primaryimage","url":"https:\/\/i.imgur.com\/3GPXO6s.png","contentUrl":"https:\/\/i.imgur.com\/3GPXO6s.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/","name":"Reinforcement Learning for Noise Steering in Diffusion-Based Driving Models","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages\/70","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/users\/245"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/comments?post=70"}],"version-history":[{"count":7,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages\/70\/revisions"}],"predecessor-version":[{"id":132,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/pages\/70\/revisions\/132"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2025fteam24\/wp-json\/wp\/v2\/media?parent=70"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}