{"id":63,"date":"2020-12-15T20:14:30","date_gmt":"2020-12-15T20:14:30","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/?page_id=63"},"modified":"2020-12-16T20:48:35","modified_gmt":"2020-12-16T20:48:35","slug":"video-demonstrations","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/","title":{"rendered":"Video Demonstrations"},"content":{"rendered":"\n<p class=\"has-normal-font-size\">All of the following video demonstrations were done on the no crash benchmark (dense) on Town 2. We primarily compare the performance of the SAC finetuned agent to the DAgger based agent. Both these agents were trained on Town 1.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Yellow lights : Deciding to stop or pass<\/h4>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<figure class=\"wp-block-embed-youtube alignright wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/youtu.be\/NPzpLOPuHIg\n<\/div><figcaption>Video 1 : SAC finetuned agent stops for yellow light<\/figcaption><\/figure>\n<\/div><\/div>\n\n\n\n<p class=\"has-drop-cap\">Our initial RL expert agent was trained on eight dimensional features. Only one of these eight features describe the traffic light&#8217;s state. The feature specifies the distance to red light if it is near enough. If the light is yellow or green or too far, the feature is set to 1.  Thus, we&#8217;d expect our final agent to not know anything about yellow lights. However, we see in video 1, at around <a href=\"https:\/\/youtu.be\/NPzpLOPuHIg?t=35\">0:40<\/a>, that the car comes to a complete stop for a yellow light. <\/p>\n\n\n\n<p><\/p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<figure class=\"wp-block-embed-youtube alignright wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/youtu.be\/Q2StymXj7pk\n<\/div><figcaption>Video 2 : The same agent rushes though yellow light at<br>another intersection.<\/figcaption><\/figure>\n\n\n\n<p>In contrast, in video 2, around <a href=\"https:\/\/youtu.be\/Q2StymXj7pk?t=25\">0:25<\/a>, we see the opposite; the car speeds through the yellow light to take a left turn.  This is very interesting learned behavior exhibited by the SAC fine-tuned model. Since the RL agent seeks to maximize it&#8217;s reward while avoiding penalties for red light violations, it learns this behavior. If we only did behavior cloning we&#8217;d expect our agent to never stop for yellow lights as it would only notice red lights, which were part of the engineered state space.<\/p>\n<\/div><\/div>\n\n\n\n<h4 class=\"wp-block-heading\">Improving over the purely behavior cloned policy<\/h4>\n\n\n\n<figure class=\"wp-block-embed-youtube alignright wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/www.youtube.com\/watch?v=9ZjNcZxMRHY&#038;feature=youtu.be\n<\/div><figcaption>Video 3 : The behavior cloned agent is less stable<br>than the SAC finetuned agent<\/figcaption><\/figure>\n\n\n\n<p class=\"has-drop-cap\">The SAC finetuned agent outperforms the DAgger agent by 24 points in the no-crash dense benchmark. This is also reflected qualitatively in the results. We see that the behavior cloned agent often crashes into vehicles or other objects. This is demonstrated by the short video number 3 where the car initial recognizes another car and stops for it but then slowly edges forward to crash into it. We see that the finetuned agent drives much more smoothly and crashes less often.<\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube alignright wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/www.youtube.com\/watch?v=TtowklPf1vo&#038;list=PL1_aDDXU1D-b0yacDrUYcD-b1xoCvnaAa&#038;index=17\n<\/div><figcaption>Video 3 : The SAC finetuned agent performs better<\/figcaption><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Failure Case: Representation Error<\/h4>\n\n\n\n<figure class=\"wp-block-embed-youtube alignright wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/youtu.be\/64fjCyXOxC4\n<\/div><figcaption>Video 3: Behavior clones agent stops sometimes<\/figcaption><\/figure>\n\n\n\n<p class=\"has-drop-cap\">We noticed that sometimes our behavior cloned agent stops for no reason. This behavior was observed in 1 out of 25 test scenarios. We think that the car stops because it sees red lights\/other cars where there aren&#8217;t. Even so, it eventually manages to complete the episode.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube alignright wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/youtu.be\/aY-yw25DPrM\n<\/div><figcaption>Video 4: SAC finetuned agent inherits mistakes as<br>conv layers are frozen.<\/figcaption><\/figure>\n\n\n\n<p> Since we are freezing the conv layers for our SAC fine-tuning as of now,  we also see this behavior in the SAC finetuned agent. This agent also eventually completes its episode.<\/p>\n\n\n\n<p>This is not a major concern as of now because this can probably be fixed by unfreezing the conv layers and finetuning further.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">All 25 test episodes : YouTube Playlist<\/h4>\n\n\n\n<p>Finally, given below is a playlist with all the 25 episodes from the best seed of the no crash dense benchmark on the SAC finetuned agent. There is only one failure (ep 24) which happens due to gridlock.<\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube aligncenter wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\nhttps:\/\/youtube.com\/playlist?list=PL1_aDDXU1D-b0yacDrUYcD-b1xoCvnaAa\n<\/div><figcaption>YouTube playlist with the 25 test episodes of our agent on the no crash dense benchmark<\/figcaption><\/figure>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>All of the following video demonstrations were done on the no crash benchmark (dense) on Town 2. We primarily compare the performance of the SAC finetuned agent to the DAgger based agent. Both these agents were trained on Town 1. Yellow lights : Deciding to stop or pass Our initial RL expert agent was trained &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Video Demonstrations&#8221;<\/span><\/a><\/p>\n","protected":false},"author":77,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-63","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Video Demonstrations - Reinforcement Learning for Self Driving Cars<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Video Demonstrations - Reinforcement Learning for Self Driving Cars\" \/>\n<meta property=\"og:description\" content=\"All of the following video demonstrations were done on the no crash benchmark (dense) on Town 2. We primarily compare the performance of the SAC finetuned agent to the DAgger based agent. Both these agents were trained on Town 1. Yellow lights : Deciding to stop or pass Our initial RL expert agent was trained &hellip; Continue reading &quot;Video Demonstrations&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/\" \/>\n<meta property=\"og:site_name\" content=\"Reinforcement Learning for Self Driving Cars\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-16T20:48:35+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/video-demonstrations\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/video-demonstrations\\\/\",\"name\":\"Video Demonstrations - Reinforcement Learning for Self Driving Cars\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/#website\"},\"datePublished\":\"2020-12-15T20:14:30+00:00\",\"dateModified\":\"2020-12-16T20:48:35+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/video-demonstrations\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/video-demonstrations\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/video-demonstrations\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Video Demonstrations\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\",\"name\":\"Reinforcement Learning for Self Driving Cars\",\"description\":\"Mayank Gupta and Vinay Sameer Kadi, with Prof. Jeff Schneider. Sponsored by Argo AI.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Video Demonstrations - Reinforcement Learning for Self Driving Cars","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/","og_locale":"en_US","og_type":"article","og_title":"Video Demonstrations - Reinforcement Learning for Self Driving Cars","og_description":"All of the following video demonstrations were done on the no crash benchmark (dense) on Town 2. We primarily compare the performance of the SAC finetuned agent to the DAgger based agent. Both these agents were trained on Town 1. Yellow lights : Deciding to stop or pass Our initial RL expert agent was trained &hellip; Continue reading \"Video Demonstrations\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/","og_site_name":"Reinforcement Learning for Self Driving Cars","article_modified_time":"2020-12-16T20:48:35+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/","name":"Video Demonstrations - Reinforcement Learning for Self Driving Cars","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/#website"},"datePublished":"2020-12-15T20:14:30+00:00","dateModified":"2020-12-16T20:48:35+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/"},{"@type":"ListItem","position":2,"name":"Video Demonstrations"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","name":"Reinforcement Learning for Self Driving Cars","description":"Mayank Gupta and Vinay Sameer Kadi, with Prof. Jeff Schneider. Sponsored by Argo AI.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages\/63","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/users\/77"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/comments?post=63"}],"version-history":[{"count":12,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages\/63\/revisions"}],"predecessor-version":[{"id":184,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages\/63\/revisions\/184"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/media?parent=63"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}