{"id":2,"date":"2020-05-05T16:51:39","date_gmt":"2020-05-05T16:51:39","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/?page_id=2"},"modified":"2020-12-21T16:52:15","modified_gmt":"2020-12-21T16:52:15","slug":"sample-page","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","title":{"rendered":"Project overview"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\"><strong>Motivation<\/strong><\/h3>\n\n\n\n<p>The current performance of self driving cars is very good. They know all the<br> rules of the road, have a basic ability to recognize and understand the other actors on the road, and can drive in a way broadly similar to human drivers. However, they are not yet ready for the safety driver to be removed in all but the most restricted domains. Moreover, any deployment to new cities\/countries requires tremendous engineering effort which somewhat defeats the purpose of having an autonomous car.<\/p>\n\n\n\n<p>Our three main motivations\/goals are as follows<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>End-to-end system<\/strong> : An end-to-end system is easier to redeploy and fine-tune<\/li><li><strong>Exceeding the expert performance<\/strong> : Imitation learning from data can only take us so far. If we know the underlying reward function of the world, we could beat the expert and achieve better than human performance.<\/li><li><strong>Verifiable performance<\/strong> : Any algorithm we develop can be rigorously tested in simulation to ensure that it works in rare circumstances.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Problem Statement<\/strong><\/h3>\n\n\n\n<p>Our problem statement is to train an autonomous driving agent in the CARLA[1] simulator using reinforcement learning. We want to develop an off-policy algorithm that can be run on Argo&#8217;s self driving logs. Moreover, we want to aim for sample efficiency in our algorithm.<\/p>\n\n\n\n<p>For inputs and outputs to our network, we want to only use RGB images with waypoints to learn low level control of the vehicle. We build upon the work of <a rel=\"noreferrer noopener\" href=\"https:\/\/www.google.com\/url?q=https%3A%2F%2Fagarwaltanmay.github.io%2F&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNG_qdYtFHSElIcsTsyy2DXKPFN-qQ\" target=\"_blank\">Tanmay Agarwal<\/a> and <a rel=\"noreferrer noopener\" href=\"https:\/\/www.google.com\/url?q=https%3A%2F%2Fhitesh11.github.io%2F&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNHfYm5nuMtCirS2rRrxpvV0GmC1tg\" target=\"_blank\">Hitesh Arora<\/a>, showcased on their <a href=\"https:\/\/sites.google.com\/view\/rl4ad\/\">website<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Approach<\/strong><\/h3>\n\n\n\n<p>We propose the use of a low dimensional engineered state space and reward function to train an RL &#8220;expert&#8221; that can drive well using privileged information.  Note that in we could also use imitation to train the expert and inverse reinforcement learning to get the reward formulation. Next, we transfer the knowledge of this expert agent to an RGB image based policy as describe in Learning by Cheating[2] by Chen et al. to get a feature extractor. Finally, we freeze the convolution layers, reset the fully connected layers and train on the original reward function using a modified n-step soft actor critic algorithm.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Technical details<\/strong><\/h3>\n\n\n\n<p>For details on our approach please read the <a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/sample-page\/spring-2020\/\">Spring 2020 <\/a>and <a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/fall-2020\/\">Fall 2020<\/a> pages. Please visit the <a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/video-demonstrations\/\">Video Demonstration<\/a> page for our qualitative results. You could also watch our final presentation.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Motivation The current performance of self driving cars is very good. They know all the rules of the road, have a basic ability to recognize and understand the other actors on the road, and can drive in a way broadly similar to human drivers. However, they are not yet ready for the safety driver to &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Project overview&#8221;<\/span><\/a><\/p>\n","protected":false},"author":68,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Project overview - Reinforcement Learning for Self Driving Cars<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project overview - Reinforcement Learning for Self Driving Cars\" \/>\n<meta property=\"og:description\" content=\"Motivation The current performance of self driving cars is very good. They know all the rules of the road, have a basic ability to recognize and understand the other actors on the road, and can drive in a way broadly similar to human drivers. However, they are not yet ready for the safety driver to &hellip; Continue reading &quot;Project overview&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/\" \/>\n<meta property=\"og:site_name\" content=\"Reinforcement Learning for Self Driving Cars\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-21T16:52:15+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\",\"name\":\"Project overview - Reinforcement Learning for Self Driving Cars\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/#website\"},\"datePublished\":\"2020-05-05T16:51:39+00:00\",\"dateModified\":\"2020-12-21T16:52:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/\",\"name\":\"Reinforcement Learning for Self Driving Cars\",\"description\":\"Mayank Gupta and Vinay Sameer Kadi, with Prof. Jeff Schneider. Sponsored by Argo AI.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamd\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project overview - Reinforcement Learning for Self Driving Cars","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","og_locale":"en_US","og_type":"article","og_title":"Project overview - Reinforcement Learning for Self Driving Cars","og_description":"Motivation The current performance of self driving cars is very good. They know all the rules of the road, have a basic ability to recognize and understand the other actors on the road, and can drive in a way broadly similar to human drivers. However, they are not yet ready for the safety driver to &hellip; Continue reading \"Project overview\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","og_site_name":"Reinforcement Learning for Self Driving Cars","article_modified_time":"2020-12-21T16:52:15+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","name":"Project overview - Reinforcement Learning for Self Driving Cars","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/#website"},"datePublished":"2020-05-05T16:51:39+00:00","dateModified":"2020-12-21T16:52:15+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/"},{"@type":"ListItem","position":2,"name":"Project overview"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/","name":"Reinforcement Learning for Self Driving Cars","description":"Mayank Gupta and Vinay Sameer Kadi, with Prof. Jeff Schneider. Sponsored by Argo AI.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/users\/68"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/comments?post=2"}],"version-history":[{"count":13,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":214,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/pages\/2\/revisions\/214"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamd\/wp-json\/wp\/v2\/media?parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}