{"id":1,"date":"2020-05-05T16:55:34","date_gmt":"2020-05-05T16:55:34","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/?p=1"},"modified":"2020-12-17T01:22:33","modified_gmt":"2020-12-17T06:22:33","slug":"mainpage","status":"publish","type":"post","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/","title":{"rendered":"Project Overview"},"content":{"rendered":"\n<p>This is a capstone project with Aurora Innovation. Our advisor is <a href=\"http:\/\/www.cs.cmu.edu\/~vramakri\/\">Varun Ramakrishna<\/a> from Aurora and <a href=\"http:\/\/www.cs.cmu.edu\/~kkitani\/\">Kris Kitani<\/a> from CMU.<\/p>\n\n\n\n<p>Enabling the perception system to detect objects in 3D space is an important problem in autonomous driving. Existing methods can be categorized as camera-only, LiDAR-only, and multi-sensor methods. For camera-only methods, depth estimation from images struggles to compete with the precision of LiDAR points  For LiDAR only methods, it is hard to detect long-range objects since the point cloud gets sparser in remote regions. Therefore, it is intuitive to leverage multi-modal information from different sensors. Interestingly, existing multi-modal methods for 3D object detection usually fail to beat lidar only methods. This reveals that more efforts should be devoted to seeking an effective way to fuse different sensor information for 3D object detection.<\/p>\n\n\n\n<p>This project starts with a comprehensive analysis of existing 3D object detection methods. The patterns of common failure cases can be extremely informative and enlightening.  After this, the project aims to explore new sensor fusion methods in pursuit of solid improvement upon current popular methods.<\/p>\n\n\n\n<p>In the spring semester, our team members focus on reimplementing and analyzing state-of-the-art methods in multimodal 3D object detection, including AVOD, PointPillar, PointPainting and conduct experiments in nuScenes and KITTI dataset.<\/p>\n\n\n\n<p>We start our experiment by first analyzing the AVOD, which is a common benchmark method that uses both image and LiDAR information. We realize that for super close objects (&lt;10 meters), the AVOD performs surprisingly bad, and we argue that the main reason is the misalignment between frontal ROI Pooling and BEV ROI Pooling. Here is an illustrative figure:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg\" alt=\"\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-1024x532.jpg\" alt=\"\" \/><\/figure>\n\n\n\n<p>We also conduct experiments when we have perfect semantic segmentation predictions in PointPainting(PointPainting relies heavily on 2D image segmentation). Our experiment shows that the performance of 3D object detection is highly correlated to 2D image segmentation. Here is the corresponding slides:<br><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-1024x547.jpg\" alt=\"\" \/><\/figure>\n\n\n\n<p>Given this finding, we propose to improve 2D image segmentation given 3D object detection result. If that is also true, we could close the loop and let both tasks benefit each other. Here is the illustrative figure:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1-1024x533.jpg\" alt=\"\" \/><\/figure>\n\n\n\n<p>We propose to address two issues simultaneously: (1) fuse 3D information into 2D segmentation (2) sequential inference by cascading the detection model. We highlight the main process in the following two figures:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-1-1024x530.jpg\" alt=\"\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-1-1024x532.jpg\" alt=\"\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>This is a capstone project with Aurora Innovation. Our advisor is Varun Ramakrishna from Aurora and Kris Kitani from CMU. Enabling the perception system to detect objects in 3D space is an important problem in autonomous driving. Existing methods can be categorized as camera-only, LiDAR-only, and multi-sensor methods. For camera-only methods, depth estimation from images &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Project Overview&#8221;<\/span><\/a><\/p>\n","protected":false},"author":68,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-1","post","type-post","status-publish","format-standard","hentry","category-main-page"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Project Overview - Multimodal 3D Object Detection<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project Overview - Multimodal 3D Object Detection\" \/>\n<meta property=\"og:description\" content=\"This is a capstone project with Aurora Innovation. Our advisor is Varun Ramakrishna from Aurora and Kris Kitani from CMU. Enabling the perception system to detect objects in 3D space is an important problem in autonomous driving. Existing methods can be categorized as camera-only, LiDAR-only, and multi-sensor methods. For camera-only methods, depth estimation from images &hellip; Continue reading &quot;Project Overview&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/\" \/>\n<meta property=\"og:site_name\" content=\"Multimodal 3D Object Detection\" \/>\n<meta property=\"article:published_time\" content=\"2020-05-05T16:55:34+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-17T06:22:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg\" \/>\n<meta name=\"author\" content=\"bstaszel\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"bstaszel\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/\"},\"author\":{\"name\":\"bstaszel\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#\\\/schema\\\/person\\\/ca4fef96ddb313b08f220f40a67d6cc8\"},\"headline\":\"Project Overview\",\"datePublished\":\"2020-05-05T16:55:34+00:00\",\"dateModified\":\"2020-12-17T06:22:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/\"},\"wordCount\":378,\"commentCount\":1,\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1-1024x531.jpg\",\"articleSection\":[\"Main Page\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/\",\"name\":\"Project Overview - Multimodal 3D Object Detection\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1-1024x531.jpg\",\"datePublished\":\"2020-05-05T16:55:34+00:00\",\"dateModified\":\"2020-12-17T06:22:33+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#\\\/schema\\\/person\\\/ca4fef96ddb313b08f220f40a67d6cc8\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1.jpg\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1.jpg\",\"width\":2054,\"height\":1066},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/2020\\\/05\\\/05\\\/mainpage\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project Overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/\",\"name\":\"Multimodal 3D Object Detection\",\"description\":\"Advised by Kris Kitani and Varun Ramakrishna | Sponsored by Aurora Innovation\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#\\\/schema\\\/person\\\/ca4fef96ddb313b08f220f40a67d6cc8\",\"name\":\"bstaszel\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/55ba2f351a4bd0441f833514aeb22ff1a25b508019bb426d198559e564ed9408?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/55ba2f351a4bd0441f833514aeb22ff1a25b508019bb426d198559e564ed9408?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/55ba2f351a4bd0441f833514aeb22ff1a25b508019bb426d198559e564ed9408?s=96&d=mm&r=g\",\"caption\":\"bstaszel\"},\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/author\\\/bstaszel\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project Overview - Multimodal 3D Object Detection","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/","og_locale":"en_US","og_type":"article","og_title":"Project Overview - Multimodal 3D Object Detection","og_description":"This is a capstone project with Aurora Innovation. Our advisor is Varun Ramakrishna from Aurora and Kris Kitani from CMU. Enabling the perception system to detect objects in 3D space is an important problem in autonomous driving. Existing methods can be categorized as camera-only, LiDAR-only, and multi-sensor methods. For camera-only methods, depth estimation from images &hellip; Continue reading \"Project Overview\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/","og_site_name":"Multimodal 3D Object Detection","article_published_time":"2020-05-05T16:55:34+00:00","article_modified_time":"2020-12-17T06:22:33+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg","type":"","width":"","height":""}],"author":"bstaszel","twitter_card":"summary_large_image","twitter_misc":{"Written by":"bstaszel","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#article","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/"},"author":{"name":"bstaszel","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#\/schema\/person\/ca4fef96ddb313b08f220f40a67d6cc8"},"headline":"Project Overview","datePublished":"2020-05-05T16:55:34+00:00","dateModified":"2020-12-17T06:22:33+00:00","mainEntityOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/"},"wordCount":378,"commentCount":1,"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg","articleSection":["Main Page"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/","name":"Project Overview - Multimodal 3D Object Detection","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg","datePublished":"2020-05-05T16:55:34+00:00","dateModified":"2020-12-17T06:22:33+00:00","author":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#\/schema\/person\/ca4fef96ddb313b08f220f40a67d6cc8"},"breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1.jpg","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1.jpg","width":2054,"height":1066},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/2020\/05\/05\/mainpage\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/"},{"@type":"ListItem","position":2,"name":"Project Overview"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/","name":"Multimodal 3D Object Detection","description":"Advised by Kris Kitani and Varun Ramakrishna | Sponsored by Aurora Innovation","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#\/schema\/person\/ca4fef96ddb313b08f220f40a67d6cc8","name":"bstaszel","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/55ba2f351a4bd0441f833514aeb22ff1a25b508019bb426d198559e564ed9408?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/55ba2f351a4bd0441f833514aeb22ff1a25b508019bb426d198559e564ed9408?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/55ba2f351a4bd0441f833514aeb22ff1a25b508019bb426d198559e564ed9408?s=96&d=mm&r=g","caption":"bstaszel"},"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/author\/bstaszel\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/posts\/1","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/users\/68"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/comments?post=1"}],"version-history":[{"count":22,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/posts\/1\/revisions"}],"predecessor-version":[{"id":167,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/posts\/1\/revisions\/167"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/media?parent=1"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/categories?post=1"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/tags?post=1"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}