{"id":131,"date":"2020-12-15T23:27:45","date_gmt":"2020-12-16T04:27:45","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/?page_id=131"},"modified":"2020-12-17T00:49:02","modified_gmt":"2020-12-17T05:49:02","slug":"spring-semester","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/","title":{"rendered":"Spring 2020"},"content":{"rendered":"\n<p>In the spring semester, our team members focus on reimplementing and analyzing state-of-the-art methods in multimodal 3D object detection, including AVOD, PointPillar, PointPainting and conduct experiments in nuScenes and KITTI dataset. For detailed explanation, please check out our <a href=\"https:\/\/drive.google.com\/file\/d\/1foFX5wf1RM0qddF_b9ybLi1AwjPh_FC9\/view?usp=sharing\">video<\/a> and <a href=\"https:\/\/drive.google.com\/file\/d\/147wZVDH-DJFD5atNmATgWGRX4ca8oIoi\/view?usp=sharing\">slides<\/a>.<\/p>\n\n\n\n<p>We start our experiment by first analyzing the AVOD, which is a common benchmark method that uses both image and LiDAR information. We realize that for super close objects (&lt;10 meters), the AVOD performs surprisingly bad, and we argue that the main reason is the misalignment between frontal ROI Pooling and BEV ROI Pooling. Here is an illustrative figure:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"531\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg\" alt=\"\" class=\"wp-image-148\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-300x156.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-768x399.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1536x797.jpg 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-2048x1063.jpg 2048w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"532\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-1024x532.jpg\" alt=\"\" class=\"wp-image-149\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-1024x532.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-300x156.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-768x399.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2-1536x798.jpg 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im2.jpg 2048w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p>We also conduct experiments when we have perfect semantic segmentation predictions in PointPainting(PointPainting relies heavily on 2D image segmentation). Our experiment shows that the performance of 3D object detection is highly correlated to 2D image segmentation. Here is the corresponding slides:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"547\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-1024x547.jpg\" alt=\"\" class=\"wp-image-150\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-1024x547.jpg 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-300x160.jpg 300w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-768x410.jpg 768w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-1536x820.jpg 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im3-2048x1094.jpg 2048w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p>Given these findings, we decide to focus on the second finding and propose to let 2D model and 3D model benefits each other.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the spring semester, our team members focus on reimplementing and analyzing state-of-the-art methods in multimodal 3D object detection, including AVOD, PointPillar, PointPainting and conduct experiments in nuScenes and KITTI dataset. For detailed explanation, please check out our video and slides. We start our experiment by first analyzing the AVOD, which is a common benchmark &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Spring 2020&#8221;<\/span><\/a><\/p>\n","protected":false},"author":81,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-131","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Spring 2020 - Multimodal 3D Object Detection<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spring 2020 - Multimodal 3D Object Detection\" \/>\n<meta property=\"og:description\" content=\"In the spring semester, our team members focus on reimplementing and analyzing state-of-the-art methods in multimodal 3D object detection, including AVOD, PointPillar, PointPainting and conduct experiments in nuScenes and KITTI dataset. For detailed explanation, please check out our video and slides. We start our experiment by first analyzing the AVOD, which is a common benchmark &hellip; Continue reading &quot;Spring 2020&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/\" \/>\n<meta property=\"og:site_name\" content=\"Multimodal 3D Object Detection\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-17T05:49:02+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2054\" \/>\n\t<meta property=\"og:image:height\" content=\"1066\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/\",\"name\":\"Spring 2020 - Multimodal 3D Object Detection\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1-1024x531.jpg\",\"datePublished\":\"2020-12-16T04:27:45+00:00\",\"dateModified\":\"2020-12-17T05:49:02+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1.jpg\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/wp-content\\\/uploads\\\/sites\\\/36\\\/2020\\\/12\\\/im1.jpg\",\"width\":2054,\"height\":1066},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/spring-semester\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Spring 2020\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/\",\"name\":\"Multimodal 3D Object Detection\",\"description\":\"Advised by Kris Kitani and Varun Ramakrishna | Sponsored by Aurora Innovation\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamf\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spring 2020 - Multimodal 3D Object Detection","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/","og_locale":"en_US","og_type":"article","og_title":"Spring 2020 - Multimodal 3D Object Detection","og_description":"In the spring semester, our team members focus on reimplementing and analyzing state-of-the-art methods in multimodal 3D object detection, including AVOD, PointPillar, PointPainting and conduct experiments in nuScenes and KITTI dataset. For detailed explanation, please check out our video and slides. We start our experiment by first analyzing the AVOD, which is a common benchmark &hellip; Continue reading \"Spring 2020\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/","og_site_name":"Multimodal 3D Object Detection","article_modified_time":"2020-12-17T05:49:02+00:00","og_image":[{"width":2054,"height":1066,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/","name":"Spring 2020 - Multimodal 3D Object Detection","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1-1024x531.jpg","datePublished":"2020-12-16T04:27:45+00:00","dateModified":"2020-12-17T05:49:02+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1.jpg","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-content\/uploads\/sites\/36\/2020\/12\/im1.jpg","width":2054,"height":1066},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/spring-semester\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/"},{"@type":"ListItem","position":2,"name":"Spring 2020"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/","name":"Multimodal 3D Object Detection","description":"Advised by Kris Kitani and Varun Ramakrishna | Sponsored by Aurora Innovation","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/pages\/131","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/comments?post=131"}],"version-history":[{"count":5,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/pages\/131\/revisions"}],"predecessor-version":[{"id":160,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/pages\/131\/revisions\/160"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamf\/wp-json\/wp\/v2\/media?parent=131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}