{"id":12,"date":"2020-05-11T21:38:25","date_gmt":"2020-05-11T21:38:25","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teame\/?page_id=12"},"modified":"2020-12-16T23:02:33","modified_gmt":"2020-12-16T23:02:33","slug":"home","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/","title":{"rendered":"Overview"},"content":{"rendered":"\n

Self-supervised 3D object detection<\/strong><\/p>\n\n\n\n

Object detection in lidar point cloud data is a fundamental component of autonomous driving. As annotation for 3D tasks is hard to obtain, currently publicly available datasets have few hours of data which is arguably not sufficient to train a reliable object detector. At the same time we have a large amount of unannotated data available. The goal of this project is to use self-supervised methods to train or improve the classification and regression branch of an existing object detector. Specifically, for classification branch, we are using the idea that points are more likely to belong to the same object if they are close by and are moving together; they are less likely to belong to the same object if they are farther apart and moving separately. For the regression branch to estimate the orientation, we use the complementary motion information provided by the scene flow output by a self-supervised scene flow model<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Advisor: Prof. David Held <\/a><\/p>\n\n\n\n

Sponsor: Argo AI<\/a><\/p>\n\n\n\n

<\/p>\n\n\n\n

Project Description and Methods<\/h2>\n\n\n\n

Methods for the classification branch<\/h3>\n\n\n\n

Clustering to generate pseudo ground truth<\/h4>\n\n\n\n

We cluster the points using its xyz coordinates as well as the scene flow vectors (treated as the motion of a point). By particularly tuning the parameters, we can get a better clustering result than just using the xyz coordinates (say now we can deal with the case when two objects are near each other but have different motion, as the figure below shows).<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Evaluate a metric for each given anchor<\/h4>\n\n\n\n

We think that if a group of points moving together, then they tend to belong to the same object. This means that we should penalize the case when the sceneflow vectors in a given anchor box are not consistent. The concept is illustrated in the figure below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Based on this philosophical idea, we propse a variance metric to evaluate the variance of motion vectors (scene flow vectors) of points within a given anchor box, as shown in the figure below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

We validate this metric on some hand-manipulated cases. In the figure blow, we choose a moving ground truth bbox that is close to static bbox. We double its size and make a shift for its center location so that after this manipulation, this anchor box also include the points in another static bbox. We evaluate the variance metric for this case before and after manipulation, and find that the variance metric after is higher than before, which validates our idea.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

After we have this metric for each anchor box, we can train the classification branch either using policy gradient methods or using some representation learning methods (although we switch gears to another direction later and did not finish this part).<\/p>\n\n\n\n

Methods for the regression branch<\/h3>\n\n\n\n

The idea to improve the orientation estimation is embarrassingly simple. We have an initial estimation that is regressed by a 3D object detector say CenterPoint. We have an estimation from the scene flow (the average of the flow vector). At test time, if we are confident about the estimation from scene flow, we directly replace the initial estimation with this one. The concept figure is as below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

For evaluation metrics, we use the Average Orientation Error, one of the True Positive metrics in nuScenes dataset, since the mAP metric in nuScenes does not account for the orientation, and only rely on the center of a predicted bbox.<\/p>\n\n\n\n

In order to filter out the “reliable” scene flow vector, we use several filters: we get rid of the case when there are few points in a bbox, when it is a static object (the norm of average flow vector is small), when the variance of the scene flow vectors within a bbox is large.<\/p>\n\n\n\n

In the end, we get an improvement on the orientation on the remaining bboxs, as shown in the table below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Self-supervised Shape completion<\/strong><\/p>\n\n\n\n

When navigating in urban environments, many of the objects that we need to track and avoid are heavily occluded. Planning and tracking using these partial scans can be quite inaccurate. The aim of this work is to complete these partial point clouds, giving us a full understanding of the object\u2019s geometry using only partial and noisy observations. Previous methods achieve this with the help of complete, ground-truth annotations of the target objects, which are available only for simulated datasets. However, such ground truth is unavailable for diverse, real-world LiDAR data. In this work, we present a self-supervised point cloud completion algorithm, which is trained only on partial scans without assuming access to complete, ground-truth object model annotations. Our method achieves this via inpainting.
This work was done in collaboration with Himangi Mittal<\/a> and Brian Okorn<\/a> at RPAD Lab, CMU. <\/p>\n\n\n\n

Architecture<\/strong>
<\/p>\n\n\n\n

\"\"
Figure: The network takes a canonicalized input partial point cloud which is subjected to artificial occlusions through region dropout. The multi-scale encoder aggregates the local and global features of this cloud into a combined encoding P. Our multi-scale decoder uses this encoding to generate a completed point cloud, supervised at both the global and local levels using only partial point clouds<\/em><\/figcaption><\/figure>\n\n\n\n

In summary:
1. A novel inpainting-based self-supervised algorithm that enables the network to learn to fill missing local regions in an incomplete point cloud without the need for a ground-truth complete point cloud.
2. Multi-scale encoder-decoder based architecture, partitions the canonicalized point clouds to learn a combination of local and global shape priors to reduce overfitting and better generalize to unseen point clouds.
3. The approach outperforms existing methods for self-supervised point cloud completion when evaluated on the ShapeNet and SemanticKITTI datasets.<\/p>\n\n\n\n


<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Self-supervised 3D object detection Object detection in lidar point cloud data is a fundamental component of autonomous driving. As annotation for 3D tasks is hard to obtain, currently publicly available datasets have few hours of data which is arguably not sufficient to train a reliable object detector. At the same time we have a large … <\/p>\n

Continue reading “Overview”<\/span><\/a><\/p>\n","protected":false},"author":78,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-12","page","type-page","status-publish","hentry"],"yoast_head":"\nOverview - Self-supervised 3D Object Detection and Shape Completion<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Overview - Self-supervised 3D Object Detection and Shape Completion\" \/>\n<meta property=\"og:description\" content=\"Self-supervised 3D object detection Object detection in lidar point cloud data is a fundamental component of autonomous driving. As annotation for 3D tasks is hard to obtain, currently publicly available datasets have few hours of data which is arguably not sufficient to train a reliable object detector. At the same time we have a large … Continue reading "Overview"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/\" \/>\n<meta property=\"og:site_name\" content=\"Self-supervised 3D Object Detection and Shape Completion\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-16T23:02:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-content\/uploads\/sites\/35\/2020\/05\/Screen-Shot-2020-04-30-at-9.05.41-PM.png\" \/>\n\t<meta property=\"og:image:width\" content=\"401\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/\",\"name\":\"Overview - Self-supervised 3D Object Detection and Shape Completion\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/wp-content\\\/uploads\\\/sites\\\/35\\\/2020\\\/05\\\/Screen-Shot-2020-04-30-at-9.05.41-PM.png\",\"datePublished\":\"2020-05-11T21:38:25+00:00\",\"dateModified\":\"2020-12-16T23:02:33+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/wp-content\\\/uploads\\\/sites\\\/35\\\/2020\\\/05\\\/Screen-Shot-2020-04-30-at-9.05.41-PM.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/wp-content\\\/uploads\\\/sites\\\/35\\\/2020\\\/05\\\/Screen-Shot-2020-04-30-at-9.05.41-PM.png\",\"width\":401,\"height\":420},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/\",\"name\":\"Self-supervised 3D Object Detection and Shape Completion\",\"description\":\"Advised by: Prof David Held | Sponsered by: Argo AI\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teame\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Overview - Self-supervised 3D Object Detection and Shape Completion","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/","og_locale":"en_US","og_type":"article","og_title":"Overview - Self-supervised 3D Object Detection and Shape Completion","og_description":"Self-supervised 3D object detection Object detection in lidar point cloud data is a fundamental component of autonomous driving. As annotation for 3D tasks is hard to obtain, currently publicly available datasets have few hours of data which is arguably not sufficient to train a reliable object detector. At the same time we have a large … Continue reading \"Overview\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/","og_site_name":"Self-supervised 3D Object Detection and Shape Completion","article_modified_time":"2020-12-16T23:02:33+00:00","og_image":[{"width":401,"height":420,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-content\/uploads\/sites\/35\/2020\/05\/Screen-Shot-2020-04-30-at-9.05.41-PM.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/","name":"Overview - Self-supervised 3D Object Detection and Shape Completion","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-content\/uploads\/sites\/35\/2020\/05\/Screen-Shot-2020-04-30-at-9.05.41-PM.png","datePublished":"2020-05-11T21:38:25+00:00","dateModified":"2020-12-16T23:02:33+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-content\/uploads\/sites\/35\/2020\/05\/Screen-Shot-2020-04-30-at-9.05.41-PM.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-content\/uploads\/sites\/35\/2020\/05\/Screen-Shot-2020-04-30-at-9.05.41-PM.png","width":401,"height":420},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/"},{"@type":"ListItem","position":2,"name":"Overview"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/","name":"Self-supervised 3D Object Detection and Shape Completion","description":"Advised by: Prof David Held | Sponsered by: Argo AI","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/users\/78"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":11,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/12\/revisions"}],"predecessor-version":[{"id":70,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/12\/revisions\/70"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/media?parent=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}