{"id":147,"date":"2022-04-29T18:25:51","date_gmt":"2022-04-29T18:25:51","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/?page_id=147"},"modified":"2022-04-29T19:25:50","modified_gmt":"2022-04-29T19:25:50","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Overview<\/strong><\/p>\n\n\n\n

We accumulate a point cloud sequence and encode it using a voxel grid-based encoder-decoder backbone.<\/p>\n\n\n\n

After accumulating encoded points in a 3D voxel grid, we (a) down-sample the volume (via sparse 3D convolutions and pooling layers), and flatten the bottleneck layer along the height-axis to obtain BEV representation (followed by 2D convolutional layers, similar to CenterPoint).<\/p>\n\n\n\n

We use this representation to detect objects as (modal) points and regress (modal) offsets for the temporal association.<\/p>\n\n\n\n

Our decoder (b) consists of several up-sampling layers to obtain fine-frained, voxel-level semantic predictions.<\/p>\n\n\n\n

Our instance segmentation network, PointSegMLP (c) performs binary classification within spherical regions of interest (RoI) centered around predicted centers to obtain object instances. <\/p>\n\n\n\n

(d ) PointSegMLP utilizes point and center features as input to produce panoptic segmentation results.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Training<\/strong><\/p>\n\n\n\n

Overall, the network optimizes the following losses:<\/p>\n\n\n\n

\"\"
Segmentation Loss<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Detection Loss<\/figcaption><\/figure>\n\n\n\n
\"\"
Membership Loss<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
Tracking Loss<\/figcaption><\/figure>\n\n\n\n
\"\"
Total Loss<\/figcaption><\/figure><\/div>\n\n\n\n

<\/p>\n\n\n\n

We refer the reader to the presentation for an exact interpretation of all symbols, and to the figure for the function of each of these losses.<\/p>\n\n\n\n

Inference<\/strong><\/p>\n\n\n\n

During inference, we fuse segmentation branch predictions, modal center heatmaps, and point-center memberships to obtain 4D panoptic predictions. Intuitively, we listen to the segmentation labels predicted by the segmentation branch and listen to instance labels predicted by the modal centroid membership branch.<\/p>\n\n\n\n

Using the segmentation branch predictions, we assign point-level predictions to all points within the voxel. Similarly, we apply non-maximum suppression (NMS) over the predicted center heatmaps to generate predicted modal centers. We then compute the membership of each point p within the RoI of each center using PointSegMLP, resolving overlapping RoIs by the most confident center. Next, we assign the predicted center label to all points that are its members and assign a unique instance-id.<\/p>\n\n\n\n

For all stuff points, we directly utilize the predicted semantic label. To extend our method to panoptic tracking, we associate instances across sweeps by using predicted center velocities: we greedily form tracklets by matching previous-sweep centers to current sweep centers with subtracted velocity offsets. Finally, all the instances of an object belonging to a tracklet are assigned a temporally consistent unique ID.<\/p>\n","protected":false},"excerpt":{"rendered":"

Overview We accumulate a point cloud sequence and encode it using a voxel grid-based encoder-decoder backbone. After accumulating encoded points in a 3D voxel grid, we (a) down-sample the volume (via sparse 3D convolutions and pooling layers), and flatten the bottleneck layer along the height-axis to obtain BEV representation (followed by 2D convolutional layers, similar … <\/p>\n

Continue reading “Method”<\/span><\/a><\/p>\n","protected":false},"author":107,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-147","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - Panoptic Lidar Segmentation for Autonomous Driving<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Panoptic Lidar Segmentation for Autonomous Driving\" \/>\n<meta property=\"og:description\" content=\"Overview We accumulate a point cloud sequence and encode it using a voxel grid-based encoder-decoder backbone. After accumulating encoded points in a 3D voxel grid, we (a) down-sample the volume (via sparse 3D convolutions and pooling layers), and flatten the bottleneck layer along the height-axis to obtain BEV representation (followed by 2D convolutional layers, similar … Continue reading "Method"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Panoptic Lidar Segmentation for Autonomous Driving\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-29T19:25:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-content\/uploads\/sites\/50\/2022\/04\/image-1.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/\",\"name\":\"Method - Panoptic Lidar Segmentation for Autonomous Driving\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/wp-content\\\/uploads\\\/sites\\\/50\\\/2022\\\/04\\\/image-1.png\",\"datePublished\":\"2022-04-29T18:25:51+00:00\",\"dateModified\":\"2022-04-29T19:25:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/wp-content\\\/uploads\\\/sites\\\/50\\\/2022\\\/04\\\/image-1.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/wp-content\\\/uploads\\\/sites\\\/50\\\/2022\\\/04\\\/image-1.png\",\"width\":883,\"height\":289},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/\",\"name\":\"Panoptic Lidar Segmentation for Autonomous Driving\",\"description\":\"Students: Abhinav Agarwalla, Xuhua Huang | Advisor: Aljosa Osep, James Hays, Deva Ramanan | Sponsor: Argo AI\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teame\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Panoptic Lidar Segmentation for Autonomous Driving","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Panoptic Lidar Segmentation for Autonomous Driving","og_description":"Overview We accumulate a point cloud sequence and encode it using a voxel grid-based encoder-decoder backbone. After accumulating encoded points in a 3D voxel grid, we (a) down-sample the volume (via sparse 3D convolutions and pooling layers), and flatten the bottleneck layer along the height-axis to obtain BEV representation (followed by 2D convolutional layers, similar … Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/","og_site_name":"Panoptic Lidar Segmentation for Autonomous Driving","article_modified_time":"2022-04-29T19:25:50+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-content\/uploads\/sites\/50\/2022\/04\/image-1.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/","name":"Method - Panoptic Lidar Segmentation for Autonomous Driving","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-content\/uploads\/sites\/50\/2022\/04\/image-1.png","datePublished":"2022-04-29T18:25:51+00:00","dateModified":"2022-04-29T19:25:50+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-content\/uploads\/sites\/50\/2022\/04\/image-1.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-content\/uploads\/sites\/50\/2022\/04\/image-1.png","width":883,"height":289},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/","name":"Panoptic Lidar Segmentation for Autonomous Driving","description":"Students: Abhinav Agarwalla, Xuhua Huang | Advisor: Aljosa Osep, James Hays, Deva Ramanan | Sponsor: Argo AI","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/pages\/147","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/users\/107"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/comments?post=147"}],"version-history":[{"count":12,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/pages\/147\/revisions"}],"predecessor-version":[{"id":205,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/pages\/147\/revisions\/205"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teame\/wp-json\/wp\/v2\/media?parent=147"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}