{"id":2,"date":"2019-02-01T20:21:04","date_gmt":"2019-02-01T20:21:04","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/?page_id=2"},"modified":"2019-12-06T21:05:24","modified_gmt":"2019-12-06T21:05:24","slug":"project-summary","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/","title":{"rendered":"Project Summary"},"content":{"rendered":"

Abstract<\/strong><\/h1>\n

Nowadays autonomous vehicles have an array of sensors. These sensors help detect the 3D shape of their surrounding environments and also help the vehicles navigate through them safely. This 3D information is also useful in other computer vision tasks like gesture recognition and geometric reconstruction. However every sensor has certain limitations. For instance, a RADAR or a\u00a0LiDAR<\/span>\u00a0has excellent range information but provides poor spatial resolution whereas conventional cameras capture really high-quality spatial resolution but they fail to provide high range information. Our capstone thus aims at overcoming some of these limitations by fusing all these different sensors together. More precisely, we aim at fusing a Single Photon\u00a0Lidar<\/span>\u00a0(SP-Lidar<\/span>), a Stereo Pair and a\u00a0 LiDAR to form an intermediate cost volume which when passed through a deep neural network, estimates better disparity and depth information of the scene.<\/p>\n

We start by first processing the stereo information using the Pyramid Stereo Matching Network (PSMNet<\/span>)[1] which estimates disparity using a deep learning architecture. Broadly,\u00a0PSMNet<\/span>\u00a0first generates an intermediate cost volume representation using Spatial Pyramid Pooling Layer and then uses a bunch of 3D\u00a0convolutional<\/span>\u00a0layers to process this 3D volume. It finally\u00a0upsamples<\/span>\u00a0the output and performs disparity regression.<\/p>\n

We simulate our own dataset using CARLA [2] that has information from LiDAR, Single Photon LiDAR, Stereo and Monocular Cameras. In all we have around 8000 image pairs in training and 2000 images pairs in testing.<\/p>\n

\"\"<\/p>\n

Since\u00a0CARLA<\/span>\u00a0dataset<\/span>\u00a0doesn\u2019t have SP-Lidar<\/span>\u00a0measurements, we simulate the same under various conditions. We assume that the data is coming from both the signal and the ambient photons and we apply Poisson noise to these photon counts. We explore the setting where Signal-to-Background (SBR<\/span>) ratio is 1:1.<\/p>\n

For our fusion technique, we first construct cost volumes for LiDARs, SP-LiDARs and Monocular Camera. We then normalize these individual cost volumes and simply add them to create an intermediate cost volume which then goes into the deep network, as has been depicted in the figure above.<\/p>\n

We observe that fused features give the best performance for various metrics, under severe ambient conditions as well. We also test our model on the\u00a0KITTI<\/span>\u00a0dataset<\/span>\u00a0[3] and showcase the generalization of our architecture over real world images as well. The future extension for this work can be to collect real world dataset for Single Photon LiDARs and test\u00a0sensor calibration, time synchronisation problems etc. We can also try to incorporate other sensors such as RADARs in our sensor fusion pipeline.<\/span><\/p>\n

[1] J.R. Chang\u00a0et<\/span>.\u00a0al<\/span>. “Pyramid stereo matching network”,\u00a0CVPR<\/span>\u00a0(2018)<\/div>\n
[2]\u00a0Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun.\u00a0 “CARLA: An open urban driving simulator”. In Conference on Robot Learning (CoRL), 2017.<\/span><\/div>\n
[3] A. Geiger\u00a0et<\/span>.\u00a0al<\/span>. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”,\u00a0CVPR<\/span>\u00a0(2012)<\/div>\n","protected":false},"excerpt":{"rendered":"

Abstract Nowadays autonomous vehicles have an array of sensors. These sensors help detect the 3D shape of their surrounding environments and also help the vehicles navigate through them safely. This 3D information is also useful in other computer vision tasks like gesture recognition and geometric reconstruction. However every sensor has certain limitations. For instance, a … <\/p>\n

Continue reading “Project Summary”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"yoast_head":"\nProject Summary - Sensor Fusion with Single Photon Detectors<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project Summary - Sensor Fusion with Single Photon Detectors\" \/>\n<meta property=\"og:description\" content=\"Abstract Nowadays autonomous vehicles have an array of sensors. These sensors help detect the 3D shape of their surrounding environments and also help the vehicles navigate through them safely. This 3D information is also useful in other computer vision tasks like gesture recognition and geometric reconstruction. However every sensor has certain limitations. For instance, a … Continue reading "Project Summary"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/\" \/>\n<meta property=\"og:site_name\" content=\"Sensor Fusion with Single Photon Detectors\" \/>\n<meta property=\"article:modified_time\" content=\"2019-12-06T21:05:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-content\/uploads\/sites\/25\/2019\/05\/Screen-Shot-2019-12-06-at-11.35.42-AM.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2126\" \/>\n\t<meta property=\"og:image:height\" content=\"1158\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/\",\"name\":\"Project Summary - Sensor Fusion with Single Photon Detectors\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2019\\\/05\\\/Screen-Shot-2019-12-06-at-11.35.42-AM-1024x558.png\",\"datePublished\":\"2019-02-01T20:21:04+00:00\",\"dateModified\":\"2019-12-06T21:05:24+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2019\\\/05\\\/Screen-Shot-2019-12-06-at-11.35.42-AM.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/wp-content\\\/uploads\\\/sites\\\/25\\\/2019\\\/05\\\/Screen-Shot-2019-12-06-at-11.35.42-AM.png\",\"width\":2126,\"height\":1158},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/project-summary\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project Summary\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/\",\"name\":\"Sensor Fusion with Single Photon Detectors\",\"description\":\"Carnegie Mellon University, MSCV Capstone Project\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2019teamj\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project Summary - Sensor Fusion with Single Photon Detectors","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/","og_locale":"en_US","og_type":"article","og_title":"Project Summary - Sensor Fusion with Single Photon Detectors","og_description":"Abstract Nowadays autonomous vehicles have an array of sensors. These sensors help detect the 3D shape of their surrounding environments and also help the vehicles navigate through them safely. This 3D information is also useful in other computer vision tasks like gesture recognition and geometric reconstruction. However every sensor has certain limitations. For instance, a … Continue reading \"Project Summary\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/","og_site_name":"Sensor Fusion with Single Photon Detectors","article_modified_time":"2019-12-06T21:05:24+00:00","og_image":[{"width":2126,"height":1158,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-content\/uploads\/sites\/25\/2019\/05\/Screen-Shot-2019-12-06-at-11.35.42-AM.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/","name":"Project Summary - Sensor Fusion with Single Photon Detectors","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/#primaryimage"},"thumbnailUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-content\/uploads\/sites\/25\/2019\/05\/Screen-Shot-2019-12-06-at-11.35.42-AM-1024x558.png","datePublished":"2019-02-01T20:21:04+00:00","dateModified":"2019-12-06T21:05:24+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-content\/uploads\/sites\/25\/2019\/05\/Screen-Shot-2019-12-06-at-11.35.42-AM.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-content\/uploads\/sites\/25\/2019\/05\/Screen-Shot-2019-12-06-at-11.35.42-AM.png","width":2126,"height":1158},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/project-summary\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/"},{"@type":"ListItem","position":2,"name":"Project Summary"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/","name":"Sensor Fusion with Single Photon Detectors","description":"Carnegie Mellon University, MSCV Capstone Project","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/comments?post=2"}],"version-history":[{"count":21,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":98,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/pages\/2\/revisions\/98"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2019teamj\/wp-json\/wp\/v2\/media?parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}