{"id":90,"date":"2020-05-10T19:36:35","date_gmt":"2020-05-10T19:36:35","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/?page_id=90"},"modified":"2020-12-15T17:49:50","modified_gmt":"2020-12-15T17:49:50","slug":"project","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","title":{"rendered":"Project"},"content":{"rendered":"\n
\"\"
You only fly once<\/figcaption><\/figure>\n\n\n\n

Visual localization plays an essential role in Unmanned Aerial Vehicle (UAV) tasks such as delivery, seed sowing, etc. Typically for these tasks, the majority of the travel is composed of fixed, point-to-point routes. Therefore if we make it possible to collect good-quality data with just one manual pre-flight on the routes, and learn a data-driven re-localization module requiring only images, then we can effectively leverage the few expensive high-end UAVs and the large quantity of regular UAVs. This will bring great autonomy to the regular UAVs and be much more economical in large-scale deployment.<\/p>\n\n\n\n

\"\"
Keep calm and stay light-weight<\/figcaption><\/figure>\n\n\n\n

By keeping the learning phase offline, the UAV only needs the camera images on-the-fly for inference. The re-localization module is a separate approach from conventional visual odometry as a fail-safe plan. Therefore, it should be less susceptible to what conventional odometry fails at, such as drifting and tracking lost, and be able to work in the absence of GPS, network connection, etc. And of course, it should be accurate during pose estimation.<\/p>\n\n\n\n

\"\"
Separate Prediction<\/figcaption><\/figure>\n\n\n\n

We thereby propose 2 methods to achieve such goals. One is to develop a separate localization module. In case of a lost track, this module can independently predict a pose for planning and control, meanwhile help to re-initialize a state for the visual odometry. Because this is an independent prediction, it is safe even when VO loses its tracking entirely.<\/p>\n\n\n\n

\"\"
Soft-RANSAC for Separate Prediction<\/figcaption><\/figure>\n\n\n\n

Our first approach is to develop a separate prediction module, based on scene coordinate regression plus differentiable ransac. Given\u00a0the input image, the network predicts the corresponding spatial xyz coordinates for the pixels in the input. The output scene coordinates are selected by a soft RANSAC and then go through the PnP algorithm to generate the estimation of the camera pose. From RANSAC we generate a pool of pose estimations, and then we back-project the ground truth scene coordinates into the camera frame to evaluate how well it is aligned with the estimation, and select the best one as the final output.<\/p>\n\n\n\n

\"\"
Pose Correction<\/figcaption><\/figure>\n\n\n\n

Alternatively, we can also train a network which, instead of proposing its own prediction, but rather tries to correct the output of the visual odometry.This approach is more actively engaging with the visual odometry to reduce the possibility of losing a track. It should work better than a separate prediction when visual odometry is only off by a small amount, but may be challenged when VO is off by too much, in which case we can still fall back to a separate prediction.<\/p>\n\n\n\n

\"\"
The Self-Supervised Pipeline of Pose Correction<\/figcaption><\/figure>\n\n\n\n

The input\u00a0to\u00a0the network is two images and the\u00a0ego-motion\u00a0estimated by a non-learning VO algorithm. The pose correction network will output pose correction,\u00a0also depth map to\u00a0guide\u00a0the\u00a0reconstruction\u00a0of images between different frames,\u00a0and\u00a0use the reconstructed image to compute the photometric reconstruction loss. The network also outputs an\u00a0explainability\u00a0mask to filter out the\u00a0outlier regions, for instance, moving objects, occlusions, etc. Using photometric reconstruction loss seems more reasonable than just fitting the pose difference. <\/p>\n","protected":false},"excerpt":{"rendered":"

Visual localization plays an essential role in Unmanned Aerial Vehicle (UAV) tasks such as delivery, seed sowing, etc. Typically for these tasks, the majority of the travel is composed of fixed, point-to-point routes. Therefore if we make it possible to collect good-quality data with just one manual pre-flight on the routes, and learn a data-driven … <\/p>\n

Continue reading “Project”<\/span><\/a><\/p>\n","protected":false},"author":95,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-90","page","type-page","status-publish","hentry"],"yoast_head":"\nProject - Image-based Localization for Autonomous Drones<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project - Image-based Localization for Autonomous Drones\" \/>\n<meta property=\"og:description\" content=\"Visual localization plays an essential role in Unmanned Aerial Vehicle (UAV) tasks such as delivery, seed sowing, etc. Typically for these tasks, the majority of the travel is composed of fixed, point-to-point routes. Therefore if we make it possible to collect good-quality data with just one manual pre-flight on the routes, and learn a data-driven … Continue reading "Project"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/\" \/>\n<meta property=\"og:site_name\" content=\"Image-based Localization for Autonomous Drones\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-15T17:49:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/model.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1252\" \/>\n\t<meta property=\"og:image:height\" content=\"482\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/\",\"name\":\"Project - Image-based Localization for Autonomous Drones\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/wp-content\\\/uploads\\\/sites\\\/44\\\/2020\\\/05\\\/model.png\",\"datePublished\":\"2020-05-10T19:36:35+00:00\",\"dateModified\":\"2020-12-15T17:49:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/wp-content\\\/uploads\\\/sites\\\/44\\\/2020\\\/05\\\/model.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/wp-content\\\/uploads\\\/sites\\\/44\\\/2020\\\/05\\\/model.png\",\"width\":1252,\"height\":482},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/\",\"name\":\"Image-based Localization for Autonomous Drones\",\"description\":\"Sponsor: Near Earth Autonomy & Advisor: Prof. John Galeotti\",\"publisher\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#organization\",\"name\":\"Image-based Localization for Autonomous Drones\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/wp-content\\\/uploads\\\/sites\\\/44\\\/2020\\\/05\\\/cropped-NEAR_EARTH_LOGO.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/wp-content\\\/uploads\\\/sites\\\/44\\\/2020\\\/05\\\/cropped-NEAR_EARTH_LOGO.png\",\"width\":354,\"height\":250,\"caption\":\"Image-based Localization for Autonomous Drones\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamn\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project - Image-based Localization for Autonomous Drones","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","og_locale":"en_US","og_type":"article","og_title":"Project - Image-based Localization for Autonomous Drones","og_description":"Visual localization plays an essential role in Unmanned Aerial Vehicle (UAV) tasks such as delivery, seed sowing, etc. Typically for these tasks, the majority of the travel is composed of fixed, point-to-point routes. Therefore if we make it possible to collect good-quality data with just one manual pre-flight on the routes, and learn a data-driven … Continue reading \"Project\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","og_site_name":"Image-based Localization for Autonomous Drones","article_modified_time":"2020-12-15T17:49:50+00:00","og_image":[{"width":1252,"height":482,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/model.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","name":"Project - Image-based Localization for Autonomous Drones","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/model.png","datePublished":"2020-05-10T19:36:35+00:00","dateModified":"2020-12-15T17:49:50+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/model.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/model.png","width":1252,"height":482},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/"},{"@type":"ListItem","position":2,"name":"Project"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","name":"Image-based Localization for Autonomous Drones","description":"Sponsor: Near Earth Autonomy & Advisor: Prof. John Galeotti","publisher":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#organization","name":"Image-based Localization for Autonomous Drones","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#\/schema\/logo\/image\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/cropped-NEAR_EARTH_LOGO.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-content\/uploads\/sites\/44\/2020\/05\/cropped-NEAR_EARTH_LOGO.png","width":354,"height":250,"caption":"Image-based Localization for Autonomous Drones"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/pages\/90","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/users\/95"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/comments?post=90"}],"version-history":[{"count":7,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/pages\/90\/revisions"}],"predecessor-version":[{"id":114,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/pages\/90\/revisions\/114"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamn\/wp-json\/wp\/v2\/media?parent=90"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}