{"id":14,"date":"2022-04-30T14:38:49","date_gmt":"2022-04-30T18:38:49","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/?page_id=14"},"modified":"2022-12-20T01:47:35","modified_gmt":"2022-12-20T06:47:35","slug":"home","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/","title":{"rendered":"Overview"},"content":{"rendered":"\n<p>Robotic manipulation refers to the ways robots interact with the objects around them: grasping an object, opening a door, packing an order into a box, folding laundry\u2026etc. All these actions require robots to recognize and understand the environment, then plan and control the motion of their hands and arms in an intelligent way.<\/p>\n\n\n\n<p>While there are various tasks pertaining to robotic manipulation, this project focuses on Bin Picking &#8211; a 3D application that involves using a robotic system to do all the following. <\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Locate a part in a random orientation in any quadrant of a bin. The <strong>exact<\/strong> object to pick up.<\/li><li>Plan a complete path from pick to place without the robot reaching any singularities or joint limits along the way.<\/li><li>Enter the bin in a specific robot pose for that particular orientation of the matched part.<\/li><li>Do not break or damage any adjacent parts to the part you are picking.<\/li><li>Exit out of a bin and then place that part on a target, in the correct orientation without hitting anything in its environment.<\/li><\/ul>\n\n\n\n<p>We specifically focus on the perception part of the above tasks, i.e, precisely locating the part in the bin. Specifically, we are focused on industrial bin-picking in  E-commerce warehouses for order fulfillment. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Goals &amp; Objectives<\/h2>\n\n\n\n<p>Develop object pose estimation algorithms for robotics manipulation and related computer vision topics including registration, detection, reconstruction, and so on.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Come up with a way to identify poses &amp; potential grasp points for randomly packed objects in a tote. <\/li><li>Come up with a way to gather object information on the fly and identify possible object poses for potential grasp points for randomly packed objects in a tote given partial dimensions, such as height only.<\/li><li>Find grasp points for these objects without any initial information about the object.<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>Sponsored By<\/p><cite><a href=\"https:\/\/mujin-corp.com\/\"><img loading=\"lazy\" decoding=\"async\" width=\"698\" height=\"319\" class=\"wp-image-22\" style=\"width: 150px\" src=\"http:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png\" alt=\"\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png 698w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo-300x137.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo-230x105.png 230w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo-350x160.png 350w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo-480x219.png 480w\" sizes=\"auto, (max-width: 698px) 100vw, 698px\" \/><\/a><\/cite><\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<div class=\"wp-block-group is-layout-flow wp-block-group-is-layout-flow\">\n<h2 class=\"wp-block-heading\">Motivation<\/h2>\n\n\n\n<p>Mujin makes a software controller platform to provide motion planning and advanced computer vision features to industrial robots. Mujin\u2019s core software is particularly suited for picking applications in the logistics and manufacturing industries. In fact, with over 600 production deployments, we\u2019ve proven that robots, when combined with the latest sensor technology, can meet or exceed human productivity levels in high-level applications such as piece picking, depalletizing, and palletizing. For many applications, our customers are having relatively organized situations where the robot will be deployed, such as in the below figure where the robot uses data about the products\u2019 size, shape, and other characteristics.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/s3.us-west-2.amazonaws.com\/secure.notion-static.com\/6207abfd-8013-4657-b04e-e08a1b7930c8\/Untitled.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&amp;X-Amz-Credential=AKIAT73L2G45EIPT3X45%2F20220430%2Fus-west-2%2Fs3%2Faws4_request&amp;X-Amz-Date=20220430T192134Z&amp;X-Amz-Expires=86400&amp;X-Amz-Signature=4e0d811a2e276da449f1de2139afc498eddc7ad620dd73d22e71f178fd2286a8&amp;X-Amz-SignedHeaders=host&amp;response-content-disposition=filename%20%3D%22Untitled.png%22&amp;x-id=GetObject\" alt=\"\" \/><\/figure>\n\n\n\n<p>See examples of real-world applications here:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Pick and place for order fulfillment &#8211; <a href=\"https:\/\/youtu.be\/88MDtPshQ1M\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/youtu.be\/88MDtPshQ1M<\/a><\/li><\/ul>\n\n\n\n\n\n<ul class=\"wp-block-list\"><li>Sorter &#8211; <a href=\"https:\/\/youtu.be\/nlFESLYWHE8\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/youtu.be\/nlFESLYWHE8<\/a> <\/li><\/ul>\n\n\n\n\n\n<p>We\u2019re looking to take this a step further and solve a robot perception problem to be able to successfully unpack a messy box full of the same items, all with minimal data about the items.<\/p>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Problem Statement<\/h2>\n\n\n\n<p>The project focuses on these three main questions <\/p>\n\n\n\n<p class=\"has-vivid-red-color has-text-color\"><strong><em>1. How to identify the first object to pick?<\/em><\/strong><\/p>\n\n\n\n<p class=\"has-vivid-red-color has-text-color\"><strong><em>2. What can we learn from the first pick to improve subsequent picks?<\/em><\/strong><\/p>\n\n\n\n<p class=\"has-vivid-red-color has-text-color\"><strong><em>3. How to better represent scene\/object information for pick-and-place?<\/em><\/strong><\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-28f84493 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p><strong>Constraints<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Should be a model-free solution<\/li><li>Must learn the 3D object representations <\/li><li>Should be able to identify object pose<\/li><\/ul>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p><strong>Assumptions<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Single SKU type in a tote<\/li><li>Given the object model, pick-and-place is done by the existing pipeline.<\/li><\/ul>\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Why is robotic picking so difficult?<\/h3>\n\n\n\n<pre class=\"wp-block-verse has-text-color\" style=\"color:#791818\">\u2013 <em>\"If robots can build a car, they can pick orders. Right?!\"<\/em><\/pre>\n\n\n\n<p>Not really. While control &amp; planning has their own set of problems and challenges, the main difficulty in the perception component of robot picking arises from the fact that object can vary in visual appearance, texture, shape, etc. <\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Billions of SKUs <\/strong>\u2013 goods come in a variety of shapes, dimensions, weights, packaging, colors, textures, reflectivity, and fragility.<\/li><li><strong>Dynamic presentations of totes <\/strong>\u2013 no two presentations of the totes on a conveyor belt are the same<\/li><\/ul>\n\n\n\n<p>Here are some examples of various objects in a tote on a conveyer belt as seen from the robot&#8217;s camera.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"965\" height=\"592\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16.png\" alt=\"\" class=\"wp-image-108\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16.png 965w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16-300x184.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16-768x471.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16-920x564.png 920w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16-230x141.png 230w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16-350x215.png 350w, https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/12\/Screenshot-from-2022-12-20-01-46-16-480x294.png 480w\" sizes=\"auto, (max-width: 965px) 100vw, 965px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Robotic manipulation refers to the ways robots interact with the objects around them: grasping an object, opening a door, packing an order [&hellip;]<\/p>\n","protected":false},"author":134,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-14","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Overview - Automate Pose Estimation for Robotics Manipulation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Overview - Automate Pose Estimation for Robotics Manipulation\" \/>\n<meta property=\"og:description\" content=\"Robotic manipulation refers to the ways robots interact with the objects around them: grasping an object, opening a door, packing an order [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/\" \/>\n<meta property=\"og:site_name\" content=\"Automate Pose Estimation for Robotics Manipulation\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-20T06:47:35+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/\",\"name\":\"Overview - Automate Pose Estimation for Robotics Manipulation\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/wp-content\\\/uploads\\\/sites\\\/68\\\/2022\\\/04\\\/mujinLogo.png\",\"datePublished\":\"2022-04-30T18:38:49+00:00\",\"dateModified\":\"2022-12-20T06:47:35+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#primaryimage\",\"url\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/wp-content\\\/uploads\\\/sites\\\/68\\\/2022\\\/04\\\/mujinLogo.png\",\"contentUrl\":\"http:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/wp-content\\\/uploads\\\/sites\\\/68\\\/2022\\\/04\\\/mujinLogo.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/\",\"name\":\"Automate Pose Estimation for Robotics Manipulation\",\"description\":\"CMU MSCV 2022 Capstone Project - Group 13\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team13\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Overview - Automate Pose Estimation for Robotics Manipulation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/","og_locale":"en_US","og_type":"article","og_title":"Overview - Automate Pose Estimation for Robotics Manipulation","og_description":"Robotic manipulation refers to the ways robots interact with the objects around them: grasping an object, opening a door, packing an order [&hellip;]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/","og_site_name":"Automate Pose Estimation for Robotics Manipulation","article_modified_time":"2022-12-20T06:47:35+00:00","og_image":[{"url":"http:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/","name":"Overview - Automate Pose Estimation for Robotics Manipulation","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#primaryimage"},"thumbnailUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png","datePublished":"2022-04-30T18:38:49+00:00","dateModified":"2022-12-20T06:47:35+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#primaryimage","url":"http:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png","contentUrl":"http:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-content\/uploads\/sites\/68\/2022\/04\/mujinLogo.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/"},{"@type":"ListItem","position":2,"name":"Overview"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/","name":"Automate Pose Estimation for Robotics Manipulation","description":"CMU MSCV 2022 Capstone Project - Group 13","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/pages\/14","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/users\/134"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/comments?post=14"}],"version-history":[{"count":15,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/pages\/14\/revisions"}],"predecessor-version":[{"id":109,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/pages\/14\/revisions\/109"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/wp-json\/wp\/v2\/media?parent=14"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}