{"id":5,"date":"2023-05-08T14:49:46","date_gmt":"2023-05-08T18:49:46","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/?page_id=5"},"modified":"2023-12-16T21:03:08","modified_gmt":"2023-12-17T01:03:08","slug":"mixed-reality-origami-assistant-hand-and-object-understanding-for-ar-glasses","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/","title":{"rendered":"Introduction"},"content":{"rendered":"\n\n<div class=\"wp-block-image is-style-default\">\n<figure data-wp-context=\"{&quot;imageId&quot;:&quot;6a498ce288594&quot;}\" data-wp-interactive=\"core\/image\" data-wp-key=\"6a498ce288594\" class=\"aligncenter size-large is-resized wp-lightbox-container\"><img loading=\"lazy\" decoding=\"async\" width=\"910\" height=\"1287\" data-wp-class--hide=\"state.isContentHidden\" data-wp-class--show=\"state.isContentVisible\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on--click=\"actions.showLightbox\" data-wp-on--load=\"callbacks.setButtonStyles\" data-wp-on-window--resize=\"callbacks.setButtonStyles\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png\" alt=\"\" class=\"wp-image-75\" style=\"width:382px;height:540px\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png 910w, https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited-212x300.png 212w, https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited-724x1024.png 724w, https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited-768x1086.png 768w\" sizes=\"auto, (max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"state.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"state.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><figcaption class=\"wp-element-caption\">Traditional instruction manuals can be difficult to follow and often lack the detail and nuance needed to create the piece <\/figcaption><\/figure>\n<\/div>\n\n\n<p>Creative and sequential tasks using hands, like origami, welding, or knitting, are difficult for beginners to learn. Traditional methods of instruction, like instruction manuals or videos, do not offer feedback on how well a learner is performing the task. Feedback is crucial in the learning process, as it helps learners understand what they are doing well and what needs improvement. In contrast, modern computer vision systems and projection interfaces can provide real-time feedback to the user, tracking progress and correcting errors as they occur.  This is where Origami Sensei comes in! Origami Sensei is a data-driven vision-based system that provides step-by-step instructions and real-time feedback to assist users in creating origami. It is designed to be intuitive and interactive, making it easy for beginners to follow. With Origami Sensei, users can choose from a library of pre-existing designs or create their own origami. The complete system will utilize computer vision models and algorithms to achieve three main functionality: <\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Recognizing users\u2019 current step in real-time (Spring 2023);<\/li>\n\n\n\n<li>Detect the location of the origami paper and projecting guidance for the next step directly onto the table in real-time (Fall 2023);<\/li>\n\n\n\n<li>Recognizing when the user makes a mistake and giving instructions to correct it (Fall 2023). <\/li>\n<\/ol>\n\n\n\n<p>Altogether, our capstone project develops a vision-based system that can automatically track the location of the origami paper and identify the current state, and then projects the corresponding instruction for the next step directly and accurately onto the paper. Compared to existing forms, we provide a new modality of origami tutorials with an unprecedented level of user-friendly interaction and automation. With our work, we aim for anyone to easily achieve origami folding and provide insight for future vision-guided teaching system with a similar projection-based setup.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Creative and sequential tasks using hands, like origami, welding, or knitting, are difficult for beginners to learn. Traditional methods of instruction, like instruction manuals or videos, do not offer feedback on how well a learner is performing the task. Feedback is crucial in the learning process, as it helps learners understand what they are doing &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Introduction&#8221;<\/span><\/a><\/p>\n","protected":false},"author":180,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-5","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Introduction - Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses<\/title>\n<meta name=\"description\" content=\"Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction - Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses\" \/>\n<meta property=\"og:description\" content=\"Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/\" \/>\n<meta property=\"og:site_name\" content=\"Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-17T01:03:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"Report\"],\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\"},\"author\":{\"name\":\"qiyuc\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#\\\/schema\\\/person\\\/ddf9ff8e63d3e46e04a09cd91fc9b0c6\"},\"headline\":\"Introduction\",\"datePublished\":\"2023-05-08T18:49:46+00:00\",\"dateModified\":\"2023-12-17T01:03:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\"},\"wordCount\":306,\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/wp-content\\\/uploads\\\/sites\\\/92\\\/2023\\\/05\\\/R1-edited.png\",\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\",\"name\":\"Introduction - Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/wp-content\\\/uploads\\\/sites\\\/92\\\/2023\\\/05\\\/R1-edited.png\",\"datePublished\":\"2023-05-08T18:49:46+00:00\",\"dateModified\":\"2023-12-17T01:03:08+00:00\",\"description\":\"Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/wp-content\\\/uploads\\\/sites\\\/92\\\/2023\\\/05\\\/R1-edited.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/wp-content\\\/uploads\\\/sites\\\/92\\\/2023\\\/05\\\/R1-edited.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introduction\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/\",\"name\":\"Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/#\\\/schema\\\/person\\\/ddf9ff8e63d3e46e04a09cd91fc9b0c6\",\"name\":\"qiyuc\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/8d213912c5b9a8be85aba547d02df95b1805577e904a37e35395b769b177226f?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/8d213912c5b9a8be85aba547d02df95b1805577e904a37e35395b769b177226f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/8d213912c5b9a8be85aba547d02df95b1805577e904a37e35395b769b177226f?s=96&d=mm&r=g\",\"caption\":\"qiyuc\"},\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team15\\\/author\\\/qiyuc\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction - Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses","description":"Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/","og_locale":"en_US","og_type":"article","og_title":"Introduction - Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses","og_description":"Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/","og_site_name":"Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses","article_modified_time":"2023-12-17T01:03:08+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","Report"],"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#article","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/"},"author":{"name":"qiyuc","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#\/schema\/person\/ddf9ff8e63d3e46e04a09cd91fc9b0c6"},"headline":"Introduction","datePublished":"2023-05-08T18:49:46+00:00","dateModified":"2023-12-17T01:03:08+00:00","mainEntityOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/"},"wordCount":306,"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png","inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/","name":"Introduction - Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png","datePublished":"2023-05-08T18:49:46+00:00","dateModified":"2023-12-17T01:03:08+00:00","description":"Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-content\/uploads\/sites\/92\/2023\/05\/R1-edited.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/"},{"@type":"ListItem","position":2,"name":"Introduction"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/","name":"Mixed Reality Origami Assistant: Hand and Object Understanding for AR glasses","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/#\/schema\/person\/ddf9ff8e63d3e46e04a09cd91fc9b0c6","name":"qiyuc","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/8d213912c5b9a8be85aba547d02df95b1805577e904a37e35395b769b177226f?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/8d213912c5b9a8be85aba547d02df95b1805577e904a37e35395b769b177226f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8d213912c5b9a8be85aba547d02df95b1805577e904a37e35395b769b177226f?s=96&d=mm&r=g","caption":"qiyuc"},"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/author\/qiyuc\/"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/pages\/5","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/users\/180"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/comments?post=5"}],"version-history":[{"count":15,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/pages\/5\/revisions"}],"predecessor-version":[{"id":167,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/pages\/5\/revisions\/167"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team15\/wp-json\/wp\/v2\/media?parent=5"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}