Creative and sequential tasks using hands, like origami, welding, or knitting, are difficult for beginners to learn. Traditional methods of instruction, like instruction manuals or videos, do not offer feedback on how well a learner is performing the task. Feedback is crucial in the learning process, as it helps learners understand what they are doing well and what needs improvement. In contrast, modern computer vision systems and projection interfaces can provide real-time feedback to the user, tracking progress and correcting errors as they occur. This is where Origami Sensei comes in! Origami Sensei is a data-driven vision-based system that provides step-by-step instructions and real-time feedback to assist users in creating origami. It is designed to be intuitive and interactive, making it easy for beginners to follow. With Origami Sensei, users can choose from a library of pre-existing designs or create their own origami. The complete system will utilize computer vision models and algorithms to achieve three main functionality:
- Recognizing users’ current step in real-time (Spring 2023);
- Detect the location of the origami paper and projecting guidance for the next step directly onto the table in real-time (Fall 2023);
- Recognizing when the user makes a mistake and giving instructions to correct it (Fall 2023).
Altogether, our capstone project develops a vision-based system that can automatically track the location of the origami paper and identify the current state, and then projects the corresponding instruction for the next step directly and accurately onto the paper. Compared to existing forms, we provide a new modality of origami tutorials with an unprecedented level of user-friendly interaction and automation. With our work, we aim for anyone to easily achieve origami folding and provide insight for future vision-guided teaching system with a similar projection-based setup.