{"id":48,"date":"2022-04-30T18:06:28","date_gmt":"2022-04-30T22:06:28","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/?page_id=48"},"modified":"2022-12-20T05:10:58","modified_gmt":"2022-12-20T10:10:58","slug":"technical-report","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2022team13\/technical-report\/","title":{"rendered":"Technical Report"},"content":{"rendered":"\n

Abstract<\/h2>\n\n\n\n

Object pose estimation is a fundamental requirement for robotic manipulation tasks. Most methods that have high accuracy to be deployed in real-world industrial scenarios depend on known 3D object models like CAD models, etc. While these approaches may be sufficient for certain kinds of industries like manufacturing, these approaches cause bottlenecks in scenarios that deal with objects whose appearance and shape are constantly changing, like in the e-commerce warehouses. This project proposes a mechanism to automate this process of model-free pose estimation. We develop a pipeline that enables the system to learn the shape and appearance of novel objects and identify their poses without manual supervision or datasets. <\/p>\n\n\n\n

\n\n\n\n

Data<\/h2>\n\n\n\n
\"\"
sample data from Mujin<\/figcaption><\/figure>\n\n\n\n

The input data to the system consists of high-resolution RGB images captured from a top view along with left and right grayscale images captured by a stereo camera setup. Corresponding structured point clouds are also available. Along with these images, the camera intrinsics, and extrinsics are also provided.<\/p>\n\n\n\n

\n\n\n\n

Initial Experiments<\/h2>\n\n\n\n

SD-MRCNN<\/a><\/h4>\n\n\n\n

The first step to identify object poses is to identify individual object instances. We chose Synthetic Depth MaskRCNN (SDMRCNN) as our baseline model to evaluate current SOTA in instance recognition on Mujin dataset. We chose SDMRCNN since it is an MRCNN-based approach that utilizes depth data and classifies the objects in the scene as either foreground (objects) or background (tote\/container).<\/p>\n\n\n\n

The following images show sample results of SDMRCNN pre-trained on the WISDOM dataset and inferred on Mujin data.<\/p>\n\n\n\n

\n
\n
\"\"<\/figure>\n<\/div>\n\n\n\n
\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\"\"<\/figure>\n<\/div>\n\n\n\n
\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n

These results clearly show the model failed to identify object instances, and as the object complexity increases (no texture, reflection, transparency), the model completely fails to identify anything at all. Training SDMRCNN on these data samples from Mujin would result in better instance recognition, but that would require creating new datasets for these kinds of objects. However, since the object appearing in the warehouse constantly change, it is not feasible to keep creating new datasets. Therefore, we need a better alternative.<\/p>\n\n\n\n

\n\n\n\n

Solution Pipeline<\/h2>\n\n\n\n

To overcome the issues mentioned above, we propose the following multi-stage pipeline to tackle novel object pose recognition.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

For each state of the pipeline, we identify a baseline approach to perform the corresponding task. Once the full pipeline implementation is complete, we then focus on each stage to improve results.<\/p>\n\n\n\n

Phase-1<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Stage-1: First Pick<\/h3>\n\n\n\n

The first stage of the pipeline is to pick an object from the tote in a model-free approach. It is a costly (in terms of computing, memory, and time) process to be able to identify and pick an object without knowing the 3D model of the object. However, since this process is going to be performed only once for 1000s of objects that may follow, this costly first step is acceptable. <\/p>\n\n\n\n

Since there are existing solutions to perform this task, even on complex objects, (ex: Fully Convolutional GQ-CNN by BerkeleyAutomation<\/a>), we do not tackle this problem in the current scope of the project and revisit it later once we complete the rest of the pipeline.<\/p>\n\n\n\n

Stage-2: 3D Recognition <\/h3>\n\n\n\n

The goal of this stage is to recognize the 3D representation of the object picked up in stage-1. The resulting 3D model should accurately capture the object’s 3D geometry and textures. <\/p>\n\n\n\n

While there are several approaches that can be used to recognize and reconstruct the object’s 3D model, (ex: using a multi-view solution or 3D scans, etc), we use NeRS (jasonyzhang.com)<\/a> as our baseline approach to generate the 3D model of the object.<\/p>\n\n\n\n

NeRS<\/h3>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Data<\/h3>\n\n\n\n

To obtain the initial results (and avoid logistical issues with the camera and robot calibration), we create the input data for NeRS with a handheld object that represents the complexity of objects appearing in Mujin’s warehouse environments. The object masks for these images were created manually for ease of completing the pipeline. This is a trivial task and can be obtained from various automated methods later.<\/p>\n\n\n\n

\"\"
Input images and corresponding mask for NeRS<\/figcaption><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Results<\/h3>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Result: 3D model video<\/a><\/p>\n\n\n\n

Stage-3: Instance Recognition<\/h2>\n\n\n\n

In this stage, we generate a instance recognition model that has been fine-tuned on the object picked up in stage-1. We achieve this by generating a instance recognition dataset using the 3D object model from stage-2 and fine-tuning the instance recognition model (SDMRCNN) on this dataset. The resulting model should be able to perform much better than the pre-trained one shown in the initial experiments.<\/p>\n\n\n\n

Generating dataset for instance recognition<\/h3>\n\n\n\n

Naive approach <\/strong>– Generate single instance images from different views of the object. Then overlay and blend these images on a template container image. However, this is not good as the resulting image is not realistic. And blending the depth image of each object instance with a template depth map can lead to incorrect final depth maps.<\/p>\n\n\n\n

Alternate & better approach <\/strong>– Use a 3D rendered to render a 3D scene with multiple object meshes inside a template container mesh. The resulting image is more realistic and a custom rasterizer & shader can be used to generate accurate depth maps and object instance masks.<\/p>\n\n\n\n

We use this approach and implement it with PyTorch3D as the rendering framework<\/p>\n\n\n\n

Prerequisites<\/strong> <\/p>\n\n\n\n