{"id":23,"date":"2022-12-17T21:50:00","date_gmt":"2022-12-18T02:50:00","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2023team4\/?page_id=23"},"modified":"2023-05-09T04:10:06","modified_gmt":"2023-05-09T09:10:06","slug":"approaches","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2023team4\/approaches\/","title":{"rendered":"Our Approach"},"content":{"rendered":"\n
\"\"
Overall architecture of our approach<\/figcaption><\/figure>\n\n\n\n

Here we describe each component of our pipeline for estimating depth from focal stack. <\/p>\n\n\n\n

Multiplane Image (MPI)<\/h2>\n\n\n\n

We represent the scene with Multiplane Image, which is a set of fronto-parallel planes at different predefined depths of the scene. Each plane contains RGB value and alpha or transparency value at a specific pixel location and depth layer. RGB images could be rendered with alpha compositing, and disparity map can be generated by taking expected value of the disparity map.<\/p>\n\n\n\n

\"\"
Architecture and rendering process of a Multiplane Image<\/figcaption><\/figure>\n\n\n\n

We chose the MPI representation due to its fast-rendering speed, known depth value of each layer and ability to work with patch-based losses such as GAN loss, Perceptual or SSIM losses.<\/p>\n\n\n\n

Apply Defocus effect to MPI<\/h2>\n\n\n\n

To simulate defocus effect with MPI representation and reconstruct each image of a focal stack, we blur different parts of the scene with different amount of blur based on the difference between focal distance and depth at each layer of MPI. Specifically, we apply disk kernel with different diameter to each layer of the MPI. <\/p>\n\n\n\n

Based on the Thin Lens Model, given camera parameter aperture and focal length, and focal distance which vary among images from the focal stack, diameter of defocus is at a particular scene depth:<\/p>\n\n\n\n

aperture * |focal distance – depth| * focal length \/ (focal distance-focal length) \/ depth<\/p>\n\n\n\n

\"\"
Relation between blur radius and scene depth with projective camera<\/figcaption><\/figure>\n\n\n\n

The above figure describes the relationship between blur amount and depth under a projective camera model. In our experiment, we are using an orthographic camera to capture the focal stack. With such a camera, the relationship between blur radius and scene depth is linear<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

After convolving disk kernel of appropriate radius with each MPI layer, we perform alpha compositing of the layers to get the reconstruction. The reconstruction process is illustrated below:<\/p>\n\n\n\n

\"\"
Reconstruction of focal stack <\/figcaption><\/figure>\n\n\n\n

Optimization of MPIs<\/h2>\n\n\n\n

We compare the reconstructed image from the focal stack and the ground truth and optimize for the RGB and alpha values stored in each layer of the MPI through gradient descent by applying gradient of the loss with respect to the RGB and alpha values.<\/p>\n\n\n\n

The loss functions we use include the following:<\/p>\n\n\n\n