{"id":86,"date":"2021-12-10T05:23:01","date_gmt":"2021-12-10T05:23:01","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/?p=86"},"modified":"2021-12-10T22:23:59","modified_gmt":"2021-12-10T22:23:59","slug":"approach","status":"publish","type":"post","link":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/approach\/","title":{"rendered":"Approach"},"content":{"rendered":"\n
\"\"
Figure 1: Overview of Autoregressive Modeling<\/strong>. (top)<\/strong> We use a VQ-VAE to extract a low-dimensional discrete representation of 3D shapes. Using a patch-wise encoder enables independently encoding local context and allows downstream tasks with partial observations. (bottom)<\/strong> We learn a transformer-based autoregressive model over the latent representation. Using randomized sampling orders in allows learning a ‘non-sequential’ autoregressive shape prior that can condition on arbitrary set of partial latent observations.<\/figcaption><\/figure>\n\n\n\n

Discretized Latent Space for 3D Shapes<\/strong><\/h1>\n\n\n\n

To learn an effective autoregressive model, we aim to reduce the high-dimensional continuous 3D shape representation to a lower-dimensional discrete latent space. Towards this, we adapt the VQ-VAE [1] framework and learn a 3D-VQ-VAE whose encoder E\u03c8<\/sub> can compute the desired low-dimensional representation, and the decoder D\u03c8<\/sub> can map this latent space back to\/ 3D shapes. Given a 3D shape X<\/strong> with spatial dimension of D3<\/sup> we have<\/p>\n\n\n\n

\"\"
Eq:1<\/figcaption><\/figure><\/div>\n\n\n\n

where VQ is the Vector Quantization step that maps a vector to the nearest element in the codebook. The encoder E\u03c8<\/sub> has a very large receptive field. Unfortunately, this is not a desirable property for tasks such as shape completion since the latent codes for encoded partial shapes may differ significantly from those of the encoded full shape — thus partial observations of shape may not correspond to partial observations of latent variables. To overcome this challenge, we propose Patch-wise Encoding VQ-VAE or P-VQ-VAE that encodes the local shape regions independently<\/em>, while decoding them jointl<\/em>y — this allows the discrete encodings to only depend on local context.<\/p>\n\n\n\n

Non-sequential Autoregressive Modeling<\/strong><\/h1>\n\n\n\n

The latent space Z<\/strong> is a 3D grid of tokens representing the original 3D shape. We can thus reduce the task of learning the distribution over continuous 3D shapes to learning p(Z<\/strong>). An autoregressive model like transformer can model this distribution by factoring it as a product of location specific conditionals. However, this factorization assumes a fixed ordering in which the tokens are observed\/generated. This is generally not true for conditioning tasks like shape completion, reconstruction etc. We follow the observation from [2] that the joint distribution p(Z<\/strong>) can be factorized into terms of the form p(zi<\/sub><\/strong>| O<\/strong>), where O<\/strong> is a random set of observed variables. Hence we use a randomly permuted sequence of latent variables {zg1<\/strong><\/sub>, zg2<\/strong><\/sub>, z<\/em>g3<\/strong><\/sub>, …} for autoregressively modeling the distribution p(z<\/strong>):<\/p>\n\n\n\n

\"\"
eq:2<\/figcaption><\/figure><\/div>\n\n\n\n

The non-sequential autoregressive network models the distribution over the latent variables Z<\/strong>, which can be mapped to full 3D shapes X = D\u03c8<\/sub>(Z)<\/strong><\/p>\n\n\n\n

Conditional Generation<\/strong><\/h1>\n\n\n\n