{"id":70,"date":"2021-12-10T03:58:11","date_gmt":"2021-12-10T03:58:11","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/?p=70"},"modified":"2021-12-10T05:27:37","modified_gmt":"2021-12-10T05:27:37","slug":"related-work","status":"publish","type":"post","link":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/","title":{"rendered":"Related Work"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\"><strong>Autoregressive modeling<\/strong><\/h3>\n\n\n\n<p>Autoregressive models [12] factorize the joint distribution over structured outputs into products of conditional distribution. Unlike GANs [9], these can serve as powerful density estimators [14], are more stable during training [13,14], and can generalize well on held-out data. They have been successfully leveraged for modeling distributions across domains, such as images[5,12,13], video, or language [16], and our work explores their benefits across a broad range of 3D generation tasks.<\/p>\n\n\n\n<p>Following their recent successes in autoregressive modeling[3,16] our work adapts a Transformer-based [17] architecture.  However, these approaches cannot directly be adopted to volumetric 3D representations due to their high resolutions. We build on the work by van den Oord et.al. [15] who proposed a method to learn quantized and compact latent representations for images using Vector-Quantized Variational AutoEncoder (VQ-VAE). Inspired by Esser et.al. [7] who learned autoregressive generation over the discrete VQ-VAE representations, our work extends these ideas to the domain of 3D shapes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Shape Completion<\/strong><\/h3>\n\n\n\n<p>Completing full shapes from partial inputs such as discrete parts, or single-view 3D, is an increasingly important task across robotics and graphics. Most recent approaches [1,4,18] formulate it as performing completion on point clouds and can infer plausible global shapes but have difficulty in either capturing fine-grained details, conditioning on sparse inputs, or generating diverse samples. Our work proposes an alternative approach using autoregressive shape priors.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Single View Reconstruction<\/strong><\/h3>\n\n\n\n<p>Inferring the 3D shape from a single image is an inherently ill-posed task. Several approaches have shown impressive single-view reconstruction results using voxels [6,8], point clouds [11,19],  and most recently implicit representations of 3D surfaces like SDFs [10,20] etc. However, these are often deterministic in nature and only generate a 3D single output. By treating image-based prediction as conditional distributions our work can capture the multi-modal aspect of conditional generation in a simple and elegant manner.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Language based Generation<\/strong><\/h3>\n\n\n\n<p>Language is a highly effective and parsimonious modality for describing real world shapes and objects. Chen et.al [2] proposed a method to learn a joint text-shape embedding, followed by a GAN [9] based generator for synthesizing 3D from text. However, generating shapes from text is a fundamentally multi-modal task, and a GAN based approach struggles to capture the multiple output modes. In contrast, our project aims to first learn a &#8216;naive&#8217; language guided conditional distribution and combine it with shape priors to generate diverse and plausible shapes.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">References<\/h5>\n\n\n\n<p><span style=\"color:#5c5959\" class=\"has-inline-color\">[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018.<br>[2] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In ACCV, 2018.<br>[3] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.<br>[4] Xuelin Chen, Baoquan Chen, and Niloy J Mitra. Unpaired point cloud completion on real scans using adversarial training. In ICLR, 2020.<br>[5] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In ICML, 2018<br>[6] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.<br>[7] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high resolution image synthesis. In CVPR, 2021.<br>[8] Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In CVPR,2017.<br>[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.<br>[10] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In CVPR, 2020. <br>[11] Priyanka Mandikal, Navaneet K. L., Mayank Agarwal, and Venkatesh Babu Radhakrishnan. 3d-lmnet: Latent embedding matching for accurate and diverse 3d point cloud reconstruction from a single image. In BMVC, 2018<br>[12] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016<br>[13] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: A pixelcnn implementation with discretized logistic mixture likelihood and other modifications. In ICLR, 2017<br>[14] Benigno Uria, Iain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In NeurIPS, 2013.<br>[15] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.<br>[16] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. NeurIPS, 2019.<br>[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017<br>[18] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In ICCV, 2021.<br>[19] Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In CVPR, 2020.<br>[20] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In NeurIPS, 2019.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Autoregressive modeling Autoregressive models [12] factorize the joint distribution over structured outputs into products of conditional distribution. Unlike GANs [9], these can serve as powerful density estimators [14], are more stable during training [13,14], and can generalize well on held-out data. They have been successfully leveraged for modeling distributions across domains, such as images[5,12,13], video, &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Related Work&#8221;<\/span><\/a><\/p>\n","protected":false},"author":100,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-70","post","type-post","status-publish","format-standard","hentry","category-related-work"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Related Work - Autoregressive Conditional Generation using Transformers<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Related Work - Autoregressive Conditional Generation using Transformers\" \/>\n<meta property=\"og:description\" content=\"Autoregressive modeling Autoregressive models [12] factorize the joint distribution over structured outputs into products of conditional distribution. Unlike GANs [9], these can serve as powerful density estimators [14], are more stable during training [13,14], and can generalize well on held-out data. They have been successfully leveraged for modeling distributions across domains, such as images[5,12,13], video, &hellip; Continue reading &quot;Related Work&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/\" \/>\n<meta property=\"og:site_name\" content=\"Autoregressive Conditional Generation using Transformers\" \/>\n<meta property=\"article:published_time\" content=\"2021-12-10T03:58:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-12-10T05:27:37+00:00\" \/>\n<meta name=\"author\" content=\"paritosm\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"paritosm\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/\"},\"author\":{\"name\":\"paritosm\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/#\\\/schema\\\/person\\\/0a6afd0e4ed412755a2749c183fbc369\"},\"headline\":\"Related Work\",\"datePublished\":\"2021-12-10T03:58:11+00:00\",\"dateModified\":\"2021-12-10T05:27:37+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/\"},\"wordCount\":806,\"articleSection\":[\"Related Work\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/\",\"name\":\"Related Work - Autoregressive Conditional Generation using Transformers\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/#website\"},\"datePublished\":\"2021-12-10T03:58:11+00:00\",\"dateModified\":\"2021-12-10T05:27:37+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/#\\\/schema\\\/person\\\/0a6afd0e4ed412755a2749c183fbc369\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/2021\\\/12\\\/10\\\/related-work\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Related Work\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/\",\"name\":\"Autoregressive Conditional Generation using Transformers\",\"description\":\"Students: Yen-Chi Cheng, Paritosh Mittal | Advisors: Maneesh Singh (Verisk), Shubham Tulsiani (CMU) | Sponsor: Verisk\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/#\\\/schema\\\/person\\\/0a6afd0e4ed412755a2749c183fbc369\",\"name\":\"paritosm\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e788c1e87110c6034ea67b5363857f33e54190080bcb97049b1dc788fa1bd9a9?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e788c1e87110c6034ea67b5363857f33e54190080bcb97049b1dc788fa1bd9a9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e788c1e87110c6034ea67b5363857f33e54190080bcb97049b1dc788fa1bd9a9?s=96&d=mm&r=g\",\"caption\":\"paritosm\"},\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2021teamb\\\/author\\\/paritosm\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Related Work - Autoregressive Conditional Generation using Transformers","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/","og_locale":"en_US","og_type":"article","og_title":"Related Work - Autoregressive Conditional Generation using Transformers","og_description":"Autoregressive modeling Autoregressive models [12] factorize the joint distribution over structured outputs into products of conditional distribution. Unlike GANs [9], these can serve as powerful density estimators [14], are more stable during training [13,14], and can generalize well on held-out data. They have been successfully leveraged for modeling distributions across domains, such as images[5,12,13], video, &hellip; Continue reading \"Related Work\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/","og_site_name":"Autoregressive Conditional Generation using Transformers","article_published_time":"2021-12-10T03:58:11+00:00","article_modified_time":"2021-12-10T05:27:37+00:00","author":"paritosm","twitter_card":"summary_large_image","twitter_misc":{"Written by":"paritosm","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/#article","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/"},"author":{"name":"paritosm","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/#\/schema\/person\/0a6afd0e4ed412755a2749c183fbc369"},"headline":"Related Work","datePublished":"2021-12-10T03:58:11+00:00","dateModified":"2021-12-10T05:27:37+00:00","mainEntityOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/"},"wordCount":806,"articleSection":["Related Work"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/","name":"Related Work - Autoregressive Conditional Generation using Transformers","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/#website"},"datePublished":"2021-12-10T03:58:11+00:00","dateModified":"2021-12-10T05:27:37+00:00","author":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/#\/schema\/person\/0a6afd0e4ed412755a2749c183fbc369"},"breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/2021\/12\/10\/related-work\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/"},{"@type":"ListItem","position":2,"name":"Related Work"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/","name":"Autoregressive Conditional Generation using Transformers","description":"Students: Yen-Chi Cheng, Paritosh Mittal | Advisors: Maneesh Singh (Verisk), Shubham Tulsiani (CMU) | Sponsor: Verisk","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/#\/schema\/person\/0a6afd0e4ed412755a2749c183fbc369","name":"paritosm","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/e788c1e87110c6034ea67b5363857f33e54190080bcb97049b1dc788fa1bd9a9?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/e788c1e87110c6034ea67b5363857f33e54190080bcb97049b1dc788fa1bd9a9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e788c1e87110c6034ea67b5363857f33e54190080bcb97049b1dc788fa1bd9a9?s=96&d=mm&r=g","caption":"paritosm"},"url":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/author\/paritosm\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/posts\/70","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/comments?post=70"}],"version-history":[{"count":14,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/posts\/70\/revisions"}],"predecessor-version":[{"id":105,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/posts\/70\/revisions\/105"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/media?parent=70"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/categories?post=70"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2021teamb\/wp-json\/wp\/v2\/tags?post=70"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}