{"id":2,"date":"2020-05-05T17:59:18","date_gmt":"2020-05-05T17:59:18","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/?page_id=2"},"modified":"2020-12-16T18:54:11","modified_gmt":"2020-12-16T18:54:11","slug":"sample-page","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/","title":{"rendered":"Project Description"},"content":{"rendered":"\n

Overview<\/strong><\/p>\n\n\n\n

The primary goal of this project is to achieve high-fidelity 3D reconstruction by learning implicit representations. In contrast to the existing literature, we aim to design a method that can be used to learn these representations<\/em><\/strong> for any arbitrary shape, be it open\/closed, or shapes which have ill-defined genus<\/em><\/strong>. The applications of this method include high-fidelity reconstruction of damaged vehicles<\/em><\/strong> for insurance purposes, and automatic computation of building roof surface area<\/em><\/strong> via 3D reconstruction from aerial imagery. <\/p>\n\n\n\n

Project is Sponsored by Verisk Analytics<\/em><\/strong> and advised by Prof. Laszlo Je<\/em>ni<\/strong> at CMU and Dr. Maneesh Singh<\/em><\/strong> from Verisk.<\/p>\n\n\n\n

Approach<\/strong><\/p>\n\n\n\n

We disentangle the implicit representation of a shape into an unsigned distance field uDF<\/em> and a normal vector field nVF<\/em>. This decomposition enables us to represent both open and closed shapes with arbitrary topology, which, to the best of our knowledge, is a significant improvement over existing methods. We aim to model these functions using feed-forward networks with non-linear activation functions. <\/p>\n\n\n\n

We begin with a formal definition of a uDF<\/em>: it’s a function that outputs the closest unsigned distance to the surface from any given point in 3D space. Note that this is in contrast to an sDF<\/em> which is meant to output negative distances inside the shape and positive distances outside. Several notable current approaches for representing shapes, thus bound to the assumption that the underlying surface is watertight. We argue that an sDF<\/em> can inherently describe only closed shapes. To relax this assumption, we model a 3D shape using a uDF<\/em> which can represent both watertight and non-watertight shapes equally well. <\/p>\n\n\n\n

\"uDF(\boldsymbol{x})<\/p>\n\n\n\n

\"\"
Figure 1: Overview <\/strong><\/figcaption><\/figure>\n\n\n\n

As can be noted, trivially removing the sign of the sDF<\/em> leads to a uDF<\/em>. However, this sacrifices some important properties of the sDF<\/em> leading to a new set of challenges which we need to address. (1) The uD<\/em>F is non-differentiable at the surface (see Figure 2) implying it can’t be reliably trained using points sampled from the surface. We overcome this by never actually sampling the training data points on the surface but slightly away from the surface; (2) Unlike sDFs<\/em>, we cannot reliably extract the surface normal from a uDF<\/em> due to its non-differentiability (See Figure 2). Since estimation of high quality surface normals is important for several downstream tasks, we propose to learn a normal vector field (nVF<\/em>), which, for any 3D location, \"x\", represents the nVF<\/em> normal to the surface point closest to \"x\". Formally,<\/p>\n\n\n\n

\"nVF(\boldsymbol{x})
\"\boldsymbol{v}<\/p>\n\n\n\n

\"\"
Figure 2: Non-Differentiability of uDF<\/strong><\/figcaption><\/figure>\n\n\n\n

Given a 3D shape represented by the noisy triangle soup, we construct training samples, \"\mathcal{P}\", which contain a point, \"\boldsymbol{x}\", the uDF<\/em> and the nVF<\/em> evaluated at \"\boldsymbol{x}\"<\/p>\n\n\n\n

\"\mathcal{P}
\"\boldsymbol{v}=nVF(\boldsymbol{x})\"<\/p>\n\n\n\n

We first densely sample a set of {points, surface normal} pairs from the triangle soup, by uniformly sampling points on each triangle face. Let’s call this set of points $ latex \\mathcal{X}= {(\\boldsymbol{x_s}, \\boldsymbol{v_s})}$. Since each point is sampled from a triangle face, the normal to the triangle face provides the associated surface normal for that point. <\/p>\n\n\n\n

Given this set $ latex \\mathcal{X}$, the set \"\mathcal{P}\" is constructed by sampling points \"\boldsymbol{x}\" in 3D space and finding the nearest corresponding point in \"\mathcal{X}\" to construct the training sample \"(\boldsymbol{x},. The set \"\mathcal{P}\" is used train the DNNs to approximate the uDF<\/em> and the nVF<\/em>. More concretely, we train the DNN \"f_{\theta_d}\" to approximate uDF<\/em>, and DNN \"f_{\theta_n}\" to approximate nVF<\/em>.<\/p>\n\n\n\n

Before we describe how \"f_{\theta_n}\" is trained, please note that uDF<\/em> naturally correspond to unoriented surfaces (which are also logically necessitated by open surfaces). However, for most ray-casting applications this is not an issue as the direction of the first intersected surface can be chosen based on the direction of the ray. So, the ambiguity of \"n\" or \"-n\" can be handled. This implies a \"\textit{modulo representation in the DNN suffices. However, such a representation needs to be learnt from a noisy triangle soup with oriented surface normals with possible directional incoherence (in the \"\textit{modulo sense) between adjacent triangles (See Figure 3). To allow for this, we optimize the minimum of the two possible losses, computed from each \"n\" or \"-n\". More concretely, <\/p>\n\n\n\n

\"\mathcal{L}_{nVF}^{(1)}
\"\mathcal{L}_{nVF}^{(2)}
\"\mathcal{L}_{nVF}<\/p>\n\n\n\n

This allows for the network to learn surface normals \"\textit{modulo. The incoherence in the noisy triangle soup is handled by the continuity property of the DNNs and, practically, coherent normal fields are learnt as verified in our experiments. Thus, after training, the zero-level set of \"f_{\theta_d}\" which approximates the uDF<\/em>, represents points on the surface, while \"f_{\theta_n}\" approximating nVF<\/em>, represents the surface normals of the corresponding points on this level set. <\/p>\n\n\n\n

\"\"
Figure 3<\/strong>: Adjacent disconnect triangles<\/strong><\/figcaption><\/figure>\n\n\n\n

Sphere Tracing uDFs<\/strong><\/p>\n\n\n\n

Sphere tracing is a standard technique to render images from a distance field that represents the shape. To create an image, rays are cast from the focal point of the camera, and their intersection with the scene is computed using sphere tracing. Roughly speaking, irradiance\/ radiance computations are performed at the point of intersection to obtain the color of the pixel for that ray. <\/p>\n\n\n\n

The sphere tracing process can be described as follows: given a ray, \"\boldsymbol{r}\", originating at point, \"\boldsymbol{p_0}\", iterative marching along the ray is performed to obtain its intersection with the surface. In the first iteration, this translates to taking a step along the ray with a step size of \"\textit{uDF}(\boldsymbol{p_0})\" to obtain the next point \"\boldsymbol{p_1}. Since \"uDF(\boldsymbol{p_0})\" is the smallest distance to the surface, the line segment \"[\boldsymbol{p_0}, of the ray is guaranteed not to intersect the surface (\"\boldsymbol{p_1}\" can touch but not transcend the surface). The above step is iterated \"i\" times till \"\boldsymbol{p_i}\" is \"\epsilon\" close to the surface. The i-th iteration is given by \"\boldsymbol{p_i} and the stopping criteria \"uDF(\boldsymbol{p_i})\leq\epsilon\". <\/p>\n\n\n\n

Note that for uDF<\/em>, the above procedure can be used to get close to the surface but doesn’t obtain a point on the surface. One we are close enough to the surface, we can use a local planarity assumption (without loss of generalization) to obtain the intersection estimate. This is illustrated in Figure 4 and is obtained in the following manner: if we stop the sphere tracing of the uDF<\/em> at a point \"\boldsymbol{p_i}\", we evaluate the nVF<\/em> as \"\boldsymbol{n}, and compute the cosine of the angle between the nVF<\/em> and the ray direction. The estimate is then obtained as \"\boldsymbol{p_{proj}}. <\/p>\n\n\n\n

\"\"
Figure 4: Sphere Tracing uDFs<\/strong><\/figcaption><\/figure>\n\n\n\n

Benefit of using nVFs<\/strong><\/p>\n\n\n\n

In Figure 5, we visualize ray casting using normals obtained by differentiating the uDF<\/em> (Left) and ray casting using the surface normals estimated by the nVF<\/em> (Right). This high-quality rendering empirically validates the need for learning an nVF<\/em> alongside a uDF<\/em>.<\/p>\n\n\n\n

\"\"
Figure 5: High Quality Rendering using nVF<\/strong><\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

Overview The primary goal of this project is to achieve high-fidelity 3D reconstruction by learning implicit representations. In contrast to the existing literature, we aim to design a method that can be used to learn these representations for any arbitrary shape, be it open\/closed, or shapes which have ill-defined genus. The applications of this method … <\/p>\n

Continue reading “Project Description”<\/span><\/a><\/p>\n","protected":false},"author":68,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"yoast_head":"\nProject Description - Learning Implicit 3D Representation for Arbitrary Shapes<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Project Description - Learning Implicit 3D Representation for Arbitrary Shapes\" \/>\n<meta property=\"og:description\" content=\"Overview The primary goal of this project is to achieve high-fidelity 3D reconstruction by learning implicit representations. In contrast to the existing literature, we aim to design a method that can be used to learn these representations for any arbitrary shape, be it open\/closed, or shapes which have ill-defined genus. The applications of this method … Continue reading "Project Description"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/\" \/>\n<meta property=\"og:site_name\" content=\"Learning Implicit 3D Representation for Arbitrary Shapes\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-16T18:54:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/38\/2020\/12\/image-1.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/\",\"name\":\"Project Description - Learning Implicit 3D Representation for Arbitrary Shapes\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/wp-content\\\/uploads\\\/sites\\\/38\\\/2020\\\/12\\\/image-1.png\",\"datePublished\":\"2020-05-05T17:59:18+00:00\",\"dateModified\":\"2020-12-16T18:54:11+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/wp-content\\\/uploads\\\/sites\\\/38\\\/2020\\\/12\\\/image-1.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/wp-content\\\/uploads\\\/sites\\\/38\\\/2020\\\/12\\\/image-1.png\",\"width\":759,\"height\":435},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Project Description\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/\",\"name\":\"Learning Implicit 3D Representation for Arbitrary Shapes\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamh\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Project Description - Learning Implicit 3D Representation for Arbitrary Shapes","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/","og_locale":"en_US","og_type":"article","og_title":"Project Description - Learning Implicit 3D Representation for Arbitrary Shapes","og_description":"Overview The primary goal of this project is to achieve high-fidelity 3D reconstruction by learning implicit representations. In contrast to the existing literature, we aim to design a method that can be used to learn these representations for any arbitrary shape, be it open\/closed, or shapes which have ill-defined genus. The applications of this method … Continue reading \"Project Description\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/","og_site_name":"Learning Implicit 3D Representation for Arbitrary Shapes","article_modified_time":"2020-12-16T18:54:11+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/38\/2020\/12\/image-1.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/","name":"Project Description - Learning Implicit 3D Representation for Arbitrary Shapes","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/38\/2020\/12\/image-1.png","datePublished":"2020-05-05T17:59:18+00:00","dateModified":"2020-12-16T18:54:11+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/38\/2020\/12\/image-1.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/38\/2020\/12\/image-1.png","width":759,"height":435},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/"},{"@type":"ListItem","position":2,"name":"Project Description"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/","name":"Learning Implicit 3D Representation for Arbitrary Shapes","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/users\/68"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/comments?post=2"}],"version-history":[{"count":26,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":139,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/2\/revisions\/139"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/media?parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}