{"id":2,"date":"2022-01-07T14:09:41","date_gmt":"2022-01-07T14:09:41","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/?page_id=2"},"modified":"2022-12-16T03:39:42","modified_gmt":"2022-12-16T03:39:42","slug":"our-project","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/","title":{"rendered":"Capstone Project"},"content":{"rendered":"\n

Overview<\/h1>\n\n\n\n

Given a collection of bokeh images of a static<\/em> scene (shown left), how would we render a new image with a different bokeh setting from the same viewpoint<\/em> (shown right)? <\/p>\n\n\n\n

\n
\n
\n
\n
\"\"
Images captured under different camera settings<\/figcaption><\/figure>\n<\/div>\n\n\n\n
\n
\"\"
Images generated using new camera settings <\/figcaption><\/figure>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

We answer this common problem in the domain of post-capture image processing through our capstone project. Through this work, we enable re-photography<\/em> using a neural rendering approach.<\/p>\n\n\n\n

Motivation<\/h1>\n\n\n\n

People often capture interesting snapshots of their lives using their smartphones and digital cameras. In the spur of the moment, we might take multiple images focusing on some parts of the scene but later wished that we captured an image focusing on some other part of the scene.<\/p>\n\n\n\n

Our project aims to bridge this gap by modeling a subset of camera settings such that we can render images with novel camera settings (focus and aperture).<\/p>\n\n\n\n

Problem Statement<\/h1>\n\n\n\n

Given a bunch of images captured using varying camera settings (focus and aperture), we want to generate images with new values of those settings that still appear photorealistic.<\/p>\n\n\n\n

Approach<\/h1>\n\n\n\n

We first break this problem into two parts – inverse rendering and forward rendering.<\/p>\n\n\n\n

\"\"
High-level pipeline for our approach<\/figcaption><\/figure>\n\n\n\n

Inverse rendering<\/h3>\n\n\n\n

From a collection of images given as input, a neural representation of the scene is built using state-of-the-art approaches in neural rendering. We chose to experiment with two scene representations – Multi-Plane Images (MPI), a 2.5D representation of the world, and Neural Radiance Fields (NeRF), a continuous 3D representation of the world. Both approaches have their advantages and drawbacks which are explained in the Approach<\/a> section.<\/p>\n\n\n\n

Forward rendering<\/h3>\n\n\n\n

New images of a scene for a new set of camera settings can be generated from the neural scene representation. In the MPI representation, this can be achieved through multi-ray sampling. In the NeRF setting, volume rendering can be used to generate a new scene image. It is to be noted that in both representations, a thin-lens-based formulation is integrated into the rendering pipelines such that photorealistic defocus effects can be generated.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Our code is available at https:\/\/github.com\/nitchith\/Neural-Rephotography<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Overview Given a collection of bokeh images of a static scene (shown left), how would we render a new image with a different bokeh setting from the same viewpoint (shown right)? We answer this common problem in the domain of post-capture image processing through our capstone project. Through this work, we enable re-photography using a … <\/p>\n

Continue reading “Capstone Project”<\/span><\/a><\/p>\n","protected":false},"author":68,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-2","page","type-page","status-publish","hentry"],"yoast_head":"\nCapstone Project - Neural Re-Photography<\/title>\n<meta name=\"description\" content=\"Capstone project on modeling a neural camera using computer vision and computational photography.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Capstone Project - Neural Re-Photography\" \/>\n<meta property=\"og:description\" content=\"Capstone project on modeling a neural camera using computer vision and computational photography.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\" \/>\n<meta property=\"og:site_name\" content=\"Neural Re-Photography\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-16T03:39:42+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-content\/uploads\/sites\/57\/2022\/05\/image-35.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\",\"name\":\"Capstone Project - Neural Re-Photography\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/wp-content\\\/uploads\\\/sites\\\/57\\\/2022\\\/05\\\/image-35.png\",\"datePublished\":\"2022-01-07T14:09:41+00:00\",\"dateModified\":\"2022-12-16T03:39:42+00:00\",\"description\":\"Capstone project on modeling a neural camera using computer vision and computational photography.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/wp-content\\\/uploads\\\/sites\\\/57\\\/2022\\\/05\\\/image-35.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/wp-content\\\/uploads\\\/sites\\\/57\\\/2022\\\/05\\\/image-35.png\",\"width\":698,\"height\":516},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Capstone Project\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/\",\"name\":\"Neural Re-Photography\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2022team2\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Capstone Project - Neural Re-Photography","description":"Capstone project on modeling a neural camera using computer vision and computational photography.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2","og_locale":"en_US","og_type":"article","og_title":"Capstone Project - Neural Re-Photography","og_description":"Capstone project on modeling a neural camera using computer vision and computational photography.","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2","og_site_name":"Neural Re-Photography","article_modified_time":"2022-12-16T03:39:42+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-content\/uploads\/sites\/57\/2022\/05\/image-35.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2","name":"Capstone Project - Neural Re-Photography","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-content\/uploads\/sites\/57\/2022\/05\/image-35.png","datePublished":"2022-01-07T14:09:41+00:00","dateModified":"2022-12-16T03:39:42+00:00","description":"Capstone project on modeling a neural camera using computer vision and computational photography.","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2022team2"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-content\/uploads\/sites\/57\/2022\/05\/image-35.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-content\/uploads\/sites\/57\/2022\/05\/image-35.png","width":698,"height":516},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/"},{"@type":"ListItem","position":2,"name":"Capstone Project"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/","name":"Neural Re-Photography","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/pages\/2","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/users\/68"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/comments?post=2"}],"version-history":[{"count":43,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/pages\/2\/revisions"}],"predecessor-version":[{"id":391,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/pages\/2\/revisions\/391"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2022team2\/wp-json\/wp\/v2\/media?parent=2"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}