{"id":10,"date":"2022-12-15T04:24:34","date_gmt":"2022-12-15T04:24:34","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/?page_id=10"},"modified":"2023-05-09T03:18:41","modified_gmt":"2023-05-09T03:18:41","slug":"methods","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/","title":{"rendered":"Methods"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Image Restoration<\/h2>\n\n\n\n<p class=\"has-text-align-left\">Our image restoration pipeline aims to take in a foggy image as input and produce a clear image as output. For this, we use a transformer based model, TransWeather [1].<\/p>\n\n\n\n<p class=\"has-text-align-left\"><strong>TransWeather <\/strong>This model is built upon on a transformer architecture, taking in any weather-degraded image and outputting a weather-free image of the same scene. In addition to basic transformer blocks, intra-patch transformer blocks propagate patch-level features and information through the network. In theory, this model should be able to encompass all weather types, expanding upon previous works where unique encoder blocks are necessary. This is done so by increasing the number of parameters learned in the model. Furthermore, weather type queries are learned in the transformer decoders that attend to the weather type in the image. <\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-1024x391.png\" alt=\"\" class=\"wp-image-86\" width=\"528\" height=\"201\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-1024x391.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-300x114.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-768x293.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM.png 1232w\" sizes=\"auto, (max-width: 528px) 100vw, 528px\" \/><figcaption class=\"wp-element-caption\">TransWeather model architecture<\/figcaption><\/figure>\n<\/div>\n\n\n<p>Synthetic Image Generation<\/p>\n\n\n\n<p>The image restoration network requires similarity score between a ground truth (clear) image and restored image of the same scene to assess the quality of the restoration. Thus, we require synthetic foggy images to train the restoration model. We have a synthetic fog generation pipeline for both day time and night time images which was applied on the BBD100k[2] dataset.<\/p>\n\n\n\n<p> Synthetic fog generation follows the atmospheric scattering model [3]:<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.55.25-AM.png\" alt=\"\" class=\"wp-image-90\" width=\"205\" height=\"57\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.55.25-AM.png 480w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.55.25-AM-300x84.png 300w\" sizes=\"auto, (max-width: 205px) 100vw, 205px\" \/><\/figure>\n<\/div>\n\n\n<p>In the above equation, the foggy image <em>I(x)<\/em> is generated using the clear image <em>J(x)<\/em>, transmission map <em>t(x)<\/em>, atmospheric scattering <em>A<\/em>, scattering coefficient <em>\u03b2, <\/em>and depth map <em>d(x)<\/em>. We apply varying values of <em>\u03b2<\/em> to the data. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Weather Classification<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"383\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2023\/05\/Screenshot-2023-05-08-at-11.15.49-PM-1024x383.png\" alt=\"\" class=\"wp-image-169\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2023\/05\/Screenshot-2023-05-08-at-11.15.49-PM-1024x383.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2023\/05\/Screenshot-2023-05-08-at-11.15.49-PM-300x112.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2023\/05\/Screenshot-2023-05-08-at-11.15.49-PM-768x288.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2023\/05\/Screenshot-2023-05-08-at-11.15.49-PM-1536x575.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2023\/05\/Screenshot-2023-05-08-at-11.15.49-PM.png 1688w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p>The weather classification model is a multi-task classifier to determine not only the weather type, but also the time of day and road condition. The model itself uses a ResNet50 backbone with engineered features corresponding to saturation, dark channel prior, local contrast, and histograms of oriented gradients (HoG). These features, except for the latter, are directly computed and then passed to the backbone. The HoG feature is represented as a binned histogram and concatenated at the linear layers to form the predictions. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Resources<\/h2>\n\n\n\n<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-8cf370e7 wp-block-group-is-layout-flex\">\n<p>[1] Valanarasu, J. M. J., Yasarla, R., &amp; Patel, V. M. &#8220;Transweather: Transformer-based restoration of images degraded by adverse weather conditions.&#8221; <em>CVPR 2022<\/em><\/p>\n\n\n\n<p>[2] Yu, X., et al. (2020). BDD100K: A Large-scale Diverse Driving Video Database. arXiv preprint arXiv:1805.04687.<\/p>\n\n\n\n<p>[3] Narasimhan, Srinivasa G., and Shree K. Nayar. &#8220;Vision and the atmosphere.&#8221;&nbsp;<em>International journal of computer vision&nbsp;2002<\/em><\/p>\n\n\n\n<p>[4] Godard, C., et al. (2019). Digging Into Self-Supervised Monocular Depth Estimation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Image Restoration Our image restoration pipeline aims to take in a foggy image as input and produce a clear image as output. For this, we use a transformer based model, TransWeather [1]. TransWeather This model is built upon on a transformer architecture, taking in any weather-degraded image and outputting a weather-free image of the same &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Methods&#8221;<\/span><\/a><\/p>\n","protected":false},"author":149,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-10","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Methods - Autonomous Driving in Adverse Weather Conditions<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Methods - Autonomous Driving in Adverse Weather Conditions\" \/>\n<meta property=\"og:description\" content=\"Image Restoration Our image restoration pipeline aims to take in a foggy image as input and produce a clear image as output. For this, we use a transformer based model, TransWeather [1]. TransWeather This model is built upon on a transformer architecture, taking in any weather-degraded image and outputting a weather-free image of the same &hellip; Continue reading &quot;Methods&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/\" \/>\n<meta property=\"og:site_name\" content=\"Autonomous Driving in Adverse Weather Conditions\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-09T03:18:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-1024x391.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/\",\"name\":\"Methods - Autonomous Driving in Adverse Weather Conditions\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/wp-content\\\/uploads\\\/sites\\\/74\\\/2022\\\/12\\\/Screen-Shot-2022-12-16-at-3.29.56-AM-1024x391.png\",\"datePublished\":\"2022-12-15T04:24:34+00:00\",\"dateModified\":\"2023-05-09T03:18:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/wp-content\\\/uploads\\\/sites\\\/74\\\/2022\\\/12\\\/Screen-Shot-2022-12-16-at-3.29.56-AM.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/wp-content\\\/uploads\\\/sites\\\/74\\\/2022\\\/12\\\/Screen-Shot-2022-12-16-at-3.29.56-AM.png\",\"width\":1232,\"height\":470},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/methods\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Methods\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/\",\"name\":\"Autonomous Driving in Adverse Weather Conditions\",\"description\":\"Students: Lulu Ricketts and Roshini Rajesh Kannan | Advisor: Srinivasa Narasimhan (CMU)\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team3\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Methods - Autonomous Driving in Adverse Weather Conditions","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/","og_locale":"en_US","og_type":"article","og_title":"Methods - Autonomous Driving in Adverse Weather Conditions","og_description":"Image Restoration Our image restoration pipeline aims to take in a foggy image as input and produce a clear image as output. For this, we use a transformer based model, TransWeather [1]. TransWeather This model is built upon on a transformer architecture, taking in any weather-degraded image and outputting a weather-free image of the same &hellip; Continue reading \"Methods\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/","og_site_name":"Autonomous Driving in Adverse Weather Conditions","article_modified_time":"2023-05-09T03:18:41+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-1024x391.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/","name":"Methods - Autonomous Driving in Adverse Weather Conditions","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM-1024x391.png","datePublished":"2022-12-15T04:24:34+00:00","dateModified":"2023-05-09T03:18:41+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-content\/uploads\/sites\/74\/2022\/12\/Screen-Shot-2022-12-16-at-3.29.56-AM.png","width":1232,"height":470},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/methods\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/"},{"@type":"ListItem","position":2,"name":"Methods"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/","name":"Autonomous Driving in Adverse Weather Conditions","description":"Students: Lulu Ricketts and Roshini Rajesh Kannan | Advisor: Srinivasa Narasimhan (CMU)","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/pages\/10","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/users\/149"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/comments?post=10"}],"version-history":[{"count":12,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/pages\/10\/revisions"}],"predecessor-version":[{"id":171,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/pages\/10\/revisions\/171"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team3\/wp-json\/wp\/v2\/media?parent=10"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}