{"id":63,"date":"2022-12-19T07:12:11","date_gmt":"2022-12-19T07:12:11","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/?page_id=63"},"modified":"2022-12-19T18:21:45","modified_gmt":"2022-12-19T18:21:45","slug":"introudction","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/","title":{"rendered":"Introudction"},"content":{"rendered":"\n<p><strong>Motivation<\/strong><\/p>\n\n\n\n<p>The accurate reflection of model prediction confidence is critical in the trustworthy usage of model artificial intelligence, especially in deep learning systems, often essentially considered black boxes. Thus the evaluation of prediction uncertainty is meant to let the system indicate \u201cI am not sure\u201d when the input data is ambiguous or out of distribution that the model fails to give a confident prediction. Often, model prediction is over-confidence, even on the mistaken class, which prevents machine learning from being applied to more error-sensitive areas like space technology.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"390\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-1024x390.png\" alt=\"\" class=\"wp-image-96\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-1024x390.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-300x114.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-768x292.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-1536x585.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2.png 1686w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p><strong>Uncertainty Quantification<\/strong><\/p>\n\n\n\n<p>Uncertainty quantification is used to measure the difference between model-predicted confidence and actual performance. It&#8217;s a way of <strong>measuring if the network knows what it knows<\/strong><\/p>\n\n\n\n<p>In general, there are two kinds of uncertainty:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Epistemic uncertainty (model uncertainty) due to lack of knowledge( for instance, Out of-distribution (OOD) data). Often reducible by adding more data. It\u2019s sensor agnostic. It&#8217;s a modeling of how same models from different weight initialization vary in prediction when they were trained on the same dataset.<\/li><li>Aleatoric uncertainty ( data uncertainty) due to noise in data. Often not reducible by adding more data and is only meaningful in distribution. It\u2019s not sensor agnostic by definition.&nbsp;<\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"462\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-3-1024x462.png\" alt=\"\" class=\"wp-image-97\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-3-1024x462.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-3-300x135.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-3-768x347.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-3.png 1112w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p><strong>Problem statement<\/strong><\/p>\n\n\n\n<p>Previous methods most fail between the trade-off of computation efficiency and usefulness.  Most approaches currently rely on Monte-Carlo or other sampling methods to provide a distribution of predictions from which uncertainty can be calculated. This poses a real issue on run-time constrained scenarios such as self-driving where at worse, sampling increases runtime linearly with number of samples. For instance, model ensemble[1] and Mote-Carlo Dropout[2] is still the SOTA in this field. In this project, we propose to investigate ways to estimate both epistemic and aleatoric uncertainty in real-time detection with a deterministic single pass method. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"365\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-1-1024x365.png\" alt=\"\" class=\"wp-image-86\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-1-1024x365.png 1024w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-1-300x107.png 300w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-1-768x274.png 768w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-1-1536x548.png 1536w, https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-1.png 1822w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/figure>\n\n\n\n<p>Reference:<\/p>\n\n\n\n<p>[1] Gal, Yarin, and Zoubin Ghahramani. &#8220;Dropout as a bayesian approximation: Representing model uncertainty in deep learning.&#8221;\u00a0<em>international conference on machine learning<\/em>. PMLR, 2016.<\/p>\n\n\n\n<p>[2] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. &#8220;Simple and scalable predictive uncertainty estimation using deep ensembles.&#8221;\u00a0<em>Advances in neural information processing systems<\/em>\u00a030 (2017).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Motivation The accurate reflection of model prediction confidence is critical in the trustworthy usage of model artificial intelligence, especially in deep learning systems, often essentially considered black boxes. Thus the evaluation of prediction uncertainty is meant to let the system indicate \u201cI am not sure\u201d when the input data is ambiguous or out of distribution &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Introudction&#8221;<\/span><\/a><\/p>\n","protected":false},"author":154,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-63","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Introudction - Uncertainty in Image classification<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introudction - Uncertainty in Image classification\" \/>\n<meta property=\"og:description\" content=\"Motivation The accurate reflection of model prediction confidence is critical in the trustworthy usage of model artificial intelligence, especially in deep learning systems, often essentially considered black boxes. Thus the evaluation of prediction uncertainty is meant to let the system indicate \u201cI am not sure\u201d when the input data is ambiguous or out of distribution &hellip; Continue reading &quot;Introudction&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/\" \/>\n<meta property=\"og:site_name\" content=\"Uncertainty in Image classification\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-19T18:21:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-1024x390.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/\",\"name\":\"Introudction - Uncertainty in Image classification\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/wp-content\\\/uploads\\\/sites\\\/77\\\/2022\\\/12\\\/image-2-1024x390.png\",\"datePublished\":\"2022-12-19T07:12:11+00:00\",\"dateModified\":\"2022-12-19T18:21:45+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/wp-content\\\/uploads\\\/sites\\\/77\\\/2022\\\/12\\\/image-2.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/wp-content\\\/uploads\\\/sites\\\/77\\\/2022\\\/12\\\/image-2.png\",\"width\":1686,\"height\":642},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introudction\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/\",\"name\":\"Uncertainty in Image classification\",\"description\":\"Students: Jia Shi | Advisors: Deva Ramanan (CMU), Shu Kong (CMU, TAMU), Francesco Ferroni (Argo AI), Arun Balajee Vasudevan (CMU) | Sponsor: Argo AI\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2023team6\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introudction - Uncertainty in Image classification","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/","og_locale":"en_US","og_type":"article","og_title":"Introudction - Uncertainty in Image classification","og_description":"Motivation The accurate reflection of model prediction confidence is critical in the trustworthy usage of model artificial intelligence, especially in deep learning systems, often essentially considered black boxes. Thus the evaluation of prediction uncertainty is meant to let the system indicate \u201cI am not sure\u201d when the input data is ambiguous or out of distribution &hellip; Continue reading \"Introudction\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/","og_site_name":"Uncertainty in Image classification","article_modified_time":"2022-12-19T18:21:45+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-1024x390.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/","name":"Introudction - Uncertainty in Image classification","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2-1024x390.png","datePublished":"2022-12-19T07:12:11+00:00","dateModified":"2022-12-19T18:21:45+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-content\/uploads\/sites\/77\/2022\/12\/image-2.png","width":1686,"height":642},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/"},{"@type":"ListItem","position":2,"name":"Introudction"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/","name":"Uncertainty in Image classification","description":"Students: Jia Shi | Advisors: Deva Ramanan (CMU), Shu Kong (CMU, TAMU), Francesco Ferroni (Argo AI), Arun Balajee Vasudevan (CMU) | Sponsor: Argo AI","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/pages\/63","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/users\/154"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/comments?post=63"}],"version-history":[{"count":7,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/pages\/63\/revisions"}],"predecessor-version":[{"id":103,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/pages\/63\/revisions\/103"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2023team6\/wp-json\/wp\/v2\/media?parent=63"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}