{"id":218,"date":"2020-12-17T00:35:26","date_gmt":"2020-12-17T00:35:26","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/?page_id=218"},"modified":"2020-12-17T01:36:30","modified_gmt":"2020-12-17T01:36:30","slug":"experiment-results","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/","title":{"rendered":"Experiment Results"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\"> Multimodal Pedestrian Detection on KAIST <\/h3>\n\n\n\n<h4 class=\"wp-block-heading\"> Quantitative Compomparison<\/h4>\n\n\n\n<p> Quantitative comparison on KAIST measured by LAMR\u2193 in percentage, on the two KAIST test-sets (old and new). We follow the literature that we evaluate in a \u201creasonable setting\u201d [1], i.e., ignoring small or occluded persons. Our Bayesian Fusion approach (wtih bounding box fusion) is comparable in Table 1. We take reported numbers from [1] for most compared methods. Clearly, our Bayesian Fusion approach outperforms the prior methods by a large margin. Bolded numbers marks the best results <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"697\" height=\"479\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png\" alt=\"\" class=\"wp-image-236\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png 697w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1-300x206.png 300w\" sizes=\"auto, (max-width: 697px) 100vw, 697px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Ablation Study<\/h4>\n\n\n\n<p> Ablation study on KAIST new test-set under the \u201creasonable\u201d setting, measured by percent LAMR\u2193. Please see text for a detailed discussion, but overall, we find our proposed BayesFusion approach to outperform all other variants, including end-toend learned approaches such as Early and MidFusion. Fig. 5 shows the corresponding MR-FPPI curves. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"706\" height=\"408\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/1608165948-1.png\" alt=\"\" class=\"wp-image-237\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/1608165948-1.png 706w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/1608165948-1-300x173.png 300w\" sizes=\"auto, (max-width: 706px) 100vw, 706px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"> Quantitative Comparison <\/h4>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"724\" height=\"415\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081687461.png\" alt=\"\" class=\"wp-image-241\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081687461.png 724w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081687461-300x172.png 300w\" sizes=\"auto, (max-width: 724px) 100vw, 724px\" \/><figcaption> Qualitative results on three random testing examples in KAIST. Top: over RGB images, we overlay the detection results from our mid-fusion model. Bottom: on the thermal images, we show results from our best-performing Bayesian Fusion model. Green, red and blue boxes stand for true positives, false negative (mis-detected persons) and false positives. Visually, our Bayesian Fusion performs much better than the mid-fusion model <\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"> Multimodal Object Detection on FLIR <\/h3>\n\n\n\n<h4 class=\"wp-block-heading\"> Quantitative comparison<\/h4>\n\n\n\n<p>Quantitative comparison on FLIR measured by AP\u2191 in percentage with IoU&gt;0.5. Following the literature, we evaluate on the three categories annotated by FLIR. Perhaps surprisingly, end-to-end training on thermal images already outperforms all the prior methods, presumably because of better augmentations and a better pre-trained model (Faster-RCNN). Moreover, our fusion methods perform even better. Lastly, our Bayesian Fusion method performs the best. These results are comparable to Table 3.  <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"701\" height=\"424\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081661421-1.png\" alt=\"\" class=\"wp-image-239\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081661421-1.png 701w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081661421-1-300x181.png 300w\" sizes=\"auto, (max-width: 701px) 100vw, 701px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Ablation Study<\/h4>\n\n\n\n<p>Breakdown analysis on FLIR day\/night scenes (AP\u2191 in percentage with IoU&gt;0.5). As FLIR does not have day\/night tags on the images, we manually annotate them for this analysis. Clearly, incorporating RGB by our learning-based fusion methods notably improves performance on both day and night scenes. We explore late-fusion with detection outputs from our three models: Thermal, Early and Mid. We find all AvgScore, NMS and BayesFusion lead to better performance than the learning-based MidFusion model. Especially, BayesFusion performs the best; using bounding box fusion (bbox) improves further <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"699\" height=\"313\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081662161-1.png\" alt=\"\" class=\"wp-image-240\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081662161-1.png 699w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081662161-1-300x134.png 300w\" sizes=\"auto, (max-width: 699px) 100vw, 699px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Quantitative Comparison <\/h4>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"703\" height=\"585\" src=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081689061.png\" alt=\"\" class=\"wp-image-242\" srcset=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081689061.png 703w, https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081689061-300x250.png 300w\" sizes=\"auto, (max-width: 703px) 100vw, 703px\" \/><figcaption> Qualitative multimodal detection results on FLIR images. We show three examples (in columns) with RGB (top) thermal images (middle and bottom). We overlay the groundtruth annotations on the RGB, highlighting that RGB and thermal images are strongly unaligned. To avoid clutter, we do not mark class labels for the bounding boxes. On the thermal images, we show qualitative results from our thermal-only (mid-row) and best-performing BayesFusion (with bounding box fusion) model (bottom-row). Green, red and blue boxes stand for true positives, false negative (mis-detected persons) and false positives. Particularly from the third column, thermal-only model has many false negatives (or mis-detections), which are \u201cbicycles\u201d. Understandably, thermal images will not deliver strong signatures for bicycles, but RGB images do. This explains why our fusion model performs better in detecting bicycles. <\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Multimodal Pedestrian Detection on KAIST Quantitative Compomparison Quantitative comparison on KAIST measured by LAMR\u2193 in percentage, on the two KAIST test-sets (old and new). We follow the literature that we evaluate in a \u201creasonable setting\u201d [1], i.e., ignoring small or occluded persons. Our Bayesian Fusion approach (wtih bounding box fusion) is comparable in Table 1. &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Experiment Results&#8221;<\/span><\/a><\/p>\n","protected":false},"author":74,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-218","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Experiment Results - Object Detection in Infrared Images<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Experiment Results - Object Detection in Infrared Images\" \/>\n<meta property=\"og:description\" content=\"Multimodal Pedestrian Detection on KAIST Quantitative Compomparison Quantitative comparison on KAIST measured by LAMR\u2193 in percentage, on the two KAIST test-sets (old and new). We follow the literature that we evaluate in a \u201creasonable setting\u201d [1], i.e., ignoring small or occluded persons. Our Bayesian Fusion approach (wtih bounding box fusion) is comparable in Table 1. &hellip; Continue reading &quot;Experiment Results&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/\" \/>\n<meta property=\"og:site_name\" content=\"Object Detection in Infrared Images\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-17T01:36:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"697\" \/>\n\t<meta property=\"og:image:height\" content=\"479\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/\",\"name\":\"Experiment Results - Object Detection in Infrared Images\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/wp-content\\\/uploads\\\/sites\\\/33\\\/2020\\\/12\\\/16081657871-1.png\",\"datePublished\":\"2020-12-17T00:35:26+00:00\",\"dateModified\":\"2020-12-17T01:36:30+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/wp-content\\\/uploads\\\/sites\\\/33\\\/2020\\\/12\\\/16081657871-1.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/wp-content\\\/uploads\\\/sites\\\/33\\\/2020\\\/12\\\/16081657871-1.png\",\"width\":697,\"height\":479},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/experiment-results\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Experiment Results\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/\",\"name\":\"Object Detection in Infrared Images\",\"description\":\"Students: Yi-Ting Chen, Jinghao Shi | Advisor: Deva Ramanan, Christoph Mertz, Shu Kong | Sponsor: Argo AI\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Experiment Results - Object Detection in Infrared Images","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/","og_locale":"en_US","og_type":"article","og_title":"Experiment Results - Object Detection in Infrared Images","og_description":"Multimodal Pedestrian Detection on KAIST Quantitative Compomparison Quantitative comparison on KAIST measured by LAMR\u2193 in percentage, on the two KAIST test-sets (old and new). We follow the literature that we evaluate in a \u201creasonable setting\u201d [1], i.e., ignoring small or occluded persons. Our Bayesian Fusion approach (wtih bounding box fusion) is comparable in Table 1. &hellip; Continue reading \"Experiment Results\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/","og_site_name":"Object Detection in Infrared Images","article_modified_time":"2020-12-17T01:36:30+00:00","og_image":[{"width":697,"height":479,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/","name":"Experiment Results - Object Detection in Infrared Images","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png","datePublished":"2020-12-17T00:35:26+00:00","dateModified":"2020-12-17T01:36:30+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/16081657871-1.png","width":697,"height":479},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/experiment-results\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/"},{"@type":"ListItem","position":2,"name":"Experiment Results"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/","name":"Object Detection in Infrared Images","description":"Students: Yi-Ting Chen, Jinghao Shi | Advisor: Deva Ramanan, Christoph Mertz, Shu Kong | Sponsor: Argo AI","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages\/218","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/users\/74"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/comments?post=218"}],"version-history":[{"count":3,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages\/218\/revisions"}],"predecessor-version":[{"id":243,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages\/218\/revisions\/243"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/media?parent=218"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}