{"id":208,"date":"2020-12-16T23:52:18","date_gmt":"2020-12-16T23:52:18","guid":{"rendered":"http:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/?page_id=208"},"modified":"2020-12-17T01:23:29","modified_gmt":"2020-12-17T01:23:29","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n

We explored different fusion strategies in RGB-thermal image pairs, including single modality, early fusion, middle fusion and late fusion. Here is the design of different strategies:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

(a)Single-modal detector
We separately trained one detector for RGB, another for thermal. (for KAIST dataset. For FLIR, since there are no RGB annotations, we trained thermal only.)

(b)Early fusion
We directly concatenated RGB and thermal into 4 channels as input to the model, and trained the detector.

(c)Middle fusion
We trained two streams of feature extractor for RGB and thermal imagery, then concatenated the two features before region proposal network (RPN) and ROI heads.

(d)Late fusion
We proposed Bayesian late fusion to fuse the detection results from two independent modalities.<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Suppose there are two detection results from RGB and thermal detectors. We can fuse the two detection results according to the following methods:

(a)Pooling
Directly pooling all the results from the two modalities. However, this results in multiple detections overlapping the same ground-truth object

(b)Non-maximum suppression (NMS)
Find overlapping boxes, and assign higher scores to it as final output. However, NMS fails to \u201cfuse\u201d information from multiple modalities together, since each of the final detections are supported by only one modality and fail to incorporate the lower score.

(c)Average fusion
A straightforward way is to modify NMS is to average the scores of overlapping detections from different modalities. Yet, averaging scores will always decrease the higher score compared with NMS.

(d)Late fusion
\u25cfAssume conditional independence in the two modalities:<\/p>\n\n\n\n

p(x1, x2 | y) = p(x1|y)p(x2|y) <\/p>\n\n\n\n

where x1 is RGB modality, x2 is thermal modality, y is class label<\/p>\n\n\n\n

\u25cfBayes rule:<\/p>\n\n\n\n

p(y|x1, x2) = p(x1, x2|y) p(y) \/ p(x1, x2)<\/p>\n\n\n\n

\u25cfMulti-modal posterior <\/p>\n\n\n\n

log p(y|x1, x2) = log p(y|x1) + log p(y|x2) – log p(y) – constant<\/p>\n","protected":false},"excerpt":{"rendered":"

We explored different fusion strategies in RGB-thermal image pairs, including single modality, early fusion, middle fusion and late fusion. Here is the design of different strategies: (a)Single-modal detectorWe separately trained one detector for RGB, another for thermal. (for KAIST dataset. For FLIR, since there are no RGB annotations, we trained thermal only.) (b)Early fusionWe directly … <\/p>\n

Continue reading “Method”<\/span><\/a><\/p>\n","protected":false},"author":74,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-208","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - Object Detection in Infrared Images<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Object Detection in Infrared Images\" \/>\n<meta property=\"og:description\" content=\"We explored different fusion strategies in RGB-thermal image pairs, including single modality, early fusion, middle fusion and late fusion. Here is the design of different strategies: (a)Single-modal detectorWe separately trained one detector for RGB, another for thermal. (for KAIST dataset. For FLIR, since there are no RGB annotations, we trained thermal only.) (b)Early fusionWe directly … Continue reading "Method"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Object Detection in Infrared Images\" \/>\n<meta property=\"article:modified_time\" content=\"2020-12-17T01:23:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/overflow-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1431\" \/>\n\t<meta property=\"og:image:height\" content=\"520\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/\",\"name\":\"Method - Object Detection in Infrared Images\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/wp-content\\\/uploads\\\/sites\\\/33\\\/2020\\\/12\\\/overflow-1-1024x372.png\",\"datePublished\":\"2020-12-16T23:52:18+00:00\",\"dateModified\":\"2020-12-17T01:23:29+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/wp-content\\\/uploads\\\/sites\\\/33\\\/2020\\\/12\\\/overflow-1.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/wp-content\\\/uploads\\\/sites\\\/33\\\/2020\\\/12\\\/overflow-1.png\",\"width\":1431,\"height\":520},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/\",\"name\":\"Object Detection in Infrared Images\",\"description\":\"Students: Yi-Ting Chen, Jinghao Shi | Advisor: Deva Ramanan, Christoph Mertz, Shu Kong | Sponsor: Argo AI\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2020teamc\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Object Detection in Infrared Images","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Object Detection in Infrared Images","og_description":"We explored different fusion strategies in RGB-thermal image pairs, including single modality, early fusion, middle fusion and late fusion. Here is the design of different strategies: (a)Single-modal detectorWe separately trained one detector for RGB, another for thermal. (for KAIST dataset. For FLIR, since there are no RGB annotations, we trained thermal only.) (b)Early fusionWe directly … Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/","og_site_name":"Object Detection in Infrared Images","article_modified_time":"2020-12-17T01:23:29+00:00","og_image":[{"width":1431,"height":520,"url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/overflow-1.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/","name":"Method - Object Detection in Infrared Images","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/overflow-1-1024x372.png","datePublished":"2020-12-16T23:52:18+00:00","dateModified":"2020-12-17T01:23:29+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/overflow-1.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-content\/uploads\/sites\/33\/2020\/12\/overflow-1.png","width":1431,"height":520},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/","name":"Object Detection in Infrared Images","description":"Students: Yi-Ting Chen, Jinghao Shi | Advisor: Deva Ramanan, Christoph Mertz, Shu Kong | Sponsor: Argo AI","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages\/208","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/users\/74"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/comments?post=208"}],"version-history":[{"count":7,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages\/208\/revisions"}],"predecessor-version":[{"id":235,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/pages\/208\/revisions\/235"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2020teamc\/wp-json\/wp\/v2\/media?parent=208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}