{"id":50,"date":"2023-05-09T01:03:35","date_gmt":"2023-05-09T01:03:35","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/?page_id=50"},"modified":"2023-12-13T00:43:57","modified_gmt":"2023-12-13T00:43:57","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n

We train a deep learning model, BodyMAP, to jointly predict the body mesh \u02c6M (3D pose & body shape), along with the 3D applied pressure map P . Specifically, our model takes the individual\u2019s gender g, the depth image d, and the 2D pressure image p as inputs. The depth image is captured by a depth camera situated above the bed, while the 2D pressure image is generated by a pressure sensing mattress system positioned beneath the individual. This arrangement captures complementary features regarding the body illustrated in the appendix Fig. 5, thereby enhancing the context available to the model for accurate predictions of both body mesh and pressure map.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Figure 3: BodyMap jointly predicts body mesh and 3D applied pressure map for an individual in-bed. (a) Model architecture that encodes depth d and 2D pressure image p to predict SMPL [30] parameters \u02c6\u03a8, used to reconstruct the SMPL mesh \u02c6M . Feature Indexing Module (FIM) accumulates features for the mesh vertices from the input images and ResNet features. Finally, PointNet predicts the 3D pressure map \u02c6P along the human body using the mesh features as input. (b) FIM overlays the predicted mesh over the ResNet feature maps and input images by mapping mesh vertex locations to pixel positions and then assigns features for each vertex. These are fused along with the mesh vertex locations and used for 3D pressure map prediction. (c) Visualized FIM\u2019s overlay and assign step.<\/p>\n\n\n\n

Figure 3: The human body is represented using the SMPL [30] mesh, simplifying the network\u2019s task to predict a handful of SMPL parameters for obtaining the mesh. A 3D pressure map is represented at the vertex level with a pressure value for each vertex of the human mesh, allowing for precise localization of high-pressure regions on the body. We train our models on both the BodyPressureSD dataset [9] of simulated humans in bed and the real-world SLP dataset [26, 28]. For both datasets, we have depth and pressure images, aligned with the 3D ground truth mesh for diverse poses (supine, left & right lateral) and multiple blanket thickness configurations.<\/p>\n\n\n\n

BodyMAP<\/strong><\/h2>\n\n\n\n

In this section, we detail the architecture of our model BodyMAP, as illustrated in Fig. 3. The depth and pressure images are resized, concatenated and processed together as
image channels by the model.<\/p>\n\n\n\n

Body mesh prediction:<\/strong><\/h2>\n\n\n\n

The input is first encoded using ResNet18 [15]. We then feed the latent features through
a multi-layer perceptron to predict the SMPL parameters \u02c6\u03a8. These parameters include body shape, joint angles, root joint translation, and root-joint rotation: \u02c6\u03a8 = [ \u02c6\u03b2, \u02c6\u0398, \u02c6s, \u02c6x, \u02c6y].
The SMPL parameters \u02c6\u03a8, in addition to the gender information g, serve as inputs to the SMPL embedding block [19].
This block does not contain any learned parameters and outputs a differentiable human body mesh \u02c6M with vertices \u02c6V , and 3D joint positions \u02c6S.<\/p>\n\n\n\n

Feature Indexing Module: <\/strong><\/h2>\n\n\n\n

We introduce Feature Indexing Module (FIM), depicted in Fig. 3(b), to accumulate features for each mesh vertex. As mentioned in [9], the mesh predictions are spatially registered with the corresponding input images. This registration provides pixel locations where each mesh vertex would project onto the input images [9]. FIM assigns features to each mesh vertex from both the input images and the latent ResNet features (before global average pooling) using the mentioned pixel locations (Fig. 3(c)). These features are fused with the vertex locations and are utilized for pressure map prediction. Pressure map prediction: To predict a per-vertex pressure map ( \u02c6P ), we employ PointNet [37] utilizing the latent features formed from FIM for each vertex as input. This establishes a strong correlation between the mesh prediction and pressure prediction, ensuring their consistency with each other. In a manner akin to point-based segmentation architectures [37, 38], ResNet features (after global average pooling) are fused with PointNet encoder features. This provides the model with an enhanced contextual understanding. The PointNet decoder then predicts a per-vertex binary contact value and pressure value. The per-vertex contact value serves as an indicator of whether each mesh vertex is in contact with the mattress. We use the predicted contact values to further tune predicted pressure values. Specifically, the 3D pressure map is finally estimated as a product of the binary contact with the corresponding pressure value for each vertex.<\/p>\n\n\n\n

Training strategy<\/strong><\/h3>\n\n\n\n

We train the network to jointly predict body mesh and 3D applied pressure map with the following loss:<\/p>\n\n\n\n


L = LSMPL <\/sub>+ \u03bb1<\/sub>Lv2v<\/sub> + \u03bb2<\/sub>LP3D<\/sub> + \u03bb3<\/sub> Lcontact <\/sub><\/code><\/p>\n\n\n\n


where LSMPL<\/sub> minimizes the absolute error on SMPL parameters \u02c6\u03a8 and squared error on the 3D joint positions \u02c6S. Lv2v<\/sub> (vertex-to-vertex loss) and LP3D<\/sub> minimize the squared error on the vertex positions and pressure map values respectively. Lcontact is applied as a cross entropy loss between predicted and ground truth 3D contact, where the ground truth 3D contact is obtained from all non-zero elements of the ground truth 3D pressure maps. The loss weighting coefficients are set empirically. <\/p>\n\n\n\n

BodyMAP-WS<\/h2>\n\n\n\n


BodyMAP-WS is a variant of the above BodyMAP model, learning without supervision for the 3D pressure map prediction. This model utilizes a pre-trained mesh regressor to obtain mesh predictions and ResNet image features as its primary inputs. Unlike BodyMAP that forms the final 3D pressure map as the product of binary contact value and pressure at each vertex, to be able to train without supervision, BodyMAP-WS instead predicts only the pressure value at each vertex. The network architecture follows a similar design as BodyMAP\u2019s pressure prediction part (Fig. 3) in its use of FIM and PointNet, and is further illustrated in the appendix Fig. 8. To facilitate learning of the accurate 3D pressure map, the model constructs a differentiable 2D projection of the predicted 3D pressure map by averaging the pressure values of the vertices that project to each pixel [9]. During training, the network aligns this 2D projection with the input pressure image, allowing for the implicit learning of the actual 3D applied pressure map onto the body mesh.
<\/p>\n\n\n\n

Training strategy<\/strong><\/h3>\n\n\n\n


To form the pre-trained mesh regressor, we first train the mesh regressor section of BodyMAP using supervision with LSMPL<\/sub> and Lv2v<\/sub>. Subsequently, BodyMAP-WS is trained
to leverage the frozen pre-trained mesh model\u2019s predicted ResNet features and mesh vertex locations as inputs, for predicting the 3D applied pressure map. This prediction is guided by the following loss function:<\/p>\n\n\n\n


L = LP2D<\/sub> + \u03bb1<\/sub>LPreg<\/sub><\/p>\n\n\n\n


Here, LPproj <\/sub>minimizes the squared error between the 2D projection of the predicted 3D pressure map and the input pressure image. The bed mattress is situated on the Z = 0 plane, and vertices predicted to be positioned above this plane (Z > 0) should ideally not have any applied pressure on them as they do not make contact with the mattress. To enforce this constraint, we utilize LPreg<\/sub>, a regularization term that penalizes for positive pressure on these vertices by minimizing the norm of predicted pressure values for these vertices.<\/p>\n","protected":false},"excerpt":{"rendered":"

We train a deep learning model, BodyMAP, to jointly predict the body mesh \u02c6M (3D pose & body shape), along with the 3D applied pressure map P . Specifically, our model takes the individual\u2019s gender g, the depth image d, and the 2D pressure image p as inputs. The depth image is captured by a … <\/p>\n

Continue reading “Method”<\/span><\/a><\/p>\n","protected":false},"author":188,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-50","page","type-page","status-publish","hentry"],"yoast_head":"\nMethod - Multimodal Visual Learning for Pressure Ulcer Prevention<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Multimodal Visual Learning for Pressure Ulcer Prevention\" \/>\n<meta property=\"og:description\" content=\"We train a deep learning model, BodyMAP, to jointly predict the body mesh \u02c6M (3D pose & body shape), along with the 3D applied pressure map P . Specifically, our model takes the individual\u2019s gender g, the depth image d, and the 2D pressure image p as inputs. The depth image is captured by a … Continue reading "Method"\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Multimodal Visual Learning for Pressure Ulcer Prevention\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-13T00:43:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-content\/uploads\/sites\/98\/2023\/12\/model_full2-1024x418.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/\",\"name\":\"Method - Multimodal Visual Learning for Pressure Ulcer Prevention\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/wp-content\\\/uploads\\\/sites\\\/98\\\/2023\\\/12\\\/model_full2-1024x418.png\",\"datePublished\":\"2023-05-09T01:03:35+00:00\",\"dateModified\":\"2023-12-13T00:43:57+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/wp-content\\\/uploads\\\/sites\\\/98\\\/2023\\\/12\\\/model_full2.png\",\"contentUrl\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/wp-content\\\/uploads\\\/sites\\\/98\\\/2023\\\/12\\\/model_full2.png\",\"width\":1145,\"height\":467},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/\",\"name\":\"Multimodal Visual Learning for Pressure Ulcer Prevention\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/f23team21\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Multimodal Visual Learning for Pressure Ulcer Prevention","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Multimodal Visual Learning for Pressure Ulcer Prevention","og_description":"We train a deep learning model, BodyMAP, to jointly predict the body mesh \u02c6M (3D pose & body shape), along with the 3D applied pressure map P . Specifically, our model takes the individual\u2019s gender g, the depth image d, and the 2D pressure image p as inputs. The depth image is captured by a … Continue reading \"Method\"","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/","og_site_name":"Multimodal Visual Learning for Pressure Ulcer Prevention","article_modified_time":"2023-12-13T00:43:57+00:00","og_image":[{"url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-content\/uploads\/sites\/98\/2023\/12\/model_full2-1024x418.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/","name":"Method - Multimodal Visual Learning for Pressure Ulcer Prevention","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/#primaryimage"},"thumbnailUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-content\/uploads\/sites\/98\/2023\/12\/model_full2-1024x418.png","datePublished":"2023-05-09T01:03:35+00:00","dateModified":"2023-12-13T00:43:57+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/#primaryimage","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-content\/uploads\/sites\/98\/2023\/12\/model_full2.png","contentUrl":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-content\/uploads\/sites\/98\/2023\/12\/model_full2.png","width":1145,"height":467},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/","name":"Multimodal Visual Learning for Pressure Ulcer Prevention","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/pages\/50","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/users\/188"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/comments?post=50"}],"version-history":[{"count":10,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/pages\/50\/revisions"}],"predecessor-version":[{"id":181,"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/pages\/50\/revisions\/181"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/f23team21\/wp-json\/wp\/v2\/media?parent=50"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}