{"id":57,"date":"2026-05-07T00:39:46","date_gmt":"2026-05-07T00:39:46","guid":{"rendered":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/?page_id=57"},"modified":"2026-05-08T01:18:07","modified_gmt":"2026-05-08T01:18:07","slug":"method","status":"publish","type":"page","link":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/","title":{"rendered":"Method"},"content":{"rendered":"\n<h3 class=\"wp-block-heading\"><strong>Overal pipeline<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img decoding=\"async\" src=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/pipeline.png\" alt=\"\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Semantic Constraint Extraction<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/semantic_constraint.png\" alt=\"\" \/><\/figure>\n\n\n\n<p class=\"has-medium-font-size\">Our system first interprets the user\u2019s task instruction together with multi-view RGB-D observations. A vision-language model identifies task-relevant objects, spatial relationships, and potential safety risks, converting open-ended semantic instructions into structured constraint predicates.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: Geometric Grounding<\/strong><\/h3>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-28f84493 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/agentview.png\" alt=\"\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-large is-resized\"><img decoding=\"async\" src=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/sq.png\" alt=\"\" style=\"aspect-ratio:0.9630753667172484;width:285px;height:auto\" \/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<p class=\"has-medium-font-size\">The extracted objects and relationships are grounded in 3D using depth observations and superquadric fitting. This creates compact geometric abstractions of safety-relevant objects, allowing language-level constraints to be represented as spatial constraints in the robot\u2019s workspace.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Safe Action Correction<\/strong><\/h3>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-28f84493 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-video\"><video controls src=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/2026_04_17-18_38_02--openvla_oft--episode=1--success=False--safety=I--task=pick_up_the_black_bowl_between_the_plate_and_the_r.mp4\"><\/video><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-video\"><video controls src=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/2026_04_26-12_55_08--pi05_libero--episode=1--success=True--safety=I--task=pick_up_the_black_bowl_between_the_plate_and_the_r.mp4\"><\/video><\/figure>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-28f84493 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p class=\"has-text-align-center\">Without CBF<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p class=\"has-text-align-center\">With CBF<\/p>\n<\/div>\n<\/div>\n\n\n\n<p>A CBF-QP module monitors the nominal action proposed by the VLA policy and minimally modifies it only when safety is at risk. This preserves the original task behavior as much as possible while preventing unsafe motions near fragile or sensitive objects.<\/p>\n\n\n\n<p><strong>Next steps:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Logic-based semantic constraint generation\n<ul class=\"wp-block-list\">\n<li>Design a structured logic language that converts human instructions and VLM scene outputs into executable safety constraints.<\/li>\n\n\n\n<li>Example: \u201cDo not move the glass over the laptop\u201d \u2192 avoid(above(glass, laptop))<\/li>\n\n\n\n<li>This logic representation can help generalize across new objects, relations, and safety scenarios.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Build a situational safety benchmark to evaluate whether VLA models can understand and follow context-dependent safety rules.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overal pipeline Step 1: Semantic Constraint Extraction Our system first interprets the user\u2019s task instruction together with multi-view RGB-D observations. A vision-language model identifies task-relevant objects, spatial relationships, and potential safety risks, converting open-ended semantic instructions into structured constraint predicates. Step 2: Geometric Grounding The extracted objects and relationships are grounded in 3D using depth [&hellip;]<\/p>\n","protected":false},"author":294,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-57","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Method - Situational Safety for Embodied Agents<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Method - Situational Safety for Embodied Agents\" \/>\n<meta property=\"og:description\" content=\"Overal pipeline Step 1: Semantic Constraint Extraction Our system first interprets the user\u2019s task instruction together with multi-view RGB-D observations. A vision-language model identifies task-relevant objects, spatial relationships, and potential safety risks, converting open-ended semantic instructions into structured constraint predicates. Step 2: Geometric Grounding The extracted objects and relationships are grounded in 3D using depth [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/\" \/>\n<meta property=\"og:site_name\" content=\"Situational Safety for Embodied Agents\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-08T01:18:07+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/pipeline.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/\",\"name\":\"Method - Situational Safety for Embodied Agents\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/www.contrib.andrew.cmu.edu\\\/~jisuq\\\/mscv\\\/pipeline.png\",\"datePublished\":\"2026-05-07T00:39:46+00:00\",\"dateModified\":\"2026-05-08T01:18:07+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/#primaryimage\",\"url\":\"http:\\\/\\\/www.contrib.andrew.cmu.edu\\\/~jisuq\\\/mscv\\\/pipeline.png\",\"contentUrl\":\"http:\\\/\\\/www.contrib.andrew.cmu.edu\\\/~jisuq\\\/mscv\\\/pipeline.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/method\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Method\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/#website\",\"url\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/\",\"name\":\"Situational Safety for Embodied Agents\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mscvprojects.ri.cmu.edu\\\/2026teamf14\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Method - Situational Safety for Embodied Agents","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/","og_locale":"en_US","og_type":"article","og_title":"Method - Situational Safety for Embodied Agents","og_description":"Overal pipeline Step 1: Semantic Constraint Extraction Our system first interprets the user\u2019s task instruction together with multi-view RGB-D observations. A vision-language model identifies task-relevant objects, spatial relationships, and potential safety risks, converting open-ended semantic instructions into structured constraint predicates. Step 2: Geometric Grounding The extracted objects and relationships are grounded in 3D using depth [&hellip;]","og_url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/","og_site_name":"Situational Safety for Embodied Agents","article_modified_time":"2026-05-08T01:18:07+00:00","og_image":[{"url":"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/pipeline.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/","name":"Method - Situational Safety for Embodied Agents","isPartOf":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/#primaryimage"},"image":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/#primaryimage"},"thumbnailUrl":"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/pipeline.png","datePublished":"2026-05-07T00:39:46+00:00","dateModified":"2026-05-08T01:18:07+00:00","breadcrumb":{"@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/#primaryimage","url":"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/pipeline.png","contentUrl":"http:\/\/www.contrib.andrew.cmu.edu\/~jisuq\/mscv\/pipeline.png"},{"@type":"BreadcrumbList","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/method\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/"},{"@type":"ListItem","position":2,"name":"Method"}]},{"@type":"WebSite","@id":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/#website","url":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/","name":"Situational Safety for Embodied Agents","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/pages\/57","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/users\/294"}],"replies":[{"embeddable":true,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/comments?post=57"}],"version-history":[{"count":8,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/pages\/57\/revisions"}],"predecessor-version":[{"id":95,"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/pages\/57\/revisions\/95"}],"wp:attachment":[{"href":"https:\/\/mscvprojects.ri.cmu.edu\/2026teamf14\/wp-json\/wp\/v2\/media?parent=57"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}