Method

Overal pipeline

Step 1: Semantic Constraint Extraction

Our system first interprets the user’s task instruction together with multi-view RGB-D observations. A vision-language model identifies task-relevant objects, spatial relationships, and potential safety risks, converting open-ended semantic instructions into structured constraint predicates.

Step 2: Geometric Grounding

The extracted objects and relationships are grounded in 3D using depth observations and superquadric fitting. This creates compact geometric abstractions of safety-relevant objects, allowing language-level constraints to be represented as spatial constraints in the robot’s workspace.

Step 3: Safe Action Correction

A CBF-QP module monitors the nominal action proposed by the VLA policy and minimally modifies it only when safety is at risk. This preserves the original task behavior as much as possible while preventing unsafe motions near fragile or sensitive objects.

Next steps:

• Logic-based semantic constraint generation
  • Design a structured logic language that converts human instructions and VLM scene outputs into executable safety constraints.
  • Example: “Do not move the glass over the laptop” → avoid(above(glass, laptop))
  • This logic representation can help generalize across new objects, relations, and safety scenarios.
• Build a situational safety benchmark to evaluate whether VLA models can understand and follow context-dependent safety rules.