Verifiable Neural Safety Filters for Human-Robot Interaction Through Conformal Prediction

Paper: 2606.02562 Authors: Haimin Hu Categories: cs.RO, cs.AI, cs.LG, eess.SY

The Gap

Traditional safety filters for robots operate purely in physical state space, treating human behavior as unpredictable chaos and responding with extreme conservatism. BeliefSF (belief-space safety filters) improved this by letting robots actively learn about humans online and adjust safety margins accordingly—less uncertainty means less conservatism. But there’s a Catch-22: BeliefSF uses neural networks to handle high-dimensional belief spaces and relies on runtime inference to reduce uncertainty. Neural nets approximate. Inference can be wrong. How do you formally guarantee safety when both your filter and your learning module have errors?

Existing formal verification methods either (1) assume perfect inference, which is fantasy, or (2) apply standard conformal prediction uniformly across all belief states, which throws away the structure of belief-space filtering and produces overly conservative certificates that defeat the whole point.

Physical-space       BeliefSF              This paper
safety filter        (prior work)          (verified BeliefSF)
                                           
Human behavior    -> Learn belief    ->  Certify the whole
  |                    about human         pipeline with
  v                       |                inference errors
Over-conservative        v                      |
  safety              Adaptive               v
                      safety (but        Provable safety
                      unverified)        + less conservative

The Increment

One sentence: Before this paper, belief-space safety filters could be adaptive but not formally safe; after, they can be both adaptive and verifiably safe with high probability by explicitly modeling inference reliability.

Core Mechanism

The method has three stages. First, during offline training, you collect data from your inference module on various belief states and measure how often it gets things wrong. This gives you a calibration dataset that captures inference reliability across the belief space.

Second, you identify a “trusted region”—a subset of belief states where inference is demonstrably reliable based on your calibration data. The key insight: you don’t need perfect inference everywhere, just in the region where the safety filter will actually operate most of the time. If the robot’s belief lands outside this trusted region, fall back to a conservative baseline filter that doesn’t rely on inference.

Third, apply conformal prediction only within the trusted region. Standard conformal prediction would give you a safety certificate that works uniformly across all beliefs, which is wasteful because beliefs far from the trusted region are either rare or handled by the baseline anyway. By focusing verification where inference is reliable, you get a tighter safety bound—meaning the certified filter permits more actions while maintaining the same probabilistic safety guarantee.

Belief Space (high-dimensional)
    
    +----------------------------------+
    |  Rare beliefs                    |
    |  (inference unreliable)          |
    |    +---------------------+       |
    |    | Trusted Region      |       |
    |    | (inference reliable)|       |
    |    |                     |       |
    |    |  Most operation     |       |
    |    |  happens here       |       |
    |    |                     |       |
    |    | Apply conformal     |       |
    |    | prediction HERE     |       |
    |    +---------------------+       |
    |                                  |
    |  Outside: use conservative       |
    |  baseline filter                 |
    +----------------------------------+

Data flow:
Calibration dataset -> Identify trusted region -> 
  Conformal prediction on trusted region -> Safety certificate

Think of it like insurance underwriting. An insurance company doesn’t need perfect risk models for every possible customer—just reliable models for the customer profiles they actually serve. For edge cases (someone who wants to insure their pet iguana for a million dollars), they fall back to “sorry, we don’t cover that” rather than building a model. This paper does the same: build a tight safety certificate for the common case (trusted region where inference works), fall back to conservatism for the rare case (outside the trusted region), and use conformal prediction to stitch it together with formal probabilistic guarantees.

The structural metaphor: You’re building a bridge across a river. Standard conformal prediction says “make every beam strong enough to hold a tank, even in sections where only bicycles pass.” This paper says “identify where tanks actually drive (trusted region), reinforce those sections appropriately, and save weight elsewhere.” The calibration dataset tells you where tanks drive. The conformal certificate guarantees the reinforced sections won’t collapse. The baseline filter is your safety net if a tank somehow wanders into a bicycle lane.

Key Concepts

• Conformal prediction: Imagine you have a machine learning model that predicts something (say, whether a human will turn left or right). You don’t fully trust it, but you have a separate calibration dataset where you know the ground truth. Conformal prediction says: look at how wrong your model was on the calibration set, rank those errors, then at test time, output a “prediction set” instead of a single prediction—a set guaranteed to contain the true answer with probability 1-δ. If your model was usually right on calibration data, the prediction set is small. If your model was often wrong, the set grows to maintain coverage. For safety filters, this translates to: the robot’s action is only allowed if it’s safe for all possibilities in the prediction set. Smaller prediction sets mean less conservatism.

• Belief-space safety filtering: Instead of checking “is this action safe given the human’s current physical state?” you check “is this action safe given my current belief (probability distribution) over what the human might do?” The robot maintains and updates beliefs online by observing the human. Better beliefs (less uncertainty) allow more aggressive actions. The challenge: beliefs live in high-dimensional spaces, so you can’t enumerate all possibilities—hence the need for neural approximations.

• Trusted region: A subset of the belief space defined by a threshold on inference confidence or calibration error. Formally, you choose a set T such that on calibration data, when the belief is in T, your inference module’s errors are bounded with high confidence. Outside T, no guarantees—so the safety filter reverts to a conservative baseline that doesn’t depend on inference. The size of T is a design choice: larger T means the adaptive filter applies more often but inference reliability drops; smaller T means more conservatism but higher reliability when adaptive filtering is used.

Framework Shift

Before (BeliefSF):                  After (this paper):
                                    
Human -> Inference -> Belief        Human -> Inference -> Belief
           |            |                      |   ^        |
           v            v                      v   |        v
     (no guarantees)  Neural                Calibrate    Neural
                      Safety                    |        Safety
                      Filter                    v        Filter
                        |                   Trusted        |
                        v                   Region         v
                    Action                  Check      Conformal
                   (adaptive but               |       Certificate
                    unverified)                v           |
                                           In region?      v
                                           /      \     Action
                                         Yes      No   (adaptive +
                                          |        |    verified)
                                      Adaptive  Baseline
                                      filter    filter

From “learn and act, hope for the best” to “learn, verify where learning is reliable, act adaptively there, fall back conservatively elsewhere.”

Expert Assessment

Problem choice: Real gap. Verification of learning-based safety filters is a central open problem in robotics. Prior work either ignores runtime inference errors or applies uniform verification that doesn’t exploit the structure of belief-space reasoning. This paper threads the needle by co-designing the trusted region and the conformal certificate. It’s a natural next step, not manufactured.

Method maturity: Conceptually clean. The trusted region idea is simple but powerful—it leverages domain structure (beliefs cluster in certain regions during normal operation) that standard conformal prediction misses. The downside: method assumes you can identify the trusted region offline, which requires good coverage in calibration. If deployment differs significantly from calibration, the trusted region may not cover actual operation, forcing frequent baseline fallbacks. The paper acknowledges this but doesn’t deeply explore how robust the approach is to distribution shift.

Experimental integrity: Single benchmark (human-vehicle interaction simulation). Results show the method certifies a less conservative filter than standard conformal prediction, which is the claim. Baselines are fair. Missing: real-world experiments, ablations on trusted region size, stress tests on calibration coverage. The numbers are believable but not exhaustive.

Writing quality: Abstract and intro are tight. Section 3 (problem formulation) is dense—could benefit from a running example introduced early and threaded through the math. The trusted region definition appears suddenly in Section 4 without sufficient motivation for why this particular formulation. Algorithm 1 is clear, but the paper doesn’t discuss computational cost or scalability. If I were reviewing, I’d ask the author to add a “limitations” subsection discussing when the method fails.

Verdict: weak accept — Solid contribution to an important problem, conceptually novel use of conformal prediction structure, but experimental validation is thin and practical deployment questions are underexplored.

Takeaways

Trusted subset verification: When verifying a learning-based system, you don’t need universal guarantees. Identify where the system operates reliably, verify there, and fall back to a conservative baseline elsewhere. This pattern generalizes beyond robotics—any system that learns online and has a safety-critical component can use this structure.

Conformal prediction for modular systems: Standard conformal prediction treats the predictor as a black box. If your system has internal structure (e.g., inference module + safety filter), you can tighten the certificate by reasoning about where each module is reliable. The trick: align the verification region with the operational region.

Calibration as a design tool: The calibration dataset doesn’t just tune parameters—it defines the trusted region, which shapes the entire system’s behavior. Investing in diverse, representative calibration data has compounding returns: better trusted region -> less conservatism -> better task performance without sacrificing safety guarantees.