Calibrating Conservatism for Scalable Oversight

Paper: 2605.28807 Authors: William Overman, Mohsen Bayati Categories: cs.AI

The Gap

Existing scalable oversight approaches face a trilemma: they either rely on complex theoretical assumptions (debate, recursive reward modeling), remain heuristic without guarantees (impact measures), or lack practical sequential decision-making methods with statistical bounds. When an AI agent becomes more capable than its human overseers, how do you maintain control without micromanaging every action? Prior work on Attainable Utility Preservation (AUP) introduced conservatism through auxiliary reward penalties, but lacked a principled way to calibrate how conservative to be.

Problem: Superhuman AI needs oversight
    |
    v
Existing approaches:
    |
    +---> Debate/Recursive RM ---> Complex assumptions
    +---> Impact measures ------> No guarantees
    +---> AUP ------------------> No calibration method
    |
    v
This paper: CCO = AUP + Conformal Decision Theory
    |
    v
Evidence: SWE-bench + MACHIAVELLI experiments
    |
    v
Conclusion: Finite-time safety bounds without distributional assumptions

The Increment

One sentence: Before this paper, conservatism in AI oversight was either uncalibrated or theoretically intractable; after, we have a practical method with finite-time statistical guarantees that works in sequential settings.

Core Mechanism

CCO operates in two layers. The bottom layer aggregates multiple “overseer” scoring functions (each measuring a different concern: code quality, ethical violations, resource usage) into a single penalty term. This penalty measures how much an action deviates from a conservative baseline policy across all auxiliary objectives. The top layer calibrates this penalty’s weight online using conformal prediction: it adjusts the penalty strength so that the empirical violation rate (actions that lead to undesirable outcomes) matches a user-specified target threshold.

The calibration works through a feedback loop. After each action, the system observes whether a violation occurred. If violations exceed the target rate, the penalty weight increases, making the agent more conservative. If violations stay below target, the weight decreases, allowing more exploration. This adjustment happens through a conformal score that measures how “nonconforming” the current trajectory is compared to past safe trajectories.

Action space
    |
    v
[Agent proposes action a] ---> [Overseer scores: s1, s2, ..., sn]
    |                                      |
    |                                      v
    |                          [Aggregate into penalty P(a)]
    |                                      |
    v                                      v
[Baseline policy b] --------> [Deviation: ||Q(a) - Q(b)||]
    |                                      |
    |                                      v
    |                          [Weighted penalty: lambda * P(a)]
    |                                      |
    v                                      v
[Execute action] -----------> [Observe violation? yes/no]
    |                                      |
    |                                      v
    +<---------------- [Update lambda via conformal prediction]

Think of CCO as a self-adjusting speed governor on a car. The auxiliary overseers are like multiple sensors (fuel efficiency, engine temperature, tire pressure) that collectively signal “concern.” The penalty is the brake pressure applied when concern rises. The baseline policy is your normal driving speed. When you want to accelerate beyond normal, the governor checks all sensors—if they’re all green, it allows acceleration; if multiple sensors flash warnings, it applies brakes proportional to the concern level. The conformal calibration is the feedback mechanism that learns how sensitive the governor should be: if you’ve been getting into too many near-misses (violations above target), it becomes more cautious; if you’ve been driving safely with room to spare, it loosens up. The key insight is that the governor doesn’t need to understand why a sensor is alarmed—it just needs to aggregate concern signals and calibrate its sensitivity to keep violation rates within bounds.

Key Concepts

• Conformal Prediction: Imagine you’re trying to predict tomorrow’s temperature, but instead of giving a single number, you give a range that’s guaranteed to contain the true temperature 90% of the time (if you want 90% coverage). Conformal prediction does this without assuming anything about the data distribution. It works by looking at past prediction errors and constructing intervals based on how “weird” the new prediction is compared to historical data. In CCO, this translates to: “Given past trajectories, how much should I penalize this action to ensure violations stay below my target rate?” The method provides finite-time guarantees—you don’t need infinite data to get valid bounds.

• Collective Conservatism: Traditional conservatism says “when in doubt, do nothing.” Collective conservatism says “when *multiple independent concerns align, then be cautious.” A single overseer might be wrong or overly cautious, but when many diverse overseers simultaneously flag an action as problematic, that’s a stronger signal. The penalty scales with accumulated concern, so high-utility actions that only worry one overseer can still proceed, while actions that worry many overseers face steep penalties. This prevents both paralysis (from one paranoid overseer) and recklessness (from ignoring distributed warning signs).

• Attainable Utility Preservation (AUP): The core idea is to penalize actions that change your ability to achieve auxiliary goals, even if those goals aren’t your primary objective. If an action makes it much harder (or easier) to achieve many different auxiliary objectives compared to doing nothing, that’s a sign the action has large side effects. AUP measures this by comparing the Q-values (expected future rewards) for auxiliary tasks under the proposed action versus a baseline. CCO extends AUP by adding the calibration layer—AUP tells you *what to penalize, conformal prediction tells you how much to penalize.

Framework Shift

Before (mainstream approach):        After (this paper):

Human designs reward function        Human specifies violation budget
         |                                    |
         v                                    v
    Agent optimizes                  Agent optimizes + penalty
         |                                    |
         v                                    v
    Hope for alignment               Overseers score actions
         |                                    |
         v                                    v
    Failures discovered              Penalty auto-calibrates
    post-deployment                  to hit violation target
         |                                    |
         v                                    v
    Manual intervention              Statistical guarantee holds

From reward engineering to violation budgeting, the core shift is moving from “specify what you want” to “specify how much risk you’ll tolerate, then let the system learn conservatism.”

Expert Assessment

Problem choice: This is a real gap. Scalable oversight is central to AI safety, and the lack of practical methods with statistical guarantees is a genuine bottleneck. The problem sits at the intersection of two mature areas (conformal prediction, impact measures) that haven’t been combined before—good positioning.

Method maturity: The core insight—using conformal prediction to calibrate AUP penalties—is elegant and non-obvious. However, the method inherits AUP’s limitations: it requires defining auxiliary tasks and a baseline policy, which can be non-trivial. The paper doesn’t deeply explore what happens when auxiliary tasks are poorly chosen or when the baseline policy is itself misaligned. The conformal layer is solid theory, but the aggregation of overseer scores (simple averaging) feels underexplored—why not weighted aggregation or learned combination?

Experimental integrity: The experiments are well-designed. Using an adversarially misaligned agent on SWE-bench is a strong test—many papers would cherry-pick benign misalignment. The MACHIAVELLI results show meaningful tradeoffs (reduced violations at modest reward cost), which is more honest than claiming Pareto improvements. The empirical violation rates closely matching theoretical targets is impressive. One concern: both domains are relatively structured (code editing, text games). How does CCO perform in continuous control or high-dimensional state spaces? The paper doesn’t address computational cost at scale.

Writing quality: The paper is clear and well-structured, but the related work section undersells the novelty—it reads like a literature review rather than positioning the contribution. The experimental section could benefit from failure case analysis: when does CCO fail to calibrate properly? What happens at the boundary of the violation budget? The appendix has important details (conformal score construction) that belong in the main text.

Verdict: weak accept — Solid theoretical contribution with honest experimental validation, but limited exploration of failure modes and scalability concerns.

Takeaways

Steal the calibration pattern: The idea of using conformal prediction to auto-tune hyperparameters (here, penalty weights) based on observed violation rates is broadly applicable. If you have any system where you want to maintain a constraint (error rate, resource usage, fairness metric) without knowing the optimal control parameter upfront, this pattern works: observe violations, compute conformal scores, adjust the control parameter to hit your target rate. No distributional assumptions needed.

Collective signals beat individual signals: When you have multiple weak oversight signals (heuristics, proxy metrics, human feedback), don’t treat them as independent constraints. Aggregate them into a single “concern score” and penalize actions where concern accumulates across signals. This is more robust than either ignoring weak signals or treating each as a hard constraint.

Violation budgets > reward engineering: For safety-critical applications, it’s often easier to specify “I can tolerate X% failures” than to engineer a reward function that captures all desirable behaviors. This paper shows you can build a control system around violation budgets with formal guarantees. The framing shift—from optimization target to risk tolerance—is worth adopting in other domains (medical AI, autonomous vehicles, financial systems).