Mechanism Design for Alignment and Control: Incentivising Honesty and Obedience

Paper: 2609.01595 Authors: Dirk Bergemann, Andrew Koh, Stephen Morris Categories: econ.TH, cs.AI, cs.GT

The Gap

Mechanism design has a long history of making self-interested parties behave well under known rules. The AI setting breaks one of its founding assumptions: both the agent’s alignment (its preferences) and its capabilities (its feasible actions and information) are unknown. So a designer cannot rely on knowing what the agent wants or what it can do.

That changes the problem in a specific way. If we want agents to act on our behalf, a mechanism must incentivise two distinct things at once:

• Honesty — about what the agent is and can do. An agent that conceals its capabilities makes every downstream decision wrong.
• Obedience — actually acting as instructed, rather than as its own preferences would suggest.

Standard mechanism design usually has one of these in view. Incentivising honesty and obedience in the same mechanism, when the agent’s type is unknown on both dimensions, is a different construction, and it had no framework.

   THE BROKEN ASSUMPTION

   classical mechanism design:
     self-interested parties, KNOWN RULES
        |
        v
   AI setting:
     agent's ALIGNMENT (preferences)     : UNKNOWN
     agent's CAPABILITIES (actions, info): UNKNOWN
        |
        v
   -> a designer cannot rely on knowing what the agent
      WANTS or what it CAN DO
        |
        v
   [TWO THINGS MUST BE INCENTIVISED AT ONCE]
     HONESTY   -- about what the agent is and can do
                  (concealment makes every downstream
                   decision wrong)
     OBEDIENCE -- acting as instructed rather than as its
                  own preferences suggest
        |
        v
   [GAP] standard design usually has ONE of these in view;
         both at once, with a doubly-unknown type, had
         no framework

The Increment

One sentence: Before this paper, there was no framework for incentivising honesty and obedience simultaneously from an agent of unknown alignment and capability; after it, a one-sided imitation structure delivers a revelation principle and a characterisation of implementable policies.

Core Mechanism

The framework’s structuring assumption is the one-sided imitation structure: capabilities can be concealed but not counterfeited. That asymmetry is the substantive modelling choice, and it matches the AI setting better than a symmetric type space would. An agent can understate what it can do — sandbagging — but it cannot pretend to abilities it does not have. Assuming symmetric misreporting would make the problem harder in a direction that does not correspond to how capability misrepresentation actually works.

Under that structure, three results come out.

A revelation principle. The standard move in mechanism design, and its presence is what makes the rest tractable: truthful reporting can be assumed without loss of generality, so characterising implementable policies reduces to characterising which truthful ones can be made incentive-compatible.

A characterisation of implementable policies via nested cyclical monotonicity. This is the technical core. Cyclical monotonicity is the familiar condition characterising implementable allocation rules; its nested form is what the two-dimensional uncertainty requires. The nesting is doing the work: honesty and obedience are incentives over different objects, and the implementability condition has to account for both simultaneously rather than separately.

Conditions under which eliciting higher-order beliefs can discipline multiple agents. A third result that speaks to the multi-agent setting directly — when does asking agents what they think other agents will do improve the mechanism’s grip? That is a question with an obvious AI-oversight reading.

The applications are where the framework shows its range, and each one is a recognised problem:

• Sandbagging — the agent that is more capable pretending to be less capable. This is the motivating case for the one-sided structure.
• An alignment–interpretability trade-off, with the striking structure that the two are substitutes in the instrument but complements in value. Substitutes in the instrument means you can buy one with the other; complements in value means having both is worth more than the sum. That combination is a genuinely informative shape for a trade-off, and it is the kind of claim a mechanism-design lens produces and an informal discussion would not.
• Discipline via peer scoring — using agents’ assessments of each other as the incentive.
• Coupling rewards to induce competition among multiple agents.
• Scalable oversight and reward shaping — the applied case the whole framework is ultimately aimed at.

   STRUCTURING ASSUMPTION: ONE-SIDED IMITATION
     capabilities can be CONCEALED
     but not COUNTERFEITED
        -> understating is possible (SANDBAGGING)
        -> overstating is not
        <- matches the AI setting, unlike a symmetric
           type space

   THREE RESULTS
     [1] REVELATION PRINCIPLE
           truthful reporting assumed without loss of
           generality -> reduces characterisation to
           which truthful policies are incentive-compatible
     [2] CHARACTERISATION OF IMPLEMENTABLE POLICIES
           via NESTED CYCLICAL MONOTONICITY
           <- cyclical monotonicity is the familiar condition;
              its NESTED form is what two-dimensional
              uncertainty requires
     [3] CONDITIONS UNDER WHICH ELICITING HIGHER-ORDER
         BELIEFS DISCIPLINES MULTIPLE AGENTS

   APPLICATIONS
     sandbagging
     alignment-interpretability trade-off:
       SUBSTITUTES in the instrument,
       COMPLEMENTS in value
     discipline via peer scoring
     coupling rewards to induce competition
     scalable oversight and reward shaping

Think of it as a hiring process where the candidate knows their own skill better than you do. The candidate can undersell — take a junior role while being capable of more, which is sandbagging — but cannot credibly claim skills they lack, because the interview tests them. Your design problem is to get both truthful disclosure and effort in the role, and those are two different incentives over the same conversation. The substitute-versus-complement result is the interesting part: paying more to learn what the candidate can do may reduce what you have to spend on getting them to work, yet the two together are worth more than either alone.

Key Concepts

• One-sided imitation: capabilities concealable but not counterfeitable. It is the modelling assumption that makes the framework tractable and matches the real asymmetry, since sandbagging is common and capability fabrication is not.
• Honesty and obedience as separate incentive problems: disclosure of type versus compliance with instruction. Most mechanism designs target one; this framework must handle both, because an agent whose type is unknown on both dimensions can fail either way.
• Nested cyclical monotonicity: the implementability condition generalising the familiar cyclical monotonicity to two-dimensional uncertainty. The nesting reflects that the two incentives act over different objects.
• Substitutes in the instrument, complements in value: the structure of the alignment–interpretability trade-off. It is a specific, non-obvious shape that a formal treatment produces and informal reasoning tends to miss.
• Higher-order belief elicitation: asking agents about other agents’ expected behaviour, and the conditions under which that disciplines a group. It is the result with the clearest scalable-oversight reading.

Framework Shift

Before (one incentive at a time, known type):
  design for a self-interested party with known rules
  -> usually incentivise honesty OR obedience
  -> alignment and capability treated as known, or as
     a single unknown parameter

After (two incentives, doubly-unknown type):
  one-sided imitation: concealable, not counterfeitable
  revelation principle + nested cyclical monotonicity
  + higher-order belief elicitation for multiple agents
  -> applications: sandbagging, interpretability trade-off,
     peer scoring, reward coupling, scalable oversight

From incentivising a partially known agent along one dimension, to designing for both honesty and obedience when alignment and capability are each unknown, the core shift is that the overseer’s uncertainty is two-dimensional and the mechanism must be too.

Expert Assessment

Problem choice: Excellent, and it brings a mature formal apparatus to a question the AI field has been handling informally. Sandbagging, scalable oversight and reward shaping are discussed constantly in prose; treating them as mechanism-design problems with unknown types gives them structure and makes their trade-offs stateable.

Method maturity: The one-sided imitation structure is a well-chosen modelling assumption rather than a convenience — concealment without counterfeiting is how capability misrepresentation actually works. The three results are ordered sensibly, with the revelation principle buying tractability and nested cyclical monotonicity carrying the technical weight. The characterisation’s nesting is the right generalisation, since the two incentives act over different objects and could not be handled by the classical condition alone.

Experimental integrity: This is theory, so the reading is of assumptions. The one-sided structure is the load-bearing one and it is clearly stated; the framework’s conclusions are conditioned on it, and an agent capable of counterfeiting capability — through deception at the level of demonstrated performance, say — would fall outside the analysis. The applications are stylised examples rather than worked case studies, which is normal for theory but does mean the practitioner gets shapes rather than numbers.

Writing quality: The abstract is dense but well ordered: assumption, three results, five applications. The alignment–interpretability claim is the most quotable line and it is stated compactly enough to be remembered. Because the audience is split between economists and AI researchers, a short paragraph translating one application — sandbagging — into the terminology of each field would widen the readership considerably.

Verdict: strong accept — it supplies a formal framework for incentive problems that the AI field currently reasons about informally, with a modelling assumption that matches the domain and a characterisation that handles its two-dimensional uncertainty.

Takeaways

• Treat oversight as two incentive problems, not one. Getting honest disclosure of capabilities and getting obedient action are separate, and a mechanism that addresses one can fail the other.
• Model capability misreporting asymmetrically. Concealment without counterfeiting is the realistic assumption, and it keeps the analysis closer to sandbagging as it actually occurs.
• Look for substitute-in-instrument, complement-in-value structures. It is a specific shape a formal treatment reveals and informal trade-off talk tends to miss.
• Ask when higher-order beliefs help. If agents’ forecasts of each other can be elicited and used, that is a scalable-oversight lever worth identifying conditions for.