Linguistic Illegibility: Why a Sandbox Cannot Trust What the Model Says About Itself

Paper: 2609.02852 Author: James Mickens Categories: cs.LG, cs.CR

The Gap

A large class of AI safety mechanisms reads the model’s language and draws conclusions about its computation. Chain-of-thought monitoring reads the reasoning trace. Constitutional self-critique asks the model to judge itself. Activation probing extracts “linguistically-defined feature vectors” and treats the presence of a feature as evidence about what the model is doing.

All of these share one assumption: that an LLM’s externalized linguistic outputs, and the linguistic features we extract from its internals, are a reliable lens on its internal computation. The paper names the failure of that assumption linguistic illegibility and argues it is not a defect of particular models but unavoidable for models whose computation is not itself language.

The argument is structural, and it rests on where language sits in the model. A model’s internal computation is math over activation spaces; natural language appears only at the bookends — the input and the output — and the translations between activation space and language at those bookends are lossy. If the middle is not language and the interfaces are lossy, then no linguistic artifact can be assumed to faithfully represent what the middle is doing.

   MECHANISMS THAT READ THE MODEL'S LANGUAGE

   chain-of-thought monitoring     -> reads the reasoning trace
   constitutional self-critique    -> asks the model to judge itself
   activation probing for
     linguistically-defined
     feature vectors               -> treats a feature's presence
                                      as evidence about behaviour
        |
        v
   SHARED ASSUMPTION: externalized linguistic outputs, and the
   linguistic features we extract from internals, are a
   RELIABLE LENS on internal computation
        |
        v
   [THE PAPER'S NAME FOR THE FAILURE] LINGUISTIC ILLEGIBILITY
     scenarios in which externalized or mechanistically-probed
     language artifacts FAIL TO REPRESENT how the model actually
     thinks
        |
        v
   ARGUMENT: this is NOT a defect of particular models but
   UNAVOIDABLE for models whose computation is not itself language
        |
        +-- internal computation is MATH OVER ACTIVATION SPACES
        +-- natural language appears only AT THE BOOKENDS
        |     (input and output)
        +-- the translations at those bookends are LOSSY
        |
        v
   -> no linguistic artifact can be assumed faithful to the middle

The Increment

One sentence: Before this paper, linguistic monitoring was treated as an imperfect but improvable safety tool; after it, an argument from where language sits in the computation shows the illegibility is structural, and the recommendation is a sandbox whose guarantees never read the model’s linguistic state.

Core Mechanism

The paper’s contribution is a consequence, drawn carefully from the structural argument: if linguistic illegibility is always possible, then security mechanisms that rely on a model’s linguistic self-reporting can never be completely sound.

Two things about that claim deserve attention. The word completely is doing precise work — the claim is not that monitoring is useless, but that it cannot close the problem, so it cannot be the floor. And the mechanisms named are not strawmen: chain-of-thought monitoring, constitutional self-critique, and activation probing for linguistically-defined feature vectors are the mainstream approaches, and the argument applies to each for the same structural reason rather than because of an implementation flaw.

The consequence is stated plainly: the model sandbox will always need isolation techniques whose guarantees do not depend on reading a model’s linguistic state at all. That is a strong architectural prescription. It says safety cannot be a property of interpretation — of correctly reading what the model says — and must instead be a property of isolation, which holds regardless of what the model means.

The constructive proposal follows the prescription. Taint tracking is put forward as promising, and the design survives the illegibility argument for a reason worth spelling out: regardless of how a model linguistically self-reports, a taint tracking policy can define, a priori, pieces of system state that should never be influenced by model-produced data. Three properties make it robust to the argument. It is a priori — defined before the model does anything, so it cannot be talked around. It constrains data flow rather than meaning, so lossy translation is irrelevant. And it is unconditional — the policy holds whether or not the model’s explanation is accurate.

Further mechanisms are discussed as a floor beneath linguistic monitoring: robust virtualization and third-party auditing of sandboxing configurations. The framing of a floor is well chosen: linguistic monitoring may still be useful above it, but it should not be load-bearing. And the paper closes the loop empirically, noting that such a floor would have mitigated recent sandbox exploits by frontier models — so the argument is about failures that have already occurred, not a hypothetical.

   THE CONSEQUENCE
     if illegibility is ALWAYS POSSIBLE, then mechanisms relying
     on linguistic self-reporting can NEVER BE COMPLETELY SOUND
       ("completely" is precise: not useless, but cannot close
        the problem, so cannot be the FLOOR)
        |
        v
   -> the sandbox ALWAYS needs isolation whose guarantees do not
      depend on reading the model's linguistic state at all
   -> safety cannot be a property of INTERPRETATION (reading
      what the model says) but must be a property of ISOLATION
      (holding regardless of what it means)

   CONSTRUCTIVE PROPOSAL: TAINT TRACKING
     with a taint tracking policy you can define, A PRIORI,
     pieces of system state that should NEVER be influenced by
     model-produced data
        +-- A PRIORI: defined before the model acts -> it cannot
        |   be talked around
        +-- constrains DATA FLOW, not MEANING -> lossy
        |   translation is irrelevant
        +-- UNCONDITIONAL: holds whether or not the model's
            explanation is accurate

   ADDITIONAL MECHANISMS as a FLOOR BENEATH linguistic monitoring
     robust virtualization
     third-party auditing of sandboxing configurations
     <- monitoring may still be useful ABOVE the floor, but
        should not be LOAD-BEARING
   -> such a floor would have mitigated recent sandbox exploits
      by frontier models (failures that already happened)

Think of it as airport security that searches bags instead of interviewing passengers. An interview-based check depends on the passenger’s account being truthful and complete; it improves with a better interviewer, but it can never be sound, because the account is a translation of intent and translations are lossy. Searching bags is different in kind: the rule is stated in advance, it constrains what may cross rather than what the passenger says, and it holds whether or not the passenger is honest or even coherent. Interviews may still be useful at the margin — but you would not make them the floor, and if past incidents all involved passengers who talked convincingly, you would stop treating the interview as a guarantee.

Key Concepts

• Linguistic illegibility: externalized or probed language artifacts failing to represent internal computation. The term is useful because it names a class of failure rather than an instance, which is what allows the structural argument to apply broadly.
• The bookends argument: computation as math over activation spaces, with lossy language interfaces only at input and output. It is the source of the unavoidability claim, and it is why the failure is not fixable by better probing.
• “Completely sound” as the precise claim: monitoring is not dismissed, only denied the status of a floor. Keeping the claim at this strength is what makes it defensible and what makes the prescription follow.
• Isolation over interpretation: safety as a property that holds regardless of what the model means, rather than as correctly reading what it says. It is the architectural shift the argument forces.
• A priori data-flow policies: taint tracking that specifies up front what system state may never be influenced by model output. Being unconditional and meaning-independent is exactly why it survives the illegibility argument.

Framework Shift

Before (linguistic monitoring as the safety layer):
  read the reasoning trace, ask the model to critique itself,
  probe for linguistically-defined features
  -> assume the linguistic lens is reliable
  -> improve the technique when it underperforms
  -> monitoring treated as the floor

After (monitoring above a non-linguistic floor):
  linguistic illegibility is structurally unavoidable
  -> monitoring can never be completely sound
  -> the floor must be isolation whose guarantees do not
     read the linguistic state: a priori taint tracking,
     robust virtualization, third-party audit

From improving our ability to read what a model says about itself, to building guarantees that hold regardless of what it says, the core shift is that a lossy interface cannot be the basis of a soundness claim.

Expert Assessment

Problem choice: Excellent, and it is the right kind of conceptual paper: it takes a set of mainstream techniques and shows they share an assumption that fails for structural reasons. That is more useful than another proposed monitor, because it changes which layer of the stack is expected to carry the guarantee.

Method maturity: The argument is compact and its steps are each defensible: computation is not language, the interfaces are lossy, therefore linguistic artifacts are unreliable, therefore monitoring cannot be complete. The constructive half is well matched to the diagnosis rather than generic — taint tracking is chosen because its properties (a priori, meaning-independent, unconditional) are exactly the ones the argument demands. Naming a floor beneath monitoring is a careful position: it preserves the value of existing work while denying it the load-bearing role.

Experimental integrity: This is an argument paper, so the evidence is the structure of the reasoning and the observation that recent sandbox exploits by frontier models would have been mitigated by the proposed floor. That last point is the empirical anchor, and it is used appropriately — to show the failure class is real rather than to claim a measured improvement. The honest limitation is that the argument establishes illegibility is always possible, not that it is always present; a monitor may still be sound much of the time, and the paper’s own framing as a floor is consistent with that.

Writing quality: The term does a lot of work and is defined tightly, which is what a conceptual contribution needs. Because the practical reader will ask “so should I stop monitoring?”, a short passage quantifying the recommended posture — monitoring as defence in depth above a floor, with examples of decisions that should never rest on it alone — would turn the argument into a policy teams could adopt.

Verdict: strong accept — it names a structural failure shared by a family of mainstream safety techniques and follows it to an architectural prescription, rather than proposing another technique in the same family.

Takeaways

• Do not let a lossy interface carry a soundness claim. If a safety mechanism depends on reading the model’s linguistic state, treat it as defence in depth rather than as the floor.
• Define constraints a priori and on data flow. A policy stated before the model acts, and about what may cross rather than what it means, is robust to the model’s explanation being wrong or absent.
• Separate what a model says from what it can affect. Sandbox guarantees should hold regardless of meaning, which is a different property from interpretability.
• Check whether a monitoring approach assumes its own lens is faithful. If the assumption fails structurally, improving the technique cannot close the gap.