Auditing Anonymous AI Models: Identifying a Model That Does Not Want to Be Identified

Paper: 2608.31142 Authors: Yisen Xi Categories: cs.SE, cs.AI, cs.CR

The Gap

The 2025-2026 AI market produced a wave of stealth releases: frontier models launched anonymously on developer platforms under codenames. For the people using them, the identity of the model is not a curiosity — it determines data-handling terms, supply-chain risk, and capability expectations. An API under a codename could be a well-known model behind a new name, a variant of one, or something new, and the terms you agreed to follow the identity, not the codename.

No validated methodology existed for black-box identity verification. Two of the available approaches are worth naming because each fails for a different reason. Practitioner checklists — the informal folk methods — lack accuracy evidence; nobody knows their error rates. And self-identification is untrustworthy by design: a model’s claim about what it is can be wrong because it was trained on stale information, or false because the deployment wants it to be. Neither can be the basis of a decision with supply-chain consequences.

   STEALTH RELEASES AND WHY IDENTITY MATTERS

   2025-2026: frontier models launched ANONYMOUSLY on
              developer platforms under codenames
        |
        v
   identity DETERMINES:
     data-handling terms        (what you agreed to)
     supply-chain risk          (what you depend on)
     capability expectations    (what it can do)
        |
        v
   [GAP] no validated methodology for black-box
         identity verification
        |
        +-- practitioner CHECKLISTS: informal folk methods,
        |     LACK ACCURACY EVIDENCE (error rates unknown)
        |
        +-- SELF-IDENTIFICATION: untrustworthy BY DESIGN
              wrong: trained on stale information
              false: the deployment wants it to be
        |
        v
   neither can ground a decision with supply-chain consequences

The Increment

One sentence: Before this paper, anonymous model identity could only be guessed from folklore or asked of the model itself; after it, a four-stage forensic protocol reconstructs, fingerprints, differentially tests the tokenizer, and corroborates — and it was validated prospectively against a case whose official reveal confirmed its inferences.

Core Mechanism

The protocol has four stages, and the ordering is part of the design: each stage narrows the space more cheaply than the next.

Stage 0 — reconstruct launch-time configuration from archived platform snapshots (Internet Archive). This is the most distinctive stage and it does something no probing can: it recovers what the deployment said about itself at launch, and thus exposes preview-to-production drift. A model’s advertised context length or modality set at launch may not match its production behaviour, and that discrepancy is itself an identifying signal as well as a risk finding.

Stage 1 — fingerprint the configuration across context, output ceiling, reasoning, and modality, against the platform catalog. These are cheap, deterministic properties of an API. Enumerating them against the known catalog is the coarse filter, and the catalog comparison prevents the stage from producing an unbounded guess.

Stage 2 — test tokenizer identity with a cross-length differential that rejects short-prompt collisions. This is the sharpest piece of engineering in the paper. Tokenizer fingerprinting by prompt comparison has an obvious failure: short prompts collide, because different tokenizers often segment a few words identically. The cross-length differential addresses that by measuring behaviour as prompt length varies, so identity is established by a trend rather than by a single agreement.

Stage 3 — corroborate with behavioural probes. The last stage is the fuzzy one, deliberately placed last because it is the least decisive.

The validation is reported with the care the topic demands, and the honesty about scope is notable. The authors test declaration consistency on 10 known-identity releases — 7 exact, 2 within precision differences, 1 partial, 0 counter-directional — and they say explicitly that this is not end-to-end identification under anonymity. That distinction matters: consistency on knowns is necessary but not sufficient, and claiming more would be overreach.

Then the prospective test, which is the real evidence. A flagship case’s 2026-08-23 analysis pointed to the GLM-5.3 version line, and the official reveal confirmed those family and version-line inferences. The paper is precise about the boundary too: the deployment variant was not pre-asserted, and the Flash variant was consistent post-reveal rather than predicted. Three further Stage-0-only cases produced a graded hypothesis or a decline rather than a guess — which is the property that makes a forensic protocol trustworthy, since a method that always answers cannot be calibrated.

A standard-library-only implementation is provided as supplementary material, which lowers the barrier to reproducing and contesting the results.

   FOUR-STAGE FORENSIC AUDIT

   [0] RECONSTRUCT launch-time configuration
         from archived platform snapshots (Internet Archive)
       -> recovers what the deployment SAID about itself
       -> exposes PREVIEW -> PRODUCTION DRIFT
       <- the one stage that probing cannot replicate

   [1] FINGERPRINT configuration against the platform catalog
         context | output ceiling | reasoning | modality
       -> cheap, deterministic; the COARSE FILTER

   [2] TOKENIZER IDENTITY via CROSS-LENGTH DIFFERENTIAL
         rejects SHORT-PROMPT COLLISIONS
       -> identity from a TREND across prompt lengths,
          not from a single agreement

   [3] CORROBORATE with behavioural probes
       -> the fuzzier stage, deliberately LAST

   VALIDATION
     declaration consistency on 10 KNOWN identities:
       7 exact | 2 precision-differences | 1 partial
       0 counter-directional
       -- explicitly NOT end-to-end identification under anonymity

     prospective: a 2026-08-23 analysis pointed to the GLM-5.3
       version line; the official reveal CONFIRMED family and
       version-line inferences
       (deployment variant NOT pre-asserted; Flash consistent
        post-reveal, not predicted)
     3 Stage-0-only cases -> a GRADED hypothesis or a DECLINE,
       rather than a guess

Think of it as identifying a car by its registration office records and its wiring, not by asking the driver. Asking the driver is the self-identification route: sometimes they will tell you honestly, sometimes they have been told to say something else, and sometimes they simply misremember. The archives give you what the registration listed at the time of issue — and if the vehicle no longer matches that listing, you have learned something. The catalogue of measurements (length, output limits, features) narrows the candidates cheaply. And the tokenizer test is like identifying an engine by how it sounds across a range of speeds rather than at idle: at idle many engines sound alike, but the way the note changes with revs is much harder to fake. The protocol’s willingness to return a graded hypothesis or nothing, rather than always a name, is the equivalent of an examiner who reports “consistent with, but not established”.

Key Concepts

• Identity as a governance fact: data-handling terms, supply-chain risk and capability expectations all follow the model’s identity, not its codename. It is why this is a compliance and procurement problem rather than a technical curiosity.
• Archived launch-time configuration as evidence: recovering what a deployment advertised at launch, and detecting preview-to-production drift. It provides a class of evidence that probing cannot generate, because it comes from before the deployment knew it would be audited.
• Cross-length differential for tokenizer identity: measuring tokenizer behaviour as prompt length varies so that short-prompt collisions cannot produce a false match. It converts a fragile point comparison into a trend, which is the difference between a fingerprint and a coincidence.
• Graded hypotheses and declines as the trustworthiness property: three cases where the protocol did not name a model. A method that always answers has an unmeasurable error rate, and declining is what makes the positive identifications meaningful.

Framework Shift

Before (folklore or self-report):
  practitioner checklists      -> no accuracy evidence
  self-identification          -> untrustworthy by design
  -> identity guessed, or taken from the party with an interest
  -> no basis for a supply-chain decision

After (forensic, staged, validated):
  stage 0: archived launch config -> exposes preview drift
  stage 1: configuration fingerprint against the catalogue
  stage 2: tokenizer cross-length differential
  stage 3: behavioural corroboration
  -> validated prospectively against an official reveal
  -> returns a graded hypothesis or declines rather than guessing

From asking a deployment what it is, or applying a checklist nobody has calibrated, to assembling external evidence in stages and reporting confidence, the core shift is that identity is a forensic conclusion to be established rather than a declaration to be accepted.

Expert Assessment

Problem choice: Excellent, and timely in a way few papers are. Anonymous frontier releases are a real market practice, and the downstream consequences — terms, supply-chain exposure, capability expectations — are the kind that governance processes need factual answers for. Establishing a validated method before the practice becomes universal is the right sequencing.

Method maturity: The staging is well designed, with each stage cheaper than the one it narrows toward and stage 0 providing evidence no probing can replicate. The cross-length differential is the most technically satisfying element, because it identifies and fixes a specific failure mode — short-prompt collisions — rather than asserting that fingerprinting works. The decision to place behavioural probes last matches their reliability.

Experimental integrity: The honesty about scope is the paper’s strongest feature. Reporting that consistency on ten known identities is not end-to-end identification under anonymity, and marking which parts of the flagship case were pre-asserted versus merely consistent afterwards, is the discipline this kind of work needs and rarely gets. The graded-hypothesis and decline outcomes are reported as results rather than as failures. The limitation is an obvious one: a single flagship prospective case is a small sample, and the protocol’s error rate under anonymity remains unmeasured — the paper says so rather than implying otherwise.

Writing quality: Each stage is stated with its purpose and its failure mode, which makes the protocol reproducible rather than just readable. Since the practical value is in adoption, a short worked trace of one stage — the specific configuration fields compared, or the differential shape that identified a tokenizer — would let a reader implement it without re-deriving the method from the description.

Verdict: strong accept — it defines a governance-relevant forensic method, validates it prospectively where validation was possible, and is unusually careful about what has and has not been demonstrated.

Takeaways

• Treat self-identification as evidence, not as a finding. A model’s claim about its own identity can be stale or strategically chosen, and neither is detectable from the claim alone.
• Use archived launch-time records. Pre-launch documentation exposes preview-to-production drift and is evidence that probing cannot generate.
• Prefer differential tests over point comparisons for fingerprinting. Short inputs collide across tokenizers; behaviour as a variable changes is much harder to fake.
• Build your forensic method so it can decline. A procedure that always produces a name has an unknown error rate, and its correct identifications cannot be distinguished from its guesses.