LLMSurgeon: Reverse-Engineering the Training Diet of Black-Box Language Models

Paper: 2605.30348 Authors: Yaxin Luo, Jiacheng Cui, Xiaohan Zhao, Xinyi Shang, Jiacheng Liu, Xinyue Bi, Zhaoyi Li, Zhiqiang Shen Categories: cs.CL, cs.AI, cs.LG

The Gap

LLM providers rarely disclose their pretraining data mixtures. Existing auditing methods either require training data access (impossible for closed models) or rely on membership inference (which detects individual examples, not domain-level distributions). Prior work on data attribution focuses on influence of specific samples, not recovering the overall recipe. The gap: no principled way to estimate “what domains, in what proportions” from black-box model outputs alone.

Problem: Closed LLMs hide pretraining mixture
    |
    v
Assumption: Generated text carries domain fingerprints
             (label-shift: test distribution differs from train)
    |
    v
Method: LLMSurgeon
    - Train domain classifier on reference corpus
    - Estimate soft confusion matrix (how domains confuse)
    - Solve constrained inverse problem to recover mixture
    |
    v
Evidence: LLMScan benchmark (open models, known recipes)
    |
    v
Conclusion: Recovers domain proportions with <5% error
            (under fixed protocol, predefined taxonomy)

The Increment

One sentence: Before, auditing LLM training data required access to the corpus itself; after, you can estimate domain-level composition from generated text alone using calibrated inverse inference.

Core Mechanism

LLMSurgeon has three stages. First, train a domain classifier on a reference corpus with known domain labels (e.g., Wikipedia, GitHub, books). This classifier learns to map text to domain probabilities. Second, generate text from the target LLM and run it through the classifier. But don’t just average the predictions—the classifier makes systematic mistakes (confuses legal text with news, code comments with documentation). Third, estimate a soft confusion matrix: for each true domain, what does the classifier predict? Then solve an inverse problem: given observed classifier outputs and the confusion matrix, what input mixture would produce these outputs? Add constraints (proportions sum to 1, non-negative) and solve via convex optimization.

Reference Corpus          Target LLM
(labeled domains)         (black box)
      |                        |
      v                        v
  Classifier <----------  Generated Text
      |                        |
      +------------------------+
      |
      v
Confusion Matrix C:
  C[i,j] = P(predict j | true domain i)
      |
      v
Inverse Problem:
  observed = C * mixture
  solve for mixture (constrained optimization)
      |
      v
Estimated Domain Proportions

Think of it like reverse-engineering a smoothie recipe by tasting the result. You have a “flavor classifier” trained on pure ingredients (strawberry, banana, spinach). You taste the mystery smoothie and the classifier says “60% strawberry, 30% banana, 10% spinach.” But you know your classifier confuses banana with mango 20% of the time, and spinach with kale 30% of the time. So you build a confusion table: when it’s actually banana, what does the classifier say? Then you work backwards: given what the classifier tasted and how it typically gets confused, what was the actual ingredient mix? The confusion matrix is your calibration—it corrects for the classifier’s systematic biases. Without it, you’d just trust the raw predictions and get the recipe wrong.

Key Concepts

• Label Shift: The test distribution (generated text from target LLM) differs from the training distribution (reference corpus) in class proportions, but the conditional distribution P(text|domain) stays the same. Concretely: if the target LLM was trained on 40% code and your reference corpus is 10% code, the domain proportions shifted, but what “code” looks like didn’t change. This assumption lets you use a classifier trained on one mixture to infer another mixture. Without label shift, you’d need the classifier to be trained on the exact same mixture as the target—impossible for black-box models.

• Soft Confusion Matrix: Instead of a binary “correct/incorrect” confusion matrix, this captures the full probability distribution of classifier predictions for each true domain. Entry C[i,j] is the probability the classifier predicts domain j when the true domain is i. Estimated by running the classifier on held-out reference data where true labels are known. This matrix encodes systematic biases: maybe the classifier always gives 15% probability to “news” even for pure code samples. The inverse problem uses this matrix to “undo” the classifier’s distortions and recover the true mixture.

• Constrained Inverse Problem: You observe y (classifier outputs on generated text), you know C (confusion matrix), and you want to find x (true domain mixture) such that y = Cx. But this is ill-posed—many solutions exist. Add constraints: x must be a probability distribution (sums to 1, all entries ≥ 0). Solve via convex optimization (quadratic programming). The constraints regularize the solution and ensure it’s interpretable as a mixture. Without constraints, you might get negative proportions or proportions summing to 3, which are nonsensical.

Framework Shift

Before (mainstream approach):        After (this paper):

Training Data                        Generated Text
     |                                    |
     v                                    v
Membership Inference              Domain Classifier
(detect specific examples)        (predict domain per sample)
     |                                    |
     v                                    v
Binary: in/out of training        Aggregate Predictions
                                         |
                                         v
                                  Confusion Matrix
                                  (calibrate biases)
                                         |
                                         v
                                  Inverse Problem
                                  (recover mixture)
                                         |
                                         v
                                  Domain Proportions

From detecting individual data points to estimating population-level composition, the core shift is treating the model as a noisy sensor of its training distribution rather than a membership oracle.

Expert Assessment

Problem choice: Real gap. Data transparency is a live issue (lawsuits, regulation, scientific reproducibility). Prior work on data attribution focused on sample-level influence, not mixture recovery. The framing as inverse problem under label shift is principled, not ad-hoc. Sits at the intersection of model auditing and distribution shift—both active areas.

Method maturity: Clever use of confusion matrix calibration. The label-shift assumption is strong but reasonable for domain-level analysis (domains are stable, proportions vary). Potential weakness: relies on predefined taxonomy and reference corpus quality. If your taxonomy doesn’t match the target’s (e.g., target has “medical” but your reference lumps it into “science”), you’ll get garbage. The method is as good as your classifier and confusion matrix estimation. No comparison to simpler baselines like just averaging classifier outputs without calibration—would be useful to see the delta.

Experimental integrity: LLMScan is a solid contribution—open models with disclosed mixtures let you verify ground truth. But all experiments use models trained on similar data distributions (CommonCrawl, GitHub, books). What happens with a model trained on proprietary data (e.g., enterprise documents, non-English web)? The paper doesn’t test robustness to taxonomy mismatch or adversarial generation (what if the target LLM deliberately obfuscates its mixture?). Error bars are reported but no statistical significance tests. Baselines are weak—mostly ablations of their own method, not competing approaches.

Writing quality: Clear motivation and method description. The confusion matrix explanation could be tightened—it’s introduced late and the reader has to backtrack. Section 4.3 (LLMScan construction) feels like supplementary material, breaks the flow. The related work section is thorough but reads like a literature dump. If rewritten to focus on the conceptual lineage (label shift → inverse problems → data auditing), it would elevate the framing.

Verdict: Weak accept — solid method and useful benchmark, but limited evaluation scope and strong assumptions reduce impact. Needs stress-testing on diverse models and taxonomies.

Takeaways

Practitioners can steal the calibration idea: if you’re aggregating classifier predictions to estimate a distribution, don’t trust raw outputs—estimate a confusion matrix on held-out data and solve an inverse problem. This applies beyond LLMs: any time you’re inferring a latent mixture from noisy observations (e.g., estimating user demographics from behavior logs, inferring content sources in a recommendation feed). The label-shift framing is also portable—if you believe the conditional distribution is stable but class proportions vary, you can adapt classifiers across distribution shifts without retraining. Concrete technique: use convex optimization with simplex constraints (non-negative, sum-to-one) to regularize mixture recovery. The LLMScan benchmark is immediately useful for anyone working on data provenance or model auditing.