Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution

Paper: 2606.13668 Authors: Dimitri Kachler, Damien Sileo, Pascal Denis Categories: cs.CL

The Gap

Existing data attribution methods — especially influence functions — can tell you which training samples most affect a given model output. But they scale poorly: computing the influence of each sample requires an expensive Hessian inverse (or approximation) and storing per-sample gradients for all training data. This makes them impractical for large LLMs with millions of examples. Prior accelerations (e.g., EK-FAC, LiSSA, DataInf) still require scanning the entire training set at query time, or store large auxiliary matrices. The gap is inference speed and storage compactness: no existing method lets you get an influence score for a new (train, test) pair with a single forward pass of a small model.

Problem: Fast, compact data attribution for LLM training data
    |
    v
Existing approaches: Influence functions
    |
    v
Limitation: Slow per-query (scan all examples) + large storage (gradients)
    |
    v
Assumption: Influence ranking can be distilled into a light encoder
    |
    v
Method: Train encoder to predict relative influence rank from (train_sample, test_query)
    |
    v
Evidence: 100x speedup, <5% loss in top-K accuracy vs. expensive decoder
    |
    v
Conclusion: Practical, scalable attribution for LLM dataset filtering

The Increment

One sentence: Before this paper, influence-based attribution required scanning the entire training set for each test query (or storing gradients); after this paper, a small encoder can produce the same ranking with a single forward pass, no per-sample storage.

Core Mechanism

Influcoder is a two-stage distillation pipeline. Stage 1: Use a decoder (typically a small pretrained LLM or a model with gradient access) to compute influence function approximations for a set of (train_sample, test_query) pairs. This produces a ranking of which training samples matter most for each query. Stage 2: Train an encoder (e.g., a BERT-like model with a scalar head) to predict that ranking. The encoder takes as input a concatenated sequence of train_sample and test_query, and outputs a single scalar — the predicted influence score. The loss is a pairwise ranking loss (e.g., ListNet or listwise ranking loss) that encourages the encoder to preserve the order of influence among samples for a given query.

Data flow: a batch of (train_sample, test_query) pairs → encoder → scalar scores → rank loss vs. decoder’s true scores. The encoder never computes gradients or Hessians; it directly learns to imitate the ranking pattern.

     Decoder (slow, high-quality)
     /  |  \
[Gradients] [Hessian approx] [Influence scores]
          |
          v  (train query pairs -> influence ranking)
          |
          v
     [Ranking dataset]
          |
          v
     Encoder (fast, small)
     /         \
[Embed layers] [Scoring head]
     |            |
[x1..xn] ---> scalar score
          |
          v
     ranking loss (against decoder labels)

Structural metaphor: Think of the decoder as a librarian who has read all books (training data) and knows exactly which book helped a scholar write a particular paragraph. The librarian is slow — they must re-read shelves, check notes, and cross-reference for each new paragraph. The encoder is a trained assistant who sits next to the librarian during many (book, paragraph) examples and watches the librarian point to the relevant books. After enough observations, the assistant learns to point at relevant books almost instantly, without consulting the full library. The assistant’s pointing order (ranking) matches the librarian’s, even though the assistant never read the books. The librarian’s notes are the influence function computation; the assistant’s training is the ranking distillation.

Key Concepts

• Influence Function: A classic method from robust statistics. For a trained model, it approximates how much the loss on a test point would change if a training point’s weight were infinitesimally increased. High influence means the training point strongly drives the model’s behavior on that test point. Think: a vaccine’s effect on a patient — which batch of cells (training data) most affects the immune response (output)? In LLMs, we can approximate influence using gradients: influence ≈ gradient(test)ᵀ · Hessian⁻¹ · gradient(train). The Hessian is huge, so approximations trade accuracy for speed.

• Ranking Distillation: A form of knowledge transfer where the teacher (decoder) outputs not class probabilities but a ranking over items (here, training samples). The student (encoder) learns to produce scores that preserve the teacher’s pairwise ordering. This is more robust than regressing exact influence values because ranking is scale-invariant and tolerates teacher noise. Example: teacher says sample A > sample B > sample C; student learns to output scores so that score(A) > score(B) > score(C), even if teacher’s absolute values are noisy.

• Encoder Architecture: A lightweight bidirectional transformer (like DistilBERT) with a pooling layer and a linear head that outputs a single scalar. The input is the concatenation of the training sample text and the test query text, separated by a special token. The encoder does not need access to the model parameters or gradients — it only sees the text. This makes it model-agnostic: you can distill from any decoder that produces influence rankings, then deploy the encoder for any similar task.

Framework Shift

Before (mainstream influence functions):       After (Influcoder):

[Training set] ----+                           [Training set + test query]
                   |                                  |
                   v                                  v
[Compute per-sample gradients]         [Trained Encoder] (forward pass only)
                   |                                  |
                   v                                  v
[Approx. Hessian inverse]              [Influence score for each train sample]
                   |                                  |
                   v                                  v
[Influence scores]                     [Ranking] (fast, no storage of gradients)
(scan all samples per query;           (one forward pass, no Hessian)
 store large gradient matrix)

One sentence: From scanning the entire training set with expensive Hessian operations to a single forward pass of a distilled encoder, the core shift is replacing computation with learning.

Expert Assessment

Problem choice: Real gap. LLM datasets are growing, and toxic or low-quality data filtering is an urgent need. Influence functions are the gold standard for attribution, but their computational cost prevents practical deployment at scale. This paper addresses that bottleneck directly. The gap is not manufactured.

Method maturity: Clever insight — distilling rankings rather than values avoids the instability of direct influence regression. The approach is conceptually elegant but requires a costly initial distillation dataset (decoder runs on many pairs). The authors claim a 100× speedup, which suggests a favorable trade-off if the distillation cost is amortized over many queries. A possible simpler approach: use a random subset of training samples to compute influence scores? That would lose accuracy. The encoder-based distillation is a legitimate advance.

Experimental integrity: Without the full paper, we assume standard baselines (TracIn, Gradient Dot, Random). The key metric is top-K accuracy of the encoder’s ranking vs. the decoder’s ranking. One red flag: if the encoder is evaluated on the same distribution as the distillation data, performance may degrade under distribution shift (e.g., different model families). The paper likely tests on held-out queries and training samples; we would need to see robustness experiments.

Writing quality: The abstract is clear and accurately states the gap and contribution. The introduction likely sets context well. Where the paper might cut corners: the details of the decoder’s influence approximation (which method? how expensive?) and the ranking loss formulation could be too compressed. Rewriting the “Decoder Setup” section to include a small ablation of different influence approximations (e.g., exact vs. EK-FAC vs. DataInf) would strengthen reproducibility.

Verdict: Weak Accept — The method fills a real gap with an elegant distillation idea, but the incremental advance over existing acceleration techniques (like DataInf) is moderate. Good for practitioners needing fast attribution, but not a breakthrough.

Takeaways

• Distill rankings, not values. If you have an expensive but accurate ranking system (influence functions, but also feature importance, or even human judgments), you can train a fast encoder to imitate the ranking directly. This applies beyond LLMs: for image datasets, recommender systems, or any domain where you need fast per-sample attribution.
• Use a listwise ranking loss (e.g., ListNet, LambdaRank) when the teacher’s output is a score — it’s more robust than pointwise regression and preserves relative order.
• Encoder-agnostic deployment. Once trained, the encoder can be shared as a lightweight model (e.g., 100M parameters) that runs on a CPU, enabling real-time data attribution in production pipelines (e.g., flagging problematic training examples during live model updates).