One Layer Is Enough: RL Post-Training Concentrates in the Middle of Transformers

Paper: 2607.01232 Authors: Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong Categories: cs.LG, cs.CL

The Gap

RL post-training (GRPO, GiGPO, Dr. GRPO, etc.) has become the standard pipeline for unlocking reasoning and agentic capabilities in LLMs after supervised fine-tuning. The universal practice is straightforward: update all parameters uniformly, as if every layer in a 60+ layer transformer contributes equally to the final reward signal. Nobody had systematically asked: does it actually work that way?

There is prior work on layer-wise analysis in supervised fine-tuning (e.g., LLM-Surgeon, layer freezing studies), and some work on parameter-efficient RL (LoRA-based methods), but these either operate under different training objectives or impose a low-rank bottleneck that conflates expressiveness with contribution. The specific gap is this: no one had cleanly isolated the per-layer contribution to RL reward gains, without architectural constraints, across multiple models, algorithms, and tasks.

The logical path from gap to conclusion:

RL post-training updates ALL layers uniformly
  |
  v
Implicit assumption: every layer contributes similarly
  |
  v
Isolate each layer: train it ALONE, freeze the rest
  |
  v
Measure "layer contribution" = fraction of full RL gain recovered
  |
  v
Result: ONE middle layer recovers 70-95% of gains
  |
  v
Middle layers dominate; input/output layers barely matter
  |
  v
Pattern is stable across 7 models, 3 RL algos, multiple tasks
  |
  v
Conclusion: full-parameter RL wastes compute on low-contribution layers

The Increment

One sentence: Before this paper, RL post-training treated every transformer layer equally; after this paper, we know that a single middle layer can do most of the work, opening the door to dramatically more efficient RL adaptation.

Core Mechanism

The method is conceptually simple but experimentally rigorous. For a given model and RL task, the authors perform a “layer isolation” procedure: they take a pretrained+SFT checkpoint and run RL training on exactly one layer at a time, keeping all other layers frozen. They then measure the downstream performance (accuracy on math benchmarks, pass rates on code, reward on agentic tasks) and compute the “layer contribution” — the ratio of the single-layer gain to the full-parameter gain.

To make this fair and systematic, they sweep through every layer in the model. They also run full-parameter RL as the baseline reference. The key variable is which layer gets trained; everything else (data, hyperparameters, algorithm, seeds) is held constant.

Model checkpoint (pretrained + SFT)
  |
  +---> Train ONLY layer i, freeze rest --> evaluate --> gain_i
  |       (repeat for i = 1...N)
  |
  +---> Train ALL layers              --> evaluate --> gain_full
  |
  v
layer_contribution[i] = gain_i / gain_full
  |
  v
Rank layers by contribution
  |
  v
Find: middle layers dominate; one layer ~ full performance

Structural metaphor: The orchestra pit.

Imagine a symphony orchestra where every musician plays during rehearsal, and the conductor (RL reward signal) asks everyone to adjust. The conventional approach is to retrain the entire orchestra. This paper asks: what if we let only one section rehearse at a time and see which one moves the audience most?

It turns out the string section (middle layers) carries the melody — adjusting them alone recovers nearly the whole emotional impact. The percussion near the entrance (early layers) and the brass near the back (late layers) add color but not substance to the RL reward. The oboe player on chair 37 (a specific middle layer) is doing most of the heavy lifting.

The “layer contribution” is like measuring how much of the applause you recover by rehearsing just one section. When the strings alone get 90% of the standing ovation, you know where to focus next time.

Key Concepts

• Layer contribution: Think of a company where a big initiative increases revenue by $1M. You want to know which department drove it. You run the initiative with only Marketing turned on — revenue goes up $800K. That’s 80% contribution. You try only Finance — $50K. That’s 5%. Layer contribution does exactly this for neural network layers during RL training. The formula is simple: (performance after training layer i alone minus SFT baseline) divided by (performance after training all layers minus SFT baseline). If this ratio is near 1.0, that layer alone almost fully explains the RL gains.

• Middle-layer concentration: In a transformer with, say, 28 layers, the pattern is striking: layers 8–20 do almost all the work for RL adaptation, while layers 1–7 and 21–28 contribute little. This is not random — it aligns with prior findings that middle layers in transformers encode abstract reasoning representations. RL is essentially teaching the model *how to reason, so it makes sense that the reasoning layers are where the gradient signal should concentrate. Early layers handle tokenization-level patterns; late layers handle output formatting. Neither changes much under RL.

• Cross-condition stability: The layer ranking doesn’t shuffle when you switch from math to code, from GRPO to Dr. GRPO, or from Qwen3-8B to Qwen2.5-32B. The same middle layers always dominate. This stability is the paper’s most surprising finding — it suggests the phenomenon is structural to transformers, not an artifact of a specific setup.

Framework Shift

Before (mainstream approach):           After (this paper):
                                       
All layers treated equally             Layer-aware RL training
  |                                      |
  +--[L1]--[L2]--[L3]-...-[LN]--+       +--[L1]--[L2]-[L*]-...-[LN]--+
  |   ^     ^     ^        ^    |       |   .     .    ^^^        .    |
  |   |     |     |        |    |       |              high              |
  |  equal gradient signal       |       |         concentration        |
  +------------------------------+       +------------------------------+
                                       
Compute cost: O(N) per step              Compute cost: O(1) per step
Same LR, same updates everywhere         Focus compute on high-contribution layers

From uniform layer adaptation to targeted layer adaptation, the core shift is recognizing that RL post-training is not a dense learning signal distributed evenly across a transformer — it is a sparse signal concentrated in the middle of the stack.

Expert Assessment

Problem choice: This is a genuine gap. The field has been blindly running full-parameter RL for two years now without understanding the internal mechanics. The question “which layers actually change under RL?” is embarrassingly basic and should have been asked sooner. It sits at the intersection of mechanistic interpretability and training efficiency — two hot trajectories — so the timing is good.

Method maturity: The approach is clean — isolate and measure. No fancy new architecture, no loss function tricks. This is a strength: the simplicity makes the findings hard to dismiss. However, the layer isolation method is essentially an ablation study, and ablation studies have known limitations (interactions between layers are invisible). The authors acknowledge this but don’t fully address it. A causal intervention (e.g., patching middle-layer activations into a non-RL model) would strengthen the story.

Experimental integrity: The sweep is impressively broad: 7 models, 3 algorithms, math/code/agentic tasks. The layer contribution metric is well-defined and reproducible. The cross-condition stability of the ranking is the strongest evidence. Minor concern: all models are from the Qwen family — testing on Llama, Mistral, or DeepSeek would make the “structural to transformers” claim more convincing. The paper is also missing compute-efficiency comparisons: if training one layer takes 1/28th the compute but needs 3x the steps to converge, the practical savings may be smaller than implied.

Writing quality: The paper is well-structured and reads clearly. The related work section does a decent job positioning against LoRA and layer-freezing baselines. However, the discussion section is thin — it raises interesting questions about why middle layers dominate but doesn’t attempt a mechanistic explanation (e.g., probing what representations change in those layers). Section 4 could be rewritten to include activation analysis, which would transform the paper from “we measured something interesting” to “we understand something deep.”

Verdict: weak accept — The empirical finding is robust and practically useful, but the paper stops at measurement without fully explaining the mechanism, and the Qwen-only evaluation limits generalizability.

Takeaways

1. Before running expensive full-parameter RL, profile your layers first. Run a cheap single-layer sweep (one epoch, small data) to identify the top 3–5 contributing layers. Then do your full RL run on only those layers. This can save 60–80% of training compute with minimal performance loss.

2. Layer contribution is a new diagnostic tool. If you’re debugging why RL isn’t working, checking whether the gradient signal actually reaches the middle layers (or whether it’s getting absorbed by the output head) is now a concrete first step.

3. The middle-layer concentration finding is a constraint for architecture search. If RL reward flows through a predictable subset of layers, future architectures could be designed with this in mind — e.g., wider middle layers with parameter-efficient early/late layers.

4. Cross-condition stability is a research agenda opener. If the same layers matter regardless of task, algorithm, or model size, this is a structural property of transformers worth investigating mechanistically — not just empirically.