Tapered Language Models: A Free Lunch Hiding in Plain Sight

Paper: 2606.23670 Authors: Reza Bayat, Ali Behrouz, Aaron Courville Categories: cs.LG, cs.AI, cs.CL

The Gap

All modern language models — Transformer, Mamba, RWKV, Titans — share a weird inheritance: they stack identical layers. The bottom layer has the same hidden dimension as the top layer, even though evidence has been mounting that layers don’t contribute equally. Probes show early layers transform the representation drastically while later layers mostly nudge the residual stream. Feature analysis reveals the last few layers are nearly redundant. Yet nobody asked: if later layers do less work, should they get fewer parameters?

Prior work explored variable depth (wider vs deeper) and conditional computation (which tokens skip which layers), but both change the computational graph. What’s missing is a fixed-budget, fixed-depth comparison: given N layers and K parameters, should we give each layer the same K/N? Or can we do better by giving early layers more and tapering down?

Problem: Identical layers waste capacity on redundant late layers
  |
  v
Assumption: Layer importance is uniform across depth   [WRONG]
  |
  +--- Evidence: Probes show early>late in representation change
  +--- Evidence: Feature redundancy increases in deep layers
  |
  v
Method: Given fixed parameter budget, allocate non-uniformly
  |       Taper MLP width with cosine schedule: early=wide, late=narrow
  v
Evidence: 4 architectures, 3 scales, held-out perplexity improves
  |
  +--- Taper forward: better than uniform baseline
  +--- Taper reverse: worse than uniform baseline   (controls)
  |
  v
Conclusion: Depth-aware capacity allocation is architecture-agnostic

The Increment

One sentence: Before this paper, the default was “same width for every layer”; after this paper, the default should be “wider early, narrower late, under the same budget.”

Core Mechanism

The method is almost laughably simple — which is the whole point. Take any modern language model. Identify the component that dominates parameter count: the MLPs. In a standard Transformer, around two-thirds of all parameters live in the MLP blocks (the two linear layers with a nonlinearity). Now instead of setting the MLP hidden dimension to a constant $d_{\text{ff}}$, set it to a function of depth $t$ that follows a cosine taper:

where $d_{\text{max}}$ is the widest (first layer), $d_{\text{min}}$ is the narrowest (last layer), and $N$ is depth. The total parameter count is matched to a uniform baseline by solving for $d_{\text{max}}$ and $d_{\text{min}}$ under the constraint $\sum d_{\text{ff}}(t) = N \cdot d_{\text{uniform}}$. Data flows identically otherwise: same attention, same residual stream, same everything. Only the MLP’s inner dimension changes across depth.

           Input
             |
        +---------+
        |  Attn   |  (unchanged across depth)
        +---------+
             |
        +---------+
        |  MLP    |  <-- hidden dimension d_ff(t) varies with depth t
        |  [wide] |      early layers: d_ff is large
        +---------+      late layers: d_ff is small
             |
           [... repeat N times, d_ff decreasing per cosine schedule ...]
             |
        +---------+
        |  MLP    |  <-- d_ff is smallest
        | [narrow]|
        +---------+
             |
          Output

Structural metaphor: the fluvial delta. Imagine a river flowing from mountains to sea. Near the source, the river carries sediment, carves canyons, does heavy lifting — it needs to be wide. As it approaches the delta, the water spreads, slows down, mostly just meanders. You wouldn’t build the same-width channel for the whole journey. Similarly, early layers of an LLM “carve” representations from token embeddings; late layers just “distribute” them into the final distribution. The cosine taper is like a channel that starts wide for the heavy work and narrows gracefully where the water is already slow. The key constraint (fixed total budget) is the “dredging budget” — you have a fixed amount of material to build the whole river. Spend it where the current is strong.

Key Concepts

• Depth-wise non-uniformity: Until now, the default assumption in all LLM families was that each layer deserves identical capacity. This paper systematically challenges that by running controlled experiments where only the *distribution of capacity across depth changes. The result: the default was suboptimal by a measurable margin. The concept is simple but hitting it with rigorous controls (forward taper works, reverse taper hurts) is what makes it a finding, not just a guess.

• MLP as the capacity lever: The paper chooses MLP width as the control variable because MLPs dominate parameter count in every modern architecture. Attention heads, KV caches, and other components have different scaling stories. By only tapering MLPs, the authors isolate the effect without changing the attention mechanism or the computation graph. This is a deliberate choice to test the hypothesis without confounding variables. A practitioner can read this and think: “I can change just the d_ff in my config and get the benefit.”

• Cosine taper schedule: The specific schedule matters. They try linear, exponential, and cosine. Cosine wins consistently, and the reason is smoothness: it starts with a slow decline (keeping early layers near their max capacity longer) and accelerates the taper only in the last few layers, where evidence shows redundancy is highest. The schedule is also parameterized to exactly match total budget, so it’s a pure allocation change, not a parameter increase.

Framework Shift

Before (mainstream):            After (this paper):
+------------+                  +----------------+
| MLP d_ff   |                  | MLP d_ff = max |
| = constant | deep            | = decreasing   |
| across all |                 | with depth     |
| layers     |                  | (cosine taper) |
+------------+                  +----------------+
| Layer 1: wide                 | Layer 1: widest
| Layer 2: wide                 | Layer 2: slightly less
| Layer 3: wide                 | Layer 3: less
| ...                           | ...
| Layer N: wide                 | Layer N: narrowest
| Total params: same            | Total params: same
| PPL: baseline                 | PPL: lower
+------------+                  +----------------+

One sentence: From “all layers equal” to “layers are tapered by importance,” the core shift is that parameter capacity is now a function of depth rather than a global constant.

Expert Assessment

Problem choice: This is a real gap. The community has been stacking identical layers for years, and a growing body of evidence shows late-layer redundancy. The paper isn’t inventing a crisis; it’s pointing out an obvious blind spot that everyone walked past. That’s a high-quality problem selection — it’s not a toy, it’s not manufactured.

Method maturity: This is a clever insight, not brute force. The experiment is elegantly simple: control for everything except the distribution of MLP width across depth. A less mature version would have thrown engineering complexity at the problem; this paper uses minimal intervention. That said, the method is incomplete in one way: it only tapers MLPs, not attention heads. Is attention also redundant in late layers? Possibly, but the paper doesn’t touch it.

Experimental integrity: The baselines are fair. They match total parameter count, training tokens, and optimization hyperparameters. The control (reverse taper: narrow early, wide late) shows worse performance, which is a good sanity check — it proves the directionality matters, not just the non-uniformity. A minor red flag: they only train from scratch at 350M, 760M, and 1.8B parameters. Would the pattern hold at 7B+? Possibly, but the scaling trend is convincing. Also, the perplexity improvements are modest (0.1-0.3 PPL), though consistent across architectures.

Writing quality: The paper is well-structured but has a blind spot: it never discusses the *cost of the taper in terms of hardware efficiency. Modern accelerators prefer uniform tensor shapes for parallelism. Tapered widths might cause underutilization in the narrow early layers. The authors should have addressed this in a limitations section. The code release (if any) is not mentioned in the abstract, which is a missed trust signal.

Verdict: weak accept — The finding is real and valuable, but the paper’s impact is in the problem framing more than the engineering solution. It’s a “why didn’t I think of that?” moment that shifts the default, but the effect size is small enough that it doesn’t force immediate adoption.

Takeaways

• The “borrow from future layers” strategy: If you’re training any deep model and your budget is tight, try shifting capacity from late layers to early ones. The paper shows this works for Transformers, Gated Attention, and other architectures. Implement with a cosine schedule for MLP width. It’s a one-line change in most codebases: replace d_ff with a depth-dependent function.
• The control experiment matters: The reverse taper (narrow early, wide late) performs worse, proving it’s not just “any non-uniformity helps.” This teaches a methodological lesson: when proposing a new distribution, always include the inverse as a control.
• Scheduling intuition: Cosine taper beats linear and exponential because most of the capacity is preserved until late layers where redundancy is highest. This pattern might generalize to other architectural decisions — for example, tapering the number of attention heads per layer could also work. A practitioner can experiment with this heuristic.