Distance Generalization: Changing the Gaps Without Changing the Length

Paper: 2609.11913 Authors: Daniel Henrik Nevermann, Claudius Gros Categories: cs.CL

The Gap

Out-of-distribution length generalization — extrapolating a task from short to longer context — has been studied intensively. The paper isolates a different axis: distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length.

Separating the two axes matters because they fail for different reasons and are confounded in the usual setup. When you make a context longer, you change both the number of tokens and the distances among the ones that matter. So a model that fails on longer contexts might be failing at holding more items, or at locating items whose separation now exceeds anything seen in training. The paper’s design holds length fixed precisely so that only the second can vary.

   TWO AXES, OFTEN CONFOUNDED

   LENGTH GENERALIZATION (studied intensively)
     extrapolate a task from SHORT to LONGER context
        |
        v
   [DISTANCE GENERALIZATION] -- A DIFFERENT AXIS
     probe performance WHEN INTER-TOKEN DISTANCES ARE CHANGED
     BETWEEN TRAINING AND INFERENCE,
     WHILE KEEPING A FIXED CONTEXT LENGTH
        |
        v
   WHY SEPARATING THEM MATTERS
     making a context longer changes BOTH
       the NUMBER OF TOKENS
       AND the DISTANCES AMONG THE ONES THAT MATTER
        |
        v
   -> a model failing on longer contexts might be failing at
      (a) HOLDING MORE ITEMS, or
      (b) LOCATING ITEMS WHOSE SEPARATION NOW EXCEEDS ANYTHING
          SEEN IN TRAINING
   -> the paper HOLDS LENGTH FIXED PRECISELY SO ONLY (b) CAN VARY

The Increment

One sentence: Before this paper, length and distance effects were entangled in length-generalization studies; after it, two synthetic delay-copy tasks isolate distance generalization and pose three questions about positional encoding, data diversity, and when transfer helps.

Core Mechanism

Two synthetic delay-copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively. The task family is chosen for the same reason the axis is: a copy task has a definite correct answer, so a failure is a failure to locate or retain rather than an ambiguous quality judgement. And the “delay” is literally the distance being varied, so the independent variable is the thing under study. The refinement from full to selective copying adds a discrimination requirement on top of pure retention.

The three questions are the paper’s structure, and they are the right decomposition of the problem:

• (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding? This is the question with the most immediate engineering relevance. If relative-position schemes help, they are the tool for this failure mode; if a model with no positional encoding does just as well, then the architectural story people tell about these schemes needs revisiting for this axis.
• (B) How does data diversity — the number of inter-token distances seen in training — affect performance? So the training distribution is a variable, not just the architecture. It is the question that determines whether the fix is architectural or a matter of data.
• (C) When is distance transfer learning positive or negative? This is the sharpest one, because it allows that training on some distances might hurt on others. A transfer effect that can be negative means diversity is not monotonically good — the same non-monotonicity that appears in other generalisation settings.

The paper’s stated conclusion is deliberately about method: it finds that it is paramount to improve our understanding of the underlying mechanisms. That is a claim that the questions are not yet settled, which is worth taking at face value — the paper positions itself as constructing the instrument and asking the right questions rather than as delivering a decisive answer.

   TWO SYNTHETIC DELAY-COPY TASKS
     both involve FINITE DISTANCES between source and recall
     tokens copied either FULLY or SELECTIVELY
       <- the task family is chosen for the SAME REASON as the axis:
          a copy task has a DEFINITE CORRECT ANSWER
          -> a failure is a failure to LOCATE OR RETAIN, not an
             ambiguous quality judgement
       <- the "DELAY" is literally THE DISTANCE BEING VARIED
          -> the independent variable IS the thing under study
       <- the refinement from FULL to SELECTIVE copying adds a
          DISCRIMINATION requirement ON TOP OF pure retention

   THE THREE QUESTIONS ARE THE PAPER'S STRUCTURE

     (A) DO POSITIONAL ENCODING SCHEMES (RoPE, ALiBi) IMPROVE
         DISTANCE RESOLUTION relative to NO POSITIONAL ENCODING?
           <- the most immediately engineering-relevant question
           <- if RELATIVE-POSITION schemes help, they are the tool
              for this failure mode
           <- if a model with NO POSITIONAL ENCODING does just as
              well, the architectural story told about these schemes
              NEEDS REVISITING FOR THIS AXIS

     (B) HOW DOES DATA DIVERSITY -- THE NUMBER OF INTER-TOKEN
         DISTANCES SEEN IN TRAINING -- AFFECT PERFORMANCE?
           <- the TRAINING DISTRIBUTION is a VARIABLE, not just the
              architecture
           <- determines whether the fix is ARCHITECTURAL or a matter
              of DATA

     (C) WHEN IS DISTANCE TRANSFER LEARNING POSITIVE OR NEGATIVE?
           <- the SHARPEST question: it ALLOWS that training on some
              distances might HURT on others
           <- a transfer effect that CAN BE NEGATIVE means DIVERSITY IS
              NOT MONOTONICALLY GOOD
              <- the same non-monotonicity that appears in other
                 generalisation settings

   THE PAPER'S STATED CONCLUSION IS DELIBERATELY ABOUT METHOD:
     it finds that IT IS PARAMOUNT TO IMPROVE OUR UNDERSTANDING OF THE
     UNDERLYING MECHANISMS
       <- a claim that the questions are NOT YET SETTLED
       -> positioned as CONSTRUCTING THE INSTRUMENT and ASKING THE
          RIGHT QUESTIONS, rather than as delivering a DECISIVE ANSWER

Think of it as testing whether a reader can still follow a text when the puzzles are spaced differently. Length generalization lengthens the page: more items, and inevitably wider spacing. Distance generalization keeps the page the same length and moves the items: the hard question becomes whether the reader locates a referenced item by counting — a positional strategy — or by recognising its relation to what came before — a relational one. A counting reader breaks as soon as the spacing shifts; a relational reader should not care. Which is why question (A) is about whether relative-position schemes help: they are the architectural bet that relational information is what is needed. And question (C) allows that exposure to some spacings might harm others, which is what makes diversity a variable rather than a dial to turn up.

Key Concepts

• Distance versus length generalization: changing inter-token distances at fixed context length, rather than lengthening the context. It isolates an axis that length studies confound with item count.
• Delay-copy tasks as the instrument: definite answers with the distance as the independent variable. Failing is unambiguous, and selective copying adds discrimination to retention.
• Question (A) as an architectural test: whether RoPE and ALiBi beat no positional encoding for distance resolution. If they do not, the received rationale for these schemes does not transfer to this axis.
• Data diversity as a variable: the number of distinct distances seen in training. It decides whether the remedy is architectural or a matter of training distribution.
• Transfer that can be negative: some distances may hurt others, making diversity non-monotone. It is why question (C) is the sharpest of the three.

Framework Shift

Before (length generalization as the axis):
  test extrapolation from short to longer contexts
  -> length and distance vary together
  -> a failure is not attributable to item count or to spacing
  -> the positional-encoding question is asked in a confounded setting

After (distance generalization isolated):
  two delay-copy tasks at fixed context length
  (A) RoPE / ALiBi versus no positional encoding
  (B) number of distances seen in training
  (C) when transfer is positive or negative
  -> found: understanding the mechanisms remains paramount

From asking whether a model handles longer contexts, to asking whether it handles the same context with different spacings, the core shift is that locating an item is a distinct capability from holding more items, and it deserves its own instrument.

Expert Assessment

Problem choice: Excellent, and the axis separation is the contribution. Length generalization is one of the more actively studied failure modes, and pointing out that the standard setup entangles two variables — count and distance — identifies a confound that has been sitting in the literature.

Method maturity: The task design is well matched to the question: copy tasks give definite answers, and varying the delay makes the distance literally the independent variable. The three questions are a sensible decomposition, and question (C) in particular is well posed, because allowing negative transfer is what makes the diversity question empirical rather than rhetorical. The structural claim about the value of mechanism understanding is honest about the paper’s position.

Experimental integrity: The stated conclusion is unusually modest for this genre, and that is the right register: the paper builds the instrument and reports that the underlying mechanisms need more work rather than over-claiming from two synthetic tasks. The limitation follows from the same choice: synthetic delay-copy tasks establish clean attribution but say little on their own about natural-language dependencies, where “distance” is not a well-defined single quantity. The three questions are framed rather than definitively answered.

Writing quality: The title’s rhetorical question is well chosen, because it names the assumption under test — that positional encoding is necessary for distance resolution — and the three questions give the reader the paper’s map immediately. Because the value is in the experimental design, a compact description of the two tasks and what “selective copying” requires would help a reader judge how far the findings could extend.

Verdict: accept — it isolates a genuinely confounded axis in a heavily studied area, supplies a clean instrument for it, and frames the right questions while being candid that the mechanisms remain to be understood.

Takeaways

• Separate count from spacing when testing extrapolation. Lengthening a context changes both, so a failure is not attributable to either.
• Make the independent variable the thing under study. A delay-copy task turns the distance into a parameter rather than a side effect.
• Allow that transfer can be negative. If training on some spacings can hurt others, diversity is not a dial to turn up.
• Prefer clean attribution over realism when building an instrument. Synthetic tasks tell you what causes what; natural-language tests come after the mechanism is understood.