Beyond Sequence Order: Syntax-Informed Positional Embeddings

Paper: 2608.06111 Authors: Haris Riaz, Hyungji Kim, Mihai Surdeanu Categories: cs.CL, cs.AI

The Gap

Self-attention is permutation-invariant, so Transformers need positional embeddings to know anything about order at all. Three families have won out: absolute (add a position vector to the input), relative (bias the attention score by the distance between two tokens), and rotary (rotate query and key vectors by an angle proportional to position). All three encode the same underlying quantity — linear distance in the token sequence.

That quantity is a poor proxy for what actually determines how words relate. In “the keys to the cabinet are on the table,” the verb agrees with keys, four tokens away, not with cabinet, one token away. Linear distance says the wrong thing; the dependency structure says the right one. Positional embeddings are, in the paper’s phrasing, largely agnostic to syntactic structure.

There is a long line of work trying to fix this by injecting syntax, and it has a consistent and telling failure signature: syntax-injection methods improve syntactic generalization while degrading perplexity. That trade is the field’s standing result, and it is the reason none of these methods became default equipment. If forcing syntactic structure into a language model makes it a worse language model, the natural conclusion is that the syntax is fighting the objective rather than helping it — that the model was already extracting whatever structure it needed, and the injected prior is a constraint rather than information.

A second, more practical problem: existing syntactic language models tend to either marginalize over many parses at inference time — expensive, and a permanent tax on serving — or use syntax during training and discard it at runtime, which limits what the syntax can do.

Positional embeddings encode LINEAR DISTANCE only
         |
         v
But grammatical relatedness != token distance
("the keys to the cabinet ARE ...")
         |
         v
Prior syntax-injection work
         |
         +-----------------------+
         |                       |
         v                       v
  syntax metrics UP        perplexity DOWN
  (nearly every method)    (the standing trade-off)
         |                       |
         +----------+------------+
                    |
                    v
   Also: marginalize over parses at inference (costly)
         OR discard syntax at runtime (limited)
                    |
                    v
   [This paper] SiPE: put the prior in the POSITION channel,
   learned during pretraining, one parse at inference
                    |
                    v
   Evidence: SyntaxGym +10.3% AND perplexity -9.0%
             GLUE +8.2%
                    |
                    v
   The trade-off was not fundamental --
   it was about WHERE the prior enters

The Increment

One sentence: Before this paper, injecting syntax into a language model cost you perplexity; after it, a syntactic prior placed in the positional channel improves syntactic generalization, perplexity, and downstream understanding simultaneously.

Core Mechanism

SiPE — Syntax-informed Positional Embeddings — learns a lightweight syntactic prior from dependency parses during pretraining and injects it into the position channel. The architectural restraint is the point: self-attention and the rest of the model are left untouched. Nothing new is bolted onto the network; the existing positional mechanism is given better information about what “position” should mean. And it is applied across all three dominant PE families — absolute, relative, and rotary — and to both encoders and decoders, which is what turns it from a trick for one architecture into a general claim about where syntax belongs.

The paper’s most useful empirical contribution is not that this works but the careful isolation of where and how the prior should enter, and the finding that the answer is architecture-dependent. For autoregressive decoders using relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score. The authors compared this against injecting into the input embeddings, into self-attention, and into the positional and attention terms jointly — and the multiplicative coupling with the relative-position term won. For encoders, the opposite: the prior is best added directly to the input embeddings, where it composes with whatever native positional mechanism the encoder already has. The same prior, entering at different points, depending on the architecture.

The headline result is that the standing trade-off simply does not appear. Models pretrained with SiPE improve on SyntaxGym by up to 10.3% while reducing perplexity by 9.0% over a base model with no syntactic supervision — where, as the paper notes, nearly every existing syntax-injection method degrades perplexity instead. The gains also extend past syntactic generalization into real language understanding, raising GLUE scores by up to 8.2%. On cost, SiPE conditions on a single parse rather than marginalizing over many, which places it on a new Pareto frontier between syntactic supervision and inference cost.

   Where the prior enters (the paper's core ablation)

   dependency parses (pretraining only)
              |
              v
      +------------------+
      | lightweight      |
      | syntactic prior  |
      +------------------+
              |
      +-------+--------+
      |                |
      v                v
   DECODER          ENCODER
   (relative PE)
      |                |
      v                v
  multiply into    add into input
  the relative-    embeddings,
  position term    composing with
  of attention     native PE
  score
      |                |
      v                v
   beat: input-emb injection      (self-attention
         self-attention injection  untouched in
         joint pos+attn injection  both cases)


   Inference cost comparison

   prior syntactic LMs:  marginalize over MANY parses   $$$
                    OR:  discard syntax at runtime      (weak)

   SiPE:                 condition on ONE parse         $

Here is the structural metaphor. Think of a Transformer reading a sentence as a traveler navigating a city, where positional embeddings are the map. Standard positional embeddings give you a map with straight-line distances only — Manhattan-style, as the crow flies. Two buildings 100 meters apart are marked as close. That is usually a decent proxy for how easy it is to get from one to the other, which is why plain PE works as well as it does.

But sometimes a river runs between them, and the nearest bridge is a kilometer away. And sometimes two buildings a kilometer apart sit on the same express line, one stop apart. Straight-line distance systematically misleads in exactly the cases that matter — which is the subject-verb agreement problem, where keys and are are far apart on the page but adjacent in the grammar.

The prior work’s mistake, in this metaphor, was to hand the traveler a separate transit map and ask them to reconcile it with the street map on the fly. That is the marginalizing-over-parses approach: expensive, and the two maps sometimes disagree. Alternatively, some methods showed the transit map during training and took it away before the trip — helpful for learning the city’s shape, useless at the moment of navigation.

SiPE instead redraws the distances on the original map. There is no second map and no reconciliation step; the notion of “how far apart” is simply corrected to account for the bridges and the express lines. And this is precisely why perplexity improves instead of degrading. A traveler with a separate, conflicting map moves worse. A traveler with a better map moves better at everything, including the ordinary walking they were already good at.

The architecture-dependence of the injection point also becomes legible here. A decoder reading left-to-right is navigating in real time, deciding each step from where it currently stands — so the correction belongs in the step-cost function, multiplied into the relative-position term as it computes each move. An encoder seeing the whole sentence at once is studying the map before setting out — so the correction belongs in the map itself, added to the input representation.

Key Concepts

• The perplexity trade-off, and why it was not fundamental: The central puzzle. Nearly every prior syntax-injection method improved syntactic metrics while making the model a worse language model, and the field largely read this as the price of imposing structure. SiPE’s result suggests the trade was an artifact of where the structure entered. Injected into input embeddings or attention for a decoder, a syntactic prior competes with the representational work those components are already doing — it displaces information. Injected multiplicatively into the relative-position term, it modulates a quantity the model was already computing and already using as distance, so it refines rather than displaces. The lesson is sharper than “syntax helps”: it is that a prior which corrects an existing signal behaves completely differently from one that adds a competing signal.

• Single parse versus marginalization: Syntactic ambiguity is real — a sentence can have many valid dependency parses — and the principled response is to marginalize over them, weighting each by probability. The problem is that this must happen at inference, every time, forever. SiPE conditions on a single parse instead. This is theoretically weaker and practically decisive, and the paper is right to frame it as a Pareto frontier rather than a strict improvement: you give up parse uncertainty and you get a model that costs the same to serve as a normal one. Note also that parses are only needed during pretraining for the prior to be learned, which is a one-time cost paid by whoever trains the model, not by whoever runs it.

• Injecting into the position channel rather than the representation channel: The architectural idea worth carrying away. A Transformer has two broad places to put information about a token: into its representation (what this token is) or into the positional machinery (how this token relates to others). Syntax is relational information — a dependency arc is a fact about a pair, not about a word — so the position channel is where it structurally belongs. Prior work often put it in the representation channel, which is a type mismatch: encoding a fact about a pair by modifying one member of the pair. Once stated this way, the perplexity result stops being surprising.

Framework Shift

Before (prior syntax injection):     After (SiPE):

  parse tree                          parse tree
      |                                   |
      v                                   v
  inject into REPRESENTATION          inject into POSITION
  (input embeddings, or a             channel -- the thing
  separate syntax module)             that already encodes
      |                               "how tokens relate"
      v                                   |
  competes with the                       v
  representational work                 refines an existing
  those components do                   signal instead of
      |                                 adding a competing one
      v                                   |
  syntax metrics UP                       v
  perplexity DOWN                     syntax UP (+10.3%)
  (the standing trade-off)            perplexity DOWN (-9.0%)
      |                               GLUE UP (+8.2%)
      v                                   |
  inference: marginalize                  v
  over many parses ($$$)              inference: one parse ($)
  or discard syntax

From treating syntax as extra content to be added to a model’s representations, to treating it as a correction to the model’s notion of distance, the core shift is recognizing that relational information belongs in the channel that already encodes relations.

Expert Assessment

Problem choice: A real gap, and an unusually well-motivated one because the paper is not merely proposing another syntax-injection method — it is explaining why the previous ones traded away perplexity. The field had accumulated a consistent negative result and largely accepted it as a law; showing it was a consequence of an implementation choice is a more valuable contribution than the accuracy numbers. That said, the strategic question hanging over all syntax-injection work applies here too: at sufficient scale, models learn syntactic structure without help, and priors that pay off at moderate scale often stop paying off. The paper’s contribution is real regardless, but its long-run relevance depends on an extrapolation not tested here.

Method maturity: Genuinely clever, and the cleverness is in restraint — leaving self-attention untouched and improving the information the existing positional mechanism receives is a much better idea than adding a syntax module. The where-and-how ablation is the strongest part of the paper and the part most likely to be reused, because “multiplicatively into the relative-position term for decoders, additively into input embeddings for encoders” is a specific, actionable finding rather than a general endorsement. The obvious dependency is on parse quality: the whole method is conditioned on a dependency parser, and the paper’s framing does not foreground what happens as parse accuracy degrades — which is exactly what happens on the informal, code-mixed, and non-English text that a large share of real pretraining data consists of. A single parse is cheap, but a single wrong parse is a confidently wrong prior.

Experimental integrity: The result that carries the paper is the simultaneous improvement in SyntaxGym and perplexity, and it is the right result to lead with precisely because it contradicts the field’s expectation — a paper claiming its method also improves perplexity is making a falsifiable claim that reviewers will check. Covering all three PE families and both encoders and decoders is more thorough than this literature usually manages and substantially raises confidence that the effect is about the mechanism rather than a lucky configuration. The GLUE improvement of “up to 8.2%” is the number I would treat most cautiously: “up to” over a multi-task benchmark usually means one task moved a lot, and GLUE contains small, high-variance tasks where large relative swings are routine. The claim that gains extend beyond syntax into real understanding needs the average, not the maximum. Scale is the other open question — these are pretraining experiments, and the relevant sizes are not evident from the abstract.

Writing quality: The abstract is dense but well-organized, and the decision to state the where-and-how finding explicitly rather than burying it in an appendix reflects good judgment about what readers will actually use. The corner cut is the reliance on “up to” for both headline downstream numbers, which is the standard way to make a result look larger than its average and invites more skepticism than the underlying work probably deserves. The section that would most elevate the paper is a direct analysis of why multiplicative coupling with the relative-position term wins for decoders — the empirical finding is established, but the mechanism is left for the reader to reconstruct, and that mechanism is the paper’s most generalizable insight.

Verdict: weak accept — A well-executed idea with the right architectural instinct and a result that overturns an assumed trade-off, held back by “up to” reporting on downstream benchmarks and an unexamined dependency on parser quality.

Takeaways

Three things a practitioner can steal:

1. Put relational information in the relational channel. The transferable principle, well beyond syntax: if the fact you want to inject is about a pair of elements, encoding it by modifying one element is a type mismatch that makes it compete with that element’s own representation. Attention biases and positional terms are where pairwise facts belong. This applies to injecting graph structure, document hierarchy, table layout, code AST relationships, or retrieval provenance into a Transformer.

2. When a prior degrades your main objective, suspect the injection point before the prior. The field read “syntax injection hurts perplexity” as evidence that syntax was in tension with language modeling. It was evidence that the syntax was displacing information rather than refining it. Before concluding that auxiliary knowledge is unhelpful, try entering it where it corrects an existing signal rather than where it adds a new one — a prior that modulates a quantity the model already computes behaves entirely differently from one that competes for representational capacity.

3. Ask whether marginalization is worth its permanent inference tax. Marginalizing over parses is the theoretically correct handling of ambiguity, and it costs you at every request forever. Conditioning on a single best hypothesis is theoretically weaker and often practically dominant. This trade recurs constantly — over retrieval results, over tool-call candidates, over segmentations — and the honest framing is a Pareto frontier, not a right answer. Also worth noting: SiPE needs parses only at pretraining time, which is the most favorable version of this trade, since the cost lands once on the trainer rather than repeatedly on the server.