Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions

Paper: 2605.31586 Authors: Wesley Scivetti, Ethan Wilcox, Nathan Schneider, Kanishka Misra, Leonie Weissweiler Categories: cs.CL, cs.AI

The Gap

Prior work showed that only the largest LLMs (GPT-4 scale) could handle rare constructions—fixed form-meaning pairings like “let alone” or “much less” that carry specific semantic constraints. The field assumed this was a capability threshold: you either have 100B+ parameters or you fail. But no one systematically tested open-source models across different scales, architectures, and training data sizes. More critically, no one tracked when during training these models acquire constructional understanding, or what other knowledge correlates with that acquisition.

This paper fills both gaps: (1) Do modestly sized open models actually understand Paired-Focus constructions? (2) What learning dynamics underlie this understanding—does it emerge early with syntax, or later with semantics?

Problem: Rare constructions assumed to need massive scale
   |
   v
Assumption: Understanding = syntax + semantics, but which comes first?
   |
   v
Method: Novel dataset testing both form and meaning across model scales
   |      + training dynamics analysis on open-checkpoint models
   v
Evidence: Modest models succeed; semantics emerges AFTER syntax,
   |       correlated with world knowledge gains
   v
Conclusion: Constructional semantics is learnable at modest scale,
            but requires substantial pretraining data

The Increment

One sentence: Before this paper, we thought rare construction understanding required 100B+ parameters; after, we know 7B-13B models can do it if trained on enough data, and that semantic understanding lags behind syntactic knowledge by a measurable training interval.

Core Mechanism

The method has three components. First, a novel dataset targeting Paired-Focus constructions (PFCs) like “let alone” and “much less.” These constructions have a specific semantic constraint: the second item must be more extreme on some scale than the first (“He can’t run a mile, let alone a marathon”). The dataset tests both syntactic knowledge (can the model detect PFC forms?) and semantic knowledge (does it understand the scalar ordering?).

Second, a cross-sectional evaluation across 20+ models varying in size (1B to 70B parameters), architecture (decoder-only, encoder-decoder), and pretraining data scale (human-scale ~10B tokens vs web-scale 1T+ tokens). This isolates which factors matter for PFC understanding.

Third, a longitudinal analysis of training dynamics using models with public checkpoints (Pythia, OLMo). By testing each checkpoint on both syntactic and semantic PFC tasks, they track when each type of knowledge emerges and what other capabilities correlate with semantic acquisition.

Dataset Construction:
  Scalar adjectives + world knowledge -> PFC test items
       |                                      |
       v                                      v
  "tall/short"                    "run mile/marathon"
       |                                      |
       +--------------------------------------+
                        |
                        v
              PFC templates with controls
              ("let alone" vs "not to mention")

Evaluation Pipeline:
  Model checkpoint -> Syntactic probe -> Form sensitivity score
       |
       +-----------> Semantic probe -> Meaning accuracy
       |
       +-----------> World knowledge tasks -> Correlation analysis

Think of this like testing whether someone truly understands sarcasm versus just recognizing the tone. You need two tests: (1) Can they identify sarcastic sentences? (That’s syntax—recognizing the form.) (2) Do they grasp that the speaker means the opposite? (That’s semantics—understanding the meaning inversion.) This paper does both for PFCs, then adds a third dimension: watching a child grow up and noting when they start catching sarcasm versus when they start using it correctly themselves. The key insight: recognizing the sarcastic tone happens earlier than understanding the meaning flip, and the meaning flip correlates with general social reasoning development.

Key Concepts

• Paired-Focus Constructions (PFCs): Fixed phrases like “let alone,” “much less,” or “never mind” that connect two items with a built-in semantic rule: the second item must be more extreme than the first on some implicit scale. Example: “I can’t afford a bike, let alone a car” works because cars are more expensive than bikes. Flip it—“I can’t afford a car, let alone a bike”—and it sounds wrong because you violated the scalar ordering. PFCs are rare in text (appearing once per ~100K words), making them a stress test for whether models truly learn form-meaning mappings versus just memorizing common patterns.

• Constructional semantics: The idea that meaning isn’t just in individual words but in the *construction itself—the fixed pattern carries semantic constraints independent of the words filling its slots. “Let alone X” doesn’t mean “let” + “alone” + X; it means “X is even more extreme than what I just said, so if the first thing is impossible, X is definitely impossible.” Models that learn constructional semantics grasp this constraint; models that only learn syntax might recognize “let alone” as a valid phrase but fail to enforce the scalar ordering.

• Training dynamics vs cross-sectional evaluation: Cross-sectional is a snapshot—test many models at their final state and compare. Training dynamics is a movie—test one model at many points during training and watch knowledge emerge. Cross-sectional tells you *what matters (model size? data scale?). Training dynamics tells you when and how knowledge develops (does syntax come before semantics? what else is learned at the same time?). This paper does both: cross-sectional to identify which models succeed, training dynamics to understand the learning process.

Framework Shift

Before (mainstream approach):        After (this paper):

Rare construction understanding:     Rare construction understanding:
                                    
  [Massive scale]                     [Modest scale] + [Enough data]
        |                                    |              |
        v                                    v              v
   100B+ params                         7B-13B params   1T+ tokens
        |                                    |
        v                                    +---> Syntax first
   Success/Failure                                    |
   (binary threshold)                                 v
                                                 Semantics later
                                                      |
                                                      v
                                              (correlated with
                                               world knowledge)

One sentence: From “you need massive scale” to “you need sufficient data and patience,” the core shift is recognizing that constructional semantics is a late-emerging capability that piggybacks on world knowledge acquisition, not a parameter count threshold.

Expert Assessment

Problem choice: Real gap. The field genuinely didn’t know if open-source models could handle rare constructions, and the training dynamics question is novel. This sits at the intersection of linguistic theory (Construction Grammar) and LLM capabilities research—a productive boundary.

Method maturity: Solid execution. The dataset design is clever: using both scalar adjectives (objective ordering) and world knowledge (subjective but testable) gives two independent semantic probes. The training dynamics analysis is the paper’s strongest contribution—most capability studies are cross-sectional snapshots. One weakness: they don’t test whether models that fail semantic PFC tasks also fail *other scalar reasoning tasks, which would strengthen the claim that PFCs specifically test constructional understanding versus general scalar reasoning.

Experimental integrity: Baselines are fair. Testing 20+ models across architectures and scales is thorough. The correlation analysis between PFC semantics and world knowledge is suggestive but not causal—they show co-occurrence, not that one causes the other. The human-scale data result (models trained on <100B tokens fail all semantic tests) is striking but based on only a few models; more data points would be reassuring.

Writing quality: The paper front-loads linguistic theory, which will lose some ML readers. Section 3 (dataset construction) is dense—a worked example walking through one test item’s creation would help. The training dynamics section (5.2) is excellent and should be expanded. The conclusion undersells the practical implications: if semantic understanding requires 1T+ tokens, that’s a concrete data budget recommendation for anyone training models.

Verdict: weak accept — Solid empirical work with a novel training dynamics angle, but the theoretical framing (Construction Grammar) may limit impact in the ML community, and the causal story behind the correlations needs strengthening.

Takeaways

For model trainers: If you’re training a <10B parameter model and want it to handle rare constructions, budget for 1T+ tokens of pretraining data. Syntactic knowledge emerges earlier, but semantic understanding requires substantial world knowledge first.

For evaluation designers: Testing rare constructions requires probing both form recognition and meaning understanding separately. A model can ace syntactic tests while failing semantic ones—don’t conflate the two.

For capability researchers: Training dynamics analysis (testing checkpoints throughout training) reveals more than cross-sectional comparisons. The lag between syntactic and semantic acquisition suggests a two-stage learning process: first memorize the form, then infer the meaning from distributional patterns. This pattern likely generalizes to other linguistic phenomena.

Steal this technique: The correlation analysis between target capability (PFC semantics) and auxiliary tasks (world knowledge domains) is a simple but powerful way to generate hypotheses about what knowledge underlies a capability. Apply it to other “mysterious” emergent abilities.