Instruction-Tuned LLMs Copy Human Syntax More Than Humans Do — But Less Precisely

Paper: 2607.26015 Authors: Zandi Eberstadt Categories: cs.CL

The Gap

We know humans converge on each other’s syntax in dialogue — you start talking like the person you’re chatting with, and you don’t even notice. This “syntactic convergence” is well-studied in psycholinguistics. On the LLM side, we have growing evidence that models mimic surface-level features of their context (lexical priming, style matching), and we know instruction tuning changes model behavior dramatically. But there’s a specific blind spot: nobody has systematically asked whether LLMs exhibit grammatical convergence toward human interlocutors the way humans do — measured across a broad range of syntactic constructions, benchmarked against human baselines, and compared between pretrained and instruction-tuned variants of the same architectures.

Prior work either looked at lexical/semantic priming (ignoring syntax), studied syntactic patterns in model training data (not live dialogue), or examined individual constructions without breadth. The gap is a missing bridge: we don’t know if the syntax-reuse behavior in LLMs is human-like in degree or selectivity, and we don’t know what instruction tuning does to this capacity.

Prior knowledge                 This paper's contribution
                                                    
Human syntactic                 Do LLMs show analogous
convergence is real     ?       convergence vs human
+ well-documented               baselines?
    |                               |
    v                               v
LLMs mimic surface      -->     Measured CFG-rule reuse
features (lexical,              in 16 models (1B-70B)
style) in context               at 1,901 matched positions
    |                               |
    v                               v
Instruction tuning              Compared pretrained vs
changes model                   instruction-tuned on
behavior broadly                same architectures
    |                               |
    v                               v
Gap: no systematic              Finding: instruction-
syntax-level measure            tuned models over-converge
in live dialogue,               vs humans, but diffusely
vs human baselines              (less selectively targeted)

The Increment

One sentence: Before this paper, we didn’t know whether LLMs reuse human grammatical structures in dialogue the way humans do — now we know they over-do it, and instruction tuning makes the over-convergence more diffuse rather than more targeted.

Core Mechanism

The study uses a “substitution paradigm” — imagine taking real human dialogues, pulling out one speaker’s turn, and replacing it with a model-generated response given the same preceding context. This lets you compare, position by position, how much the model’s output reuses the syntactic rules from the preceding human turn versus how much the original human response did.

The syntactic measurement works through context-free grammar (CFG) rule extraction. Each sentence is parsed into a set of CFG production rules (e.g., S -> NP VP, NP -> Det N). For each matched position, the study computes the overlap between the rules in the preceding human turn and the rules in the model’s output (or the original human response). To create a baseline, each model’s output is also compared against an “unrelated prime” — a human turn from elsewhere in the dialogue that isn’t contextually adjacent.

The models tested span 16 variants: Llama 2 and Gemma families, sizes from 1B to 70B parameters, each in both pretrained and instruction-tuned versions. That gives 8 matched architecture pairs. At each of 1,901 matched positions, the study measures: (1) actual-prime overlap vs. random-prime overlap, (2) how this difference varies by rule frequency, and (3) how instruction tuning shifts these patterns.

Data Pipeline:
                                          
Human Dialogue    Extract one     Preceding human turn
(corpus)          speaker's       (context for model)
    |             turn out            |
    v                 |               v
    Original     Substitution    Model generates
    human    <-- paradigm  -->   replacement response
    response                         |
        |                            |
        v                            v
   CFG parse                    CFG parse
   (set of rules)               (set of rules)
        |                            |
        +---------- compare ---------+
        |                            |
        v                            v
   Rule overlap               Rule overlap
   (human resp vs             (model resp vs
    preceding turn)            preceding turn)
        |                            |
        +---- also compare with -----+
             unrelated prime baseline
                   |
                   v
            Difference scores:
            actual-prime vs random-prime
            (broken down by rule frequency)

Think of it like a recipe-taste test. You have a dinner conversation between two people — one says something, the other responds. Now you replace the second person’s response with a dish cooked by a chef (the LLM) given only the first person’s ingredients (the preceding turn). You then check: does the chef’s dish reuse the same spice combinations (syntactic rules) as the first person’s cooking? And does it reuse them more than a random dish from a completely different meal? Humans show moderate spice-matching — they pick up on some of the interlocutor’s flavor patterns. The instruction-tuned models? They dump in all the same spices, even more than the original human respondent did. But here’s the twist: they also dump in those same spices when you give them ingredients from a random, unrelated meal. They’re not selectively tracking which spices belong to this conversation — they’re just seasoning everything heavily with whatever’s in front of them.

Key Concepts

• Syntactic convergence: When two people talk, they unconsciously start using similar grammatical structures. If your conversation partner uses lots of passive constructions or relative clauses, you’ll drift toward those patterns too — without realizing it. It’s like how you might start adopting someone’s posture or speech rhythm during a long chat. The key word is *unconscious: this happens below awareness, and it’s thought to smooth social interaction. The paper asks: do LLMs do this too, and if so, is their version of it human-like?

• CFG-rule overlap: A context-free grammar (CFG) breaks a sentence into a tree of production rules — “a sentence is a noun phrase plus a verb phrase,” “a noun phrase is a determiner plus a noun,” etc. Two sentences that share many of these rules have similar syntactic skeletons, even if they use completely different words. “The cat sat on the mat” and “A dog slept on the rug” share rules like S -> NP PP, PP -> P NP. Measuring overlap means counting how many of these structural building blocks two sentences have in common. It’s a way to quantify “syntactic similarity” without caring about meaning.

• Instruction tuning’s paradox: Pretrained models learn from raw text; instruction-tuned models are further trained to follow human instructions and produce helpful responses. You’d expect tuning to make models *more human-like. But this paper reveals a paradox: tuning increases the raw amount of syntax reuse (models copy more rules from context), yet it decreases the specificity of that copying (models also copy more from unrelated context). It’s like the difference between a mirror and a sponge — a mirror reflects what’s in front of it selectively; a sponge absorbs everything. Instruction tuning makes models more sponge-like.

Framework Shift

Before (mainstream view):           After (this paper):

"LLMs match style/lexicon"          "LLMs over-converge on
         |                           syntax beyond human
         |                           baseline, but diffusely"
         v                                   |
"Syntax is a harder,                Instruction tuning
 less-studied dimension"            increases raw convergence
         |                           but dilutes selectivity"
         |                                   |
         v                                   |
"Instruction tuning makes           Tuning turns a selective
 models more human-like"            mirror into a sponge

From “LLMs mimic surface features, and tuning makes them more human-like” to “tuning amplifies syntactic convergence past human levels while eroding the targeting precision” — the core shift is that instruction tuning trades selectivity for volume.

Expert Assessment

Problem choice: This is a genuine and interesting gap. Syntactic convergence is a well-established psycholinguistic phenomenon, and the question of whether LLMs exhibit it — and whether instruction tuning changes it — sits at a natural intersection of computational linguistics and cognitive science. The field has been moving from “do models understand language?” toward “do models process language the way humans do?”, and this paper contributes a concrete, measurable piece to that puzzle. Not manufactured.

Method maturity: The substitution paradigm is clever and well-motivated — it gives you a controlled comparison without needing paired human-model dialogues from scratch. The CFG-rule measurement is a known technique applied in a fresh context. The 1,901 matched positions across 16 models give decent statistical power. That said, CFG rules are a somewhat coarse-grained syntactic representation; they don’t capture long-distance dependencies or discourse-level patterns. The study acknowledges this implicitly by using it as a broad measure, but a cynic might ask: is this capturing syntactic *convergence or just syntactic complexity matching? The unrelated-prime baseline helps, but doesn’t fully eliminate this concern.

Experimental integrity: The architecture-pair comparison (pretrained vs. instruction-tuned, same base) is the right design and eliminates many confounds. The frequency-stratified analysis (lower-frequency rules showing larger actual-vs-random differences) is a nice touch that suggests genuine structural sensitivity rather than a trivial confound. One concern: the paper tests 16 models but all from the Llama/Gemma families — generalization to other architectures (Mistral, Qwen, etc.) is assumed but not demonstrated. Also, the “substitution paradigm” means models are generating in a dialogue context that was originally human, which introduces a distribution shift that’s hard to fully disentangle from the convergence effect.

Writing quality: The abstract is dense but well-organized — it reports each finding clearly. The paper appears to structure its results in a logical flow (overall overlap → frequency effects → instruction tuning effects → lexical/semantic comparison). Without seeing the full text, the main area I’d push on is the interpretive discussion: the finding that instruction tuning increases convergence but decreases selectivity is the headline insight, and it deserves a more developed mechanistic explanation. *Why does tuning diffuse the targeting? Is it the RLHF objective? The instruction-following format? The training data distribution? This is where the paper could go from “interesting observation” to “theoretical contribution.”

Verdict: weak accept — The question is real, the experimental design is sound, and the core finding (instruction tuning trades selectivity for volume in syntactic convergence) is a genuine insight that should influence how we think about what instruction tuning does to language processing. Limited by narrow model families and a somewhat coarse syntactic representation.

Takeaways

For LLM researchers: If you’re studying how models mirror their context, don’t stop at lexical and semantic similarity — syntax is a distinct and measurable dimension where models behave differently from humans, and instruction tuning has a non-obvious effect (more copying, less precision). The substitution paradigm is a reusable experimental framework: take any human corpus, replace turns with model outputs, and measure structural alignment.

For alignment/RLHF work: The “sponge vs. mirror” finding suggests instruction tuning may be teaching models to *absorb context broadly rather than adapt selectively. If you want models that adjust their grammar to the user without over-mimicking everything, current tuning methods may be working against you. This could inform more targeted training objectives that reward selective convergence.

For cognitive science: The paper gives you a concrete benchmark: human syntactic convergence has a specific “dose” — models exceed it. This asymmetry could be useful as a diagnostic for model-human alignment studies, or as a tool to understand what aspects of human convergence are social-cognitive (intentional, audience-designed) versus purely mechanistic (statistical priming).