Implicit Representations of Grammaticality in Language Models

Paper: 2605.05197 Authors: Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim Categories: cs.CL

The Gap

Language models generate grammatical text and distinguish grammatical from ungrammatical sentences in minimal pairs. But their string probabilities don’t cleanly separate grammatical from ungrammatical sentences overall. Prior work assumed that if LMs don’t show grammaticality in their output probabilities, they don’t represent it internally. This paper challenges that assumption: maybe grammaticality lives in the hidden layers, not the probability distribution.

Problem: LM probabilities blur grammaticality
         (grammatical sentences can have low P, ungrammatical can have high P)
           |
           v
Assumption: Grammaticality ≠ likelihood in human language
            Maybe LMs learn this distinction internally?
           |
           v
Method: Train linear probe on hidden states
        (grammatical vs synthetic ungrammatical sentences)
           |
           v
Evidence: Probe generalizes to human benchmarks
          Outperforms string probability
          Works cross-lingually
          Weakly correlates with probability
           |
           v
Conclusion: LMs acquire implicit grammaticality representation
            distinct from string probability

The Increment

One sentence: Before this paper, we judged LM grammatical knowledge by output probabilities; after, we know grammaticality is encoded in hidden layers and can be extracted with simple probes.

Core Mechanism

The method has three components: a synthetic ungrammatical dataset, a linear probe, and a generalization test. First, they create ungrammatical sentences by applying rule-based perturbations to naturalistic text (swap word order, delete articles, violate agreement). Second, they train a linear classifier on LM hidden states to distinguish grammatical from ungrammatical sentences. Third, they test whether this probe generalizes to human-curated grammaticality benchmarks it never saw during training.

Naturalistic corpus
      |
      v
[Perturbation rules] ---> Synthetic ungrammatical sentences
      |                              |
      v                              v
Grammatical sentences          Ungrammatical sentences
      |                              |
      +------------------------------+
                    |
                    v
            [Language Model]
                    |
                    v
            Hidden states (h)
                    |
                    v
        [Linear probe: w·h + b]
                    |
                    v
        Grammaticality score
                    |
                    v
    Test on human benchmarks (BLiMP, etc.)

Think of the LM as a factory inspector examining products on a conveyor belt. The inspector has two jobs: estimate how common each product is (string probability) and check if it meets quality standards (grammaticality). Prior work only looked at the inspector’s frequency estimates. This paper discovered the inspector keeps internal quality notes in a logbook (hidden states). By training a simple decoder (linear probe) to read those notes, they can extract the quality judgment directly, which turns out to be more accurate than inferring quality from frequency alone. The key insight: the logbook exists and is readable, even though the inspector doesn’t explicitly publish quality scores.

Key Concepts

• Linear probe: A simple classifier (just multiplication and addition) trained to predict a property from a model’s internal representations. If a linear probe can extract information, it means that information is explicitly represented in the hidden states, not buried in complex nonlinear patterns. It’s like checking if a book’s index can help you find a topic—if yes, the topic is well-organized in the text.

• Synthetic ungrammatical data: Sentences created by applying rule-based perturbations to real text (e.g., “The cat sit on mat” from “The cat sits on the mat”). This is cheaper than human annotation but risks creating artifacts—patterns that distinguish synthetic errors from real grammatical sentences but don’t reflect true grammaticality. The test of quality: does a probe trained on synthetic data generalize to human-curated benchmarks?

• Grammaticality vs likelihood: In human language, a sentence can be grammatically perfect but unlikely (“Colorless green ideas sleep furiously”) or grammatically broken but predictable in context. LMs trained on likelihood might conflate these. The question: do LMs learn to separate them internally, even if their output probabilities don’t?

Framework Shift

Before (mainstream approach):        After (this paper):

LM output probabilities              LM hidden states
        |                                   |
        v                                   v
P(sentence) as proxy            [Linear probe]
for grammaticality                      |
        |                               v
        v                    Explicit grammaticality
Conflates frequency              representation
and grammaticality                      |
                                        v
                              Generalizes cross-lingually
                              Outperforms P(sentence)

From treating probability as the only window into LM knowledge to recognizing that internal representations encode distinctions not visible in output distributions, the core shift is: LMs are not just probability estimators—they’re structured representers.

Expert Assessment

Problem choice: Real gap. The disconnect between LM probabilities and human grammaticality judgments has been a known issue, but most work either accepted it or tried to fix probabilities. Asking whether grammaticality is represented internally but not surfaced is a natural next question that hadn’t been systematically explored.

Method maturity: Elegant simplicity. Linear probes are the right tool here—if grammaticality is linearly separable in hidden states, it’s a strong signal. The synthetic perturbation approach is pragmatic but introduces risk: the probe might learn to detect perturbation artifacts rather than grammaticality. The cross-lingual and human-benchmark generalization tests partially address this, but I’d want to see ablations on perturbation types.

Experimental integrity: Baselines are fair. The comparison to string probability is the right one. The cross-lingual results are impressive and suggest the probe isn’t just overfitting to English-specific patterns. One concern: the probe is trained on synthetic data and tested on human benchmarks, which is good for generalization but makes it hard to know the probe’s ceiling. What’s the human agreement on these benchmarks?

Writing quality: Clear and well-structured. The introduction sets up the problem crisply. The results section could be tighter—some tables feel redundant. The discussion of semantic plausibility (where the probe fails) is honest and important but could be expanded. If I were revising, I’d add a section on failure modes and probe interpretability.

Verdict: strong accept — Challenges a core assumption about how LMs represent linguistic knowledge, with clean experiments and honest reporting of limitations.

Takeaways

• Probing as diagnostic: When a model’s outputs don’t reflect a distinction you care about, check if the distinction exists in hidden states. Linear probes are cheap to train and can reveal what the model knows but doesn’t say.

• Synthetic data for probing: You can bootstrap probe training with rule-based perturbations, but validate generalization on human-curated data. The gap between synthetic and real is where you learn whether you’re measuring the right thing.

• Cross-lingual transfer as validation: If a probe trained on one language works on others, it’s evidence you’re capturing something fundamental rather than surface patterns. This is a useful sanity check for any representation learning claim.