Language Models Show Selective Typological Alignment in Differential Argument Marking

Paper: 2602.17653 Authors: Iskar Deng, Nathalia Xu, Shane Steinert-Threlkeld Categories: cs.CL

Abstract

This paper investigates whether language models can acquire typological preferences for differential argument marking (DAM), a morphological system where case marking depends on semantic prominence. By training GPT-2 models on 18 synthetic corpora implementing different DAM configurations, the authors discover a critical dissociation: models successfully learn the natural markedness direction (marking semantically atypical arguments), mirroring human language universals, but fail to reproduce the strong cross-linguistic preference for marking objects over subjects. This asymmetry suggests that different typological tendencies in human languages may emerge from fundamentally different cognitive or communicative pressures.

Key Contributions

• Extends synthetic language learning paradigm from syntactic phenomena to semantic licensing systems (differential argument marking)
• Demonstrates that GPT-2 models trained on controlled synthetic corpora can acquire natural markedness direction preferences consistent with human language typology
• Reveals a systematic failure of models to learn object preference in DAM systems, despite this being a robust cross-linguistic pattern
• Provides evidence that typological universals may arise from multiple distinct sources rather than a unified underlying mechanism
• Introduces a rigorous minimal pairs evaluation methodology for testing DAM generalization in language models

Methodology and Experimental Design

The researchers employed a controlled synthetic learning approach to isolate specific typological variables. They generated 18 distinct synthetic corpora, each implementing a different DAM system along two key dimensions:

Markedness Direction: Systems varied in whether overt morphological marking targeted semantically typical or atypical arguments. In natural languages, marking typically targets atypical arguments—for instance, animate objects receive overt case marking more often than inanimate ones, since objects are prototypically inanimate.

Argument Role: Systems varied in whether DAM applied to subjects or objects. Cross-linguistically, DAM systems overwhelmingly target objects rather than subjects, a pattern known as the “object preference.”

Each synthetic corpus was carefully controlled for vocabulary size, sentence structure complexity, and frequency distributions to ensure that observed learning patterns reflected genuine typological biases rather than statistical artifacts. GPT-2 models were trained from scratch on each corpus using identical hyperparameters, then evaluated using minimal pairs that tested generalization to novel semantic contexts.

The minimal pairs methodology involved presenting models with sentence pairs differing only in the presence or absence of DAM marking, then measuring whether models assigned higher probability to grammatical versus ungrammatical variants. This approach directly tests whether models have internalized the underlying semantic licensing conditions rather than merely memorizing surface patterns.

Results and Analysis

The experimental results revealed a striking dissociation between the two typological dimensions:

Markedness Direction: Models consistently demonstrated human-like preferences, assigning higher probabilities to systems where overt marking targeted semantically atypical arguments. For example, in object-marking systems, models preferred configurations where animate objects (atypical) received overt case marking while inanimate objects (typical) remained unmarked. This pattern held across different semantic features (animacy, definiteness, person) and emerged reliably across multiple training runs.

Object Preference: In stark contrast, models showed no systematic preference for object-marking over subject-marking DAM systems. When trained on subject-DAM and object-DAM corpora with identical statistical properties, models learned both systems equally well, failing to reproduce the strong cross-linguistic bias toward object marking observed in human languages.

Statistical analysis confirmed these patterns were robust and not attributable to random variation or specific corpus characteristics. The dissociation persisted across different model sizes and training durations, suggesting it reflects a fundamental property of the learning dynamics rather than insufficient capacity or data.

Implications for Linguistic Theory and AI

These findings carry significant implications for both linguistic typology and our understanding of language model capabilities:

Multiple Sources of Typological Universals: The dissociation suggests that different typological patterns may arise from distinct underlying mechanisms. Natural markedness direction may emerge from domain-general learning biases or information-theoretic principles that language models naturally capture. In contrast, object preference may reflect functional pressures specific to human communication (such as discourse management or processing efficiency) that are absent from pure distributional learning.

Limits of Distributional Learning: The results demonstrate that not all cross-linguistic regularities are equally learnable from distributional patterns alone. While models successfully acquire some typological preferences, others require additional inductive biases or learning signals beyond what standard language modeling provides.

Cognitive vs. Functional Explanations: The findings support theoretical frameworks that distinguish between cognitive constraints (which may be captured by neural networks) and functional pressures (which may require explicit communicative goals). Natural markedness direction may reflect cognitive processing preferences, while object preference may serve discourse-functional purposes.

Implications for Multilingual Models: Understanding which typological patterns emerge naturally from distributional learning versus which require explicit architectural biases can inform the design of more linguistically sophisticated multilingual language models.

Takeaways

1. Language models trained on synthetic corpora can acquire some but not all typological preferences observed in human languages, revealing selective alignment with linguistic universals.

2. GPT-2 models reliably learn natural markedness direction in DAM systems, preferring configurations where overt marking targets semantically atypical arguments—consistent with cross-linguistic patterns.

3. Models fail to reproduce the strong object preference in human DAM systems, showing no bias toward object-marking over subject-marking configurations despite identical training conditions.

4. The dissociation between markedness direction and object preference suggests that different typological universals may arise from fundamentally different sources—some from domain-general learning biases, others from human-specific functional pressures.

5. These findings highlight both the potential and limitations of using language models as cognitive models, demonstrating that distributional learning alone cannot account for all aspects of linguistic typology.