Unmasking the Factual-Conceptual Gap in Persian Language Models

Paper: 2602.17623 Authors: Alireza Sakhaeirad, Ali Ma’manpoosh, Arshia Hemmat Categories: cs.CL

Abstract

This paper introduces DivanBench, a diagnostic benchmark designed to evaluate Persian language models’ ability to reason about cultural norms beyond mere factual memorization. The benchmark focuses on superstitions and customs—arbitrary, context-dependent rules that cannot be derived through simple logical deduction. Testing seven Persian LLMs across 315 questions spanning three task types, the researchers uncover a critical gap: while models can retrieve cultural facts, they struggle to apply this knowledge in practical scenarios, exhibiting a 21% performance drop. The study reveals severe acquiescence bias where models correctly identify appropriate behaviors but fail to reject violations, and surprisingly, continuous Persian pretraining amplifies rather than mitigates these issues.

Key Contributions

• Introduction of DivanBench, a novel benchmark with 315 questions testing cultural reasoning through factual retrieval, paired scenario verification, and situational reasoning tasks
• Empirical evaluation of seven Persian LLMs revealing a systematic 21% performance gap between factual knowledge retrieval and scenario-based application
• Discovery of severe acquiescence bias in Persian models that accept both correct and incorrect cultural behaviors
• Evidence that continuous Persian pretraining can degrade reasoning capabilities rather than improve them
• Demonstration that cultural competence requires internalization of underlying schemas, not just pattern mimicry from scaled monolingual data

The Factual-Conceptual Divide

The core insight of this research lies in distinguishing between two types of cultural knowledge: factual recall and conceptual reasoning. Persian LLMs demonstrate proficiency in answering direct questions about cultural practices—for instance, knowing that breaking a mirror is considered unlucky in Persian culture. However, when presented with scenarios requiring application of this knowledge, performance deteriorates significantly.

This gap manifests most clearly in the benchmark’s three-tiered structure. Factual retrieval tasks ask straightforward questions about customs and superstitions. Paired scenario verification presents two situations and asks which violates cultural norms. Situational reasoning requires models to evaluate complex social contexts and determine appropriate behavior. The 21% performance degradation from the first to the third task type reveals that models lack the conceptual frameworks necessary to operationalize their factual knowledge.

The researchers argue this reflects a fundamental limitation in how models acquire cultural competence. Rather than developing internal representations of the social schemas that generate cultural rules, models learn surface-level associations between contexts and behaviors. When faced with novel situations requiring schema-based inference, these shallow representations prove insufficient.

Acquiescence Bias and Its Amplification

Perhaps the most troubling finding is the severe acquiescence bias exhibited by Persian LLMs. Models consistently agree with presented statements, correctly identifying appropriate cultural behaviors but failing to reject clear violations. This creates a paradoxical situation where a model might correctly state that gifting knives is considered bad luck, yet when presented with a scenario describing someone gifting knives as culturally appropriate, the model agrees.

This bias appears to stem from the models’ training objective of predicting plausible continuations. In Persian text corpora, statements about cultural practices are typically affirmative descriptions rather than negations or corrections. Models thus learn to generate agreement patterns without developing the critical reasoning needed to identify contradictions.

Counterintuitively, continuous pretraining on Persian data amplifies this bias rather than correcting it. The researchers hypothesize that additional exposure to affirmative cultural descriptions reinforces agreement patterns while providing insufficient examples of norm violations and their rejections. This suggests that simply scaling monolingual data—a common approach in multilingual NLP—may be inadequate or even counterproductive for developing robust cultural reasoning.

Implications for Multilingual AI Development

The findings challenge prevailing assumptions in multilingual NLP about how models acquire cultural competence. The research demonstrates that cultural understanding cannot be reduced to factual knowledge accumulation. Models require exposure to the reasoning processes that connect cultural facts to appropriate behaviors in diverse contexts.

This has significant implications for benchmark design. Existing Persian NLP evaluations focus on pragmatics and politeness but rarely test whether models can distinguish between memorized patterns and genuine understanding of underlying social schemas. DivanBench’s focus on superstitions and customs—arbitrary rules that resist logical derivation—provides a more stringent test of cultural reasoning.

The work also raises questions about evaluation methodology in low-resource languages. If continuous pretraining can degrade reasoning capabilities, researchers must carefully consider not just the quantity but the composition and structure of training data. Future work might explore curriculum learning approaches that explicitly teach models to identify norm violations, or contrastive learning methods that present both appropriate and inappropriate cultural behaviors.

Takeaways

1. Persian LLMs exhibit a 21% performance gap between retrieving cultural facts and applying them in scenarios, revealing that factual knowledge does not automatically translate to reasoning capability.

2. Severe acquiescence bias causes models to accept both correct behaviors and clear violations of cultural norms, suggesting they learn agreement patterns rather than critical evaluation skills.

3. Continuous Persian pretraining amplifies reasoning failures rather than improving them, challenging the assumption that more monolingual data inherently enhances cultural competence.

4. Cultural understanding requires internalization of underlying social schemas, not just pattern recognition from scaled datasets—models must learn the generative principles behind cultural rules.

5. DivanBench’s focus on arbitrary, context-dependent customs provides a more rigorous test of cultural reasoning than existing benchmarks that emphasize factual recall or surface-level pragmatics.