VIRAASAT: Advancing Cultural Reasoning in LLMs Through Multi-Hop Knowledge Graphs

Paper: 2602.18429 Authors: Harshul Raj Surana, Arijit Maji, Aryan Vats, Akash Ghosh, Sriparna Saha, Amit Sheth Categories: cs.CL, cs.IR

Abstract

While Large Language Models excel at mathematical and coding reasoning, they struggle with tasks requiring deep socio-cultural knowledge, particularly in non-Western contexts like Indian culture. This paper introduces VIRAASAT, a semi-automated framework for generating multi-hop cultural reasoning questions. Built on a knowledge graph of 700+ expert-curated cultural artifacts spanning 13 attributes across all Indian states and union territories, VIRAASAT produces over 3,200 questions requiring chained reasoning. The authors identify that standard Chain-of-Thought fine-tuning fails to ground low-probability cultural facts, and propose Symbolic Chain-of-Manipulation (SCoM), which trains models to simulate atomic knowledge graph operations internally. SCoM achieves up to 20% improvement over CoT baselines in supervised fine-tuning experiments.

Key Contributions

• VIRAASAT Dataset: A comprehensive multi-hop QA dataset with 3,200+ questions covering Indian cultural reasoning across 28 states and 8 union territories, spanning 13 cultural attributes including history, festivals, cuisine, and traditions
• Knowledge Graph Foundation: An expert-curated knowledge graph containing 700+ cultural artifacts that enables systematic generation of complex reasoning chains
• Symbolic Chain-of-Manipulation (SCoM): A novel training paradigm that teaches models to perform atomic knowledge graph manipulations internally, improving their ability to traverse topological structures
• Empirical Analysis: Comprehensive evaluation of SOTA LLMs revealing specific failure modes in cultural reasoning, particularly in grounding and synthesizing low-probability facts

The Cultural Reasoning Gap

Current LLM benchmarks predominantly focus on Western-centric knowledge and single-hop factual recall. The authors identify three critical limitations in existing cultural benchmarks: they are manually crafted (limiting scale), contain primarily single-hop questions (testing memorization rather than reasoning), and are prohibitively expensive to expand. This creates a significant blind spot in evaluating LLMs’ ability to reason about diverse cultural contexts.

VIRAASAT addresses this gap by focusing on multi-hop reasoning that requires models to chain together multiple cultural facts. For example, a question might require understanding the historical context of a festival, its regional variations, and its connection to specific cultural practices—demanding genuine reasoning rather than simple recall.

The semi-automated generation approach leverages the structured knowledge graph to systematically create questions that traverse different reasoning paths, ensuring coverage across geographical regions and cultural dimensions while maintaining quality through expert curation of the underlying artifacts.

Symbolic Chain-of-Manipulation Framework

The core innovation of SCoM lies in teaching models to internalize knowledge graph operations rather than generating free-form reasoning chains. Traditional Chain-of-Thought approaches struggle with cultural reasoning because they rely on the model’s pre-training distribution, which often assigns low probability to specific cultural facts.

SCoM adapts the Chain-of-Manipulation paradigm by training models to simulate atomic operations like:

• Node traversal: Moving from one cultural entity to related entities
• Attribute extraction: Retrieving specific properties of cultural artifacts
• Relation following: Navigating typed relationships in the knowledge graph
• Fact synthesis: Combining information from multiple graph paths

This symbolic approach provides several advantages. First, it grounds reasoning in the explicit structure of the knowledge graph, reducing hallucination. Second, it makes the reasoning process more interpretable by exposing the specific graph operations performed. Third, it enables the model to reliably handle low-probability facts by treating them as discrete graph elements rather than continuous probability distributions.

The training process involves creating supervision data where each reasoning step corresponds to a specific graph manipulation. During inference, the model learns to generate these symbolic operations, which can then be executed against the knowledge graph to produce reliable answers.

Experimental Results and Analysis

Evaluation on VIRAASAT reveals significant performance gaps in current SOTA LLMs. The authors test multiple model families and find that even the most capable models struggle with multi-hop cultural reasoning, often failing to maintain consistency across reasoning chains or incorrectly synthesizing cultural facts.

Standard CoT fine-tuning shows limited improvement, particularly on questions requiring three or more reasoning hops. The authors attribute this to CoT’s reliance on the model’s implicit knowledge representation, which poorly captures the long-tail distribution of cultural facts.

In contrast, SCoM demonstrates substantial gains:

• Up to 20% improvement in accuracy over CoT baselines
• Better performance on longer reasoning chains (3+ hops)
• Reduced hallucination rates when dealing with low-frequency cultural entities
• Improved consistency in multi-step reasoning

The results suggest that explicit symbolic manipulation provides a more robust foundation for cultural reasoning than implicit neural representations. The performance gap widens as question complexity increases, indicating that SCoM’s structured approach scales better to challenging reasoning scenarios.

Error analysis reveals that remaining failures often stem from ambiguities in cultural knowledge itself (where multiple interpretations exist) or gaps in the knowledge graph coverage, pointing to directions for future work.

Implications for Culturally-Aware AI

VIRAASAT represents a significant step toward building AI systems that can reason about diverse cultural contexts. The framework’s semi-automated approach offers a scalable path to creating cultural reasoning benchmarks for other regions and cultures, addressing a critical gap in current AI evaluation.

The success of SCoM suggests that hybrid approaches combining symbolic reasoning with neural learning may be essential for handling knowledge domains with long-tail distributions. This has implications beyond cultural reasoning, potentially benefiting specialized domains like medicine, law, or regional history where facts don’t follow the power-law distributions typical of general web text.

The release of the VIRAASAT dataset and findings provides the research community with tools to develop and evaluate culturally-aware models. This is particularly important as LLMs are increasingly deployed in diverse global contexts where cultural competence is essential for appropriate and effective interaction.

Future directions include expanding the knowledge graph to cover more cultural dimensions, developing methods to handle cultural knowledge that varies by perspective or interpretation, and exploring how symbolic reasoning approaches can be integrated with retrieval-augmented generation for even more robust cultural understanding.

Takeaways

1. Current LLMs exhibit significant deficiencies in multi-hop cultural reasoning, particularly for non-Western contexts like Indian culture
2. VIRAASAT provides a scalable, semi-automated framework for generating complex cultural reasoning benchmarks using expert-curated knowledge graphs
3. Symbolic Chain-of-Manipulation (SCoM) outperforms standard Chain-of-Thought approaches by up to 20% by teaching models to simulate explicit knowledge graph operations
4. The structured symbolic approach proves more effective than implicit neural representations for handling low-probability cultural facts
5. The framework and dataset establish a foundation for developing culturally-aware AI systems that can reason across diverse global contexts