Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Paper: 2602.22207 Authors: Hanna Yukhymenko, Anton Alexandrov, Martin Vechev Categories: cs.CL, cs.AI, cs.LG

Abstract

This paper addresses a critical bottleneck in multilingual LLM evaluation: the poor quality of translated benchmarks. Current translated resources suffer from semantic drift and context loss, leading to unreliable performance metrics across languages. The authors present an automated framework that applies test-time compute scaling strategies—specifically Universal Self-Improvement (USI) and a novel multi-round ranking method called T-RANK—to generate high-quality benchmark translations. The framework was validated by translating popular benchmarks into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Both reference-based metrics and LLM-as-a-judge evaluations demonstrate that these translations significantly outperform existing resources, enabling more accurate multilingual model assessment.

Key Contributions

• Introduction of a fully automated translation pipeline that preserves benchmark integrity and linguistic nuances during localization
• Adaptation of test-time compute scaling strategies (USI and T-RANK) for translation quality improvement
• T-RANK: a novel multi-round ranking method that iteratively refines translation candidates
• Comprehensive translation of popular benchmarks into eight underrepresented European languages
• Empirical validation showing superior quality compared to existing translated resources using both automated metrics and LLM-based evaluation
• Open release of both the translation framework and improved benchmark datasets

Methodology: Test-Time Compute Scaling for Translation

The core innovation lies in applying test-time compute scaling to the translation task. Traditional translation pipelines rely on single-pass generation, which often fails to capture nuanced semantic relationships and task-specific requirements in benchmarks.

The framework employs two key strategies:

Universal Self-Improvement (USI): This approach generates multiple translation candidates and uses the model’s own judgment to select the best option. By leveraging the model’s understanding of both source and target languages, USI can identify translations that better preserve semantic content and task structure.

T-RANK (Translation Ranking): The authors’ proposed multi-round ranking method extends USI by iteratively refining the candidate pool. In each round, the system:

1. Generates $n$ translation candidates
2. Ranks them based on semantic fidelity, fluency, and task preservation
3. Uses top-ranked candidates to inform subsequent generation rounds
4. Converges on translations that optimize multiple quality dimensions simultaneously

This iterative refinement process allows the framework to escape local optima that single-pass translation often encounters, particularly for complex benchmark items with specialized terminology or intricate logical structures.

Results and Evaluation

The authors conducted extensive evaluation across two dimensions:

Reference-Based Metrics: When gold-standard human translations were available, the framework’s outputs showed higher BLEU, COMET, and chrF scores compared to existing translated benchmarks. The improvements were particularly pronounced for languages with limited training data (Estonian, Lithuanian) and for benchmarks requiring precise logical reasoning.

LLM-as-a-Judge Evaluation: Using GPT-4 and other frontier models as evaluators, the translated benchmarks demonstrated better preservation of:

• Original task difficulty and discriminative power
• Semantic equivalence across language pairs
• Cultural and linguistic appropriateness for target languages
• Consistency in multiple-choice options and answer keys

Downstream model evaluation using the improved benchmarks revealed more consistent performance patterns across languages, suggesting that the translations better capture the true multilingual capabilities of evaluated models rather than introducing translation-induced artifacts.

Implications for Multilingual AI Development

This work has significant implications for the broader multilingual AI ecosystem:

Democratization of Evaluation: By providing high-quality benchmarks for underrepresented European languages, the framework enables more equitable assessment of LLM capabilities beyond English and a few high-resource languages.

Scalability: The fully automated nature of the pipeline means it can be applied to new languages and benchmarks as they emerge, without requiring extensive human translation efforts.

Reproducibility: The open release of both framework and datasets establishes a reproducible standard for benchmark translation, addressing the current fragmentation in multilingual evaluation resources.

Quality-Compute Tradeoff: The test-time compute scaling approach demonstrates that investing additional computation during translation can yield substantial quality improvements, offering a practical alternative to expensive human translation for many use cases.

The framework also highlights a broader principle: techniques developed for improving model outputs (like self-consistency and iterative refinement) can be effectively repurposed for data curation and benchmark development.

Takeaways

1. Test-time compute scaling strategies like USI and T-RANK can dramatically improve automated translation quality for specialized content like AI benchmarks
2. The proposed framework successfully translates benchmarks into eight European languages while preserving task structure and semantic integrity
3. Automated translations from this pipeline outperform existing resources on both reference-based metrics and LLM-judge evaluations
4. High-quality multilingual benchmarks enable more accurate and equitable assessment of LLM capabilities across languages
5. The open-source release of framework and datasets provides a reproducible foundation for future multilingual AI evaluation efforts