Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

Paper: 2602.05547 Authors: Shyam Sundhar Ramesh, Xiaotong Ji, Matthieu Zimmer, Sangwoong Yoon, Zhiyong Wang, Haitham Bou Ammar, Aurelien Lucchi, Ilija Bogunovic Categories: cs.CL, cs.AI, cs.LG

Abstract

A straightforward multi-task adaptation of GRPO often leads to imbalanced outcomes, with some tasks dominating optimization while others stagnate. MT-GRPO incorporates dynamic task weight adaptation targeting worst-case performance and a ratio-preserving sampler ensuring policy gradients align with adapted weights.

Key Contributions

• Dynamic task weight adaptation: Automatically adjusts task weights to target worst-case performance
• Ratio-preserving sampler: Ensures policy gradients align with the adapted weights
• Significant improvements: 16-28% absolute improvement on worst-task performance over standard GRPO

The Multi-Task Problem

When applying GRPO across multiple reasoning tasks simultaneously, a naive approach leads to task imbalance: some tasks dominate the optimization while others stagnate or regress. This happens because:

• Tasks have different difficulty levels and reward scales
• Easier tasks generate stronger gradient signals
• The model preferentially improves on tasks where it already performs well

MT-GRPO: Two Key Innovations

1. Dynamic Task Weight Adaptation

Instead of fixed task weights, MT-GRPO dynamically adjusts weights to focus on the worst-performing task. This minimax-style objective ensures no task is left behind.

2. Ratio-Preserving Sampler

A critical implementation detail: the sampler ensures that the actual ratio of training examples matches the dynamically adapted weights. Without this, the policy gradient estimates become biased, undermining the weight adaptation.

Results

• 16-28% absolute improvement on worst-task performance over standard GRPO
• 6% improvement over DAPO (another multi-task baseline)
• 50% fewer training steps to reach 50% worst-task accuracy in 3-task settings
• Competitive average accuracy maintained — improving worst-case doesn’t sacrifice overall performance

Why This Matters

1. Real-world relevance: Production LLMs need to reason across many task types, not just one
2. Fairness in capability: Ensures the model doesn’t sacrifice some reasoning abilities for others
3. Training efficiency: Faster convergence through better gradient utilization
4. Practical GRPO extension: Drop-in replacement for standard multi-task GRPO training