MediX-R1: Open-Ended Reinforcement Learning for Medical Multimodal Models

Paper: 2602.23363 Authors: Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed, Mohamed Zidan, Fahad Khan, Salman Khan, Rao Anwer, Hisham Cholakkal Categories: cs.CV

Abstract

MediX-R1 introduces an open-ended reinforcement learning framework designed specifically for medical multimodal large language models (MLLMs). Unlike traditional approaches that rely on multiple-choice question formats, this framework enables models to generate clinically grounded, free-form responses. The system employs Group-Based RL with a sophisticated composite reward mechanism that includes LLM-based accuracy judgments, medical embedding-based semantic rewards for capturing terminology variants, and format/modality rewards for interpretable reasoning. The framework also proposes a unified evaluation methodology using LLM-as-judge instead of brittle string-matching metrics. Despite training on only approximately 51,000 instruction examples, MediX-R1 demonstrates excellent performance across both text-only and multimodal medical benchmarks, significantly outperforming existing open-source baselines particularly on open-ended clinical tasks.

Key Contributions

• Introduction of an open-ended RL framework for medical MLLMs that moves beyond restrictive multiple-choice formats to enable free-form clinical reasoning
• Design of a composite reward system combining LLM-based accuracy assessment, medical embedding-based semantic rewards, and format/modality rewards for stable training
• Proposal of a unified evaluation framework using Reference-based LLM-as-judge methodology that captures semantic correctness and contextual alignment
• Achievement of strong performance across medical benchmarks using only $\sim51$K training examples, demonstrating data efficiency
• Release of trained models, curated datasets, and source code for reproducibility and community advancement

Technical Methodology

The MediX-R1 framework builds upon a vision-language backbone and applies Group-Based Reinforcement Learning with a carefully designed multi-signal reward structure. The composite reward function addresses the unique challenges of medical reasoning:

The LLM-based accuracy reward employs a strict YES/NO decision mechanism to judge semantic correctness of generated responses. This component ensures that the model’s outputs align with medically accurate information, providing clear binary feedback that stabilizes training.

The medical embedding-based semantic reward captures the nuanced nature of medical terminology, recognizing paraphrases and terminology variants that are semantically equivalent but lexically different. This is crucial in medical contexts where multiple valid ways exist to express the same clinical concept.

The format and modality rewards are lightweight components that enforce interpretable reasoning chains and proper modality recognition. These ensure that the model not only generates correct content but also presents it in a clinically useful format and appropriately processes multimodal inputs (text and images).

This multi-signal design provides stable, informative feedback for open-ended outputs where traditional verifiable rewards or MCQ-only approaches fall short. The Group-Based RL approach allows for efficient exploration and exploitation of the reward landscape while maintaining training stability.

Evaluation Framework and Results

A significant contribution of this work is the unified evaluation framework that addresses limitations of traditional metrics. Instead of relying on brittle string-overlap measures like BLEU or exact match, MediX-R1 employs a Reference-based LLM-as-judge approach. This evaluation methodology assesses:

• Semantic correctness: Whether the generated answer conveys medically accurate information
• Reasoning quality: The logical coherence and clinical grounding of explanations
• Contextual alignment: How well the response addresses the specific clinical question

The framework demonstrates effectiveness across both text-only medical LLM benchmarks and vision-language medical VLM benchmarks. Despite the relatively small training dataset of approximately 51,000 instruction examples, MediX-R1 achieves excellent results, outperforming strong open-source baselines. The gains are particularly pronounced on open-ended clinical tasks, where the model’s ability to generate free-form, contextually appropriate responses provides substantial advantages over multiple-choice constrained approaches.

The results validate that open-ended RL with comprehensive reward signals and LLM-based evaluation represents a practical and effective path toward reliable medical reasoning in multimodal models.

Implications for Medical AI

MediX-R1 addresses a critical gap in medical AI systems: the ability to provide nuanced, free-form clinical reasoning rather than being constrained to predefined answer choices. This capability is essential for real-world clinical applications where physicians need detailed explanations, differential diagnoses, and contextual reasoning.

The data efficiency demonstrated by the framework—achieving strong performance with only $\sim51$K examples—is particularly significant for medical domains where large-scale annotated datasets are expensive and difficult to obtain. The composite reward design provides a template for training medical AI systems that can handle the complexity and variability of clinical language while maintaining accuracy.

The open release of models, datasets, and code at https://medix.cvmbzuai.com facilitates further research and development in medical multimodal AI, potentially accelerating progress toward clinically deployable systems that can assist healthcare professionals with complex reasoning tasks.

Takeaways

1. MediX-R1 demonstrates that open-ended RL with composite rewards can effectively train medical multimodal models for free-form clinical reasoning, moving beyond restrictive multiple-choice formats.

2. The multi-signal reward design—combining LLM-based accuracy, medical embedding semantics, and format/modality rewards—provides stable training feedback for complex medical reasoning tasks.

3. LLM-as-judge evaluation frameworks offer a more robust alternative to traditional string-matching metrics for assessing open-ended medical responses, capturing semantic correctness and contextual alignment.

4. Remarkable data efficiency is achievable in medical AI: strong performance across benchmarks with only approximately 51,000 training examples demonstrates the effectiveness of the RL approach.

5. The framework’s particularly strong performance on open-ended clinical tasks validates the importance of enabling free-form responses for practical medical AI applications.