ClinEnv: Testing AI Doctors Through Real Hospital Admissions, Not Multiple Choice

Paper: 2606.02568 Authors: Yuxing Lu, Yushuhong Lin, Wenqi Shi, J. Ben Tamo, Xukai Zhao, Jinzhuo Wang, May Dongmei Wang Categories: cs.AI, cs.CL, cs.ET, cs.MA

The Gap

Medical AI benchmarks treat clinical reasoning like a standardized test: given a complete patient description, select the correct diagnosis from options A through E. But real medicine doesn’t work this way. A physician in a hospital doesn’t receive the full story upfront—they gather lab results piece by piece, order tests based on evolving hunches, and commit to treatments before knowing the outcome. Existing interactive medical benchmarks (DDXPlus, MIMIC-based setups) compromise on at least one critical dimension: they either pre-package all information, use simulated rather than real cases, or don’t track how models gather information—only what they decide.

This paper addresses the evaluation gap: we can’t measure whether AI doctors know how to work up a patient, only whether they guess correctly after being handed the answer key.

Problem: Static benchmarks    Assumption: Real clinical
don't capture sequential   ->  practice = incremental
decision-making under          information gathering +
uncertainty                    irreversible decisions

         |
         v

Method: Longitudinal          Evidence: 7 LLMs tested,
Inpatient Simulation       ->  best achieves 0.31 F1;
(ordered stages, active        outcome/process quality
queries, real EHR data)        sharply decoupled

         |
         v

Conclusion: Models can guess discharge diagnoses (0.51 F1)
but fail at management decisions (0.17 F1) and issue
redundant queries—hidden gap now measurable

The Increment

One sentence: Before this paper, we knew LLMs could answer medical questions; after it, we know they can’t reliably work up patients even when they guess diagnoses correctly.

Core Mechanism

ClinEnv constructs each benchmark case from a real inpatient admission by automatically parsing the EHR into an ordered sequence of decision stages—admission, days 1 through N, and discharge. At each stage, the model plays the attending physician: it must actively query four specialized agents (vital signs, labs, notes, procedures) to gather information, then commit to three types of decisions: medications to prescribe, procedures to order, and working diagnoses to document. The model can’t go back—decisions are irreversible, mimicking real hospital workflows.

Scoring happens on two axes. What the model decides is evaluated through deterministic ontology-grounded matching: medications, procedures, and diagnoses are mapped to standardized medical codes (RxNorm, CPT, ICD-10), so “aspirin 81mg” and “acetylsalicylic acid low-dose” both match. How the model gathers information is tracked by logging every query issued to the four agents. The benchmark counts redundant queries (asking for the same lab twice) and premature queries (requesting tomorrow’s data during today’s decision).

The environment surfaces 250 real MIMIC-IV inpatient admissions, each split into 3-7 decision stages. Models see only the information they actively request at each stage, not the full case file.

Stage t:
  Model receives: [summary of decisions made so far]
         |
         v
  Model queries:  Vital Agent  -->  returns BP, temp, O2 sat
                  Lab Agent    -->  returns WBC, creatinine
                  Note Agent   -->  returns nurse notes
                  Proc Agent   -->  returns imaging results
         |
         v
  Model commits:  Medications: [furosemide 40mg IV]
                  Procedures:  [chest X-ray]
                  Diagnoses:   [acute decompensated HF]
         |
         v
  [Decisions locked, advance to stage t+1]

Think of ClinEnv as a flight simulator for doctors. A pilot trainee doesn’t get a multiple-choice test about what to do if the left engine fails—they sit in a cockpit where the left engine actually fails, and they must scan the right instruments, in the right order, and execute the right sequence of actions before the plane hits the ground. Each decision affects the next stage; reading the altimeter after you’ve already crashed doesn’t count. ClinEnv does the same for clinical reasoning: the model must notice the rising creatinine in the labs (not handed to it automatically), connect it to the patient’s fluid status in the notes, and order the appropriate diuretic adjustment—all before progressing to the next day when the consequences of that choice play out.

Key Concepts

• Longitudinal Inpatient Simulation: Clinical practice unfolds over time, not in a single snapshot. A patient admitted for chest pain on Monday might develop complications by Wednesday, requiring the physician to revise their working diagnosis and treatment plan. Longitudinal Inpatient Simulation means each benchmark case is divided into chronological stages (admission, day 1, day 2, …, discharge), and the model must make decisions at every stage based only on information available up to that point. This mirrors how real attending physicians round on patients daily, adjusting management as new data arrives. The key shift: evaluation moves from “guess the final diagnosis” to “manage the patient correctly at every decision point along the way.”

• Active Information Gathering: In most medical benchmarks, the model receives a pre-written case summary containing all relevant facts. In ClinEnv, the model starts with minimal context and must explicitly query specialized agents to retrieve vitals, labs, notes, or procedure results. If the model doesn’t ask for the chest X-ray report, it doesn’t get to see it. This design choice exposes whether models know what information is diagnostically relevant. A model that issues 50 queries at every stage (asking for every possible lab and imaging study) versus a model that strategically requests targeted tests reveals very different clinical reasoning quality—even if both reach the same final diagnosis.

• Ontology-Grounded Matching: Medical terminology is messy. The same medication can be written as “aspirin,” “acetylsalicylic acid,” “ASA,” or “Ecotrin” in different EHRs. To score whether a model’s decision matches the ground truth, ClinEnv maps all medications, procedures, and diagnoses to standardized medical ontologies (RxNorm for drugs, CPT for procedures, ICD-10 for diagnoses). If the model prescribes “furosemide 40mg PO” and the ground truth is “Lasix 40mg oral,” the system recognizes these as equivalent because both map to the same RxNorm code. This prevents false negatives from superficial string mismatches while still requiring semantic correctness—“furosemide” won’t match “insulin” because they map to different codes.

Framework Shift

Before (static benchmarks):         After (ClinEnv):

  [Case Text]                         Day 0: [Admission Note]
       |                                   |
       v                                   v
  Model reads everything            Model queries: Labs? Vitals?
       |                                   |
       v                                   v
  Model outputs:                    Model decides: [Meds, Dx]
  [Single diagnosis]                      |
       |                              [Lock decisions]
       v                                   |
  Evaluation: correct/incorrect           v
                                     Day 1: [New context]
                                          |
                                          v
                                    Model queries again...
                                          |
                                          v
                                    [Repeat for N stages]
                                          |
                                          v
                                    Evaluation: F1 per stage
                                    + query efficiency score

From one-shot diagnosis to multi-stage management, the core shift is measuring clinical process, not just outcome.

Expert Assessment

Problem choice: This is a genuine gap. The medical AI community has known for years that static benchmarks don’t reflect clinical reality, but building interactive environments from real EHR data is non-trivial engineering work that most research groups avoid. The authors tackled the hard problem rather than publishing yet another dataset of curated clinical vignettes. The longitudinal framing is particularly sharp—it’s the natural extension of how doctors actually work, and it’s been conspicuously absent from benchmarks until now.

Method maturity: The automatic construction pipeline (parsing EHRs into decision stages) is clever but under-documented. The paper doesn’t explain how admission vs. day-1 vs. day-2 boundaries are determined—is it calendar days? Physician notes? Significant clinical events? This matters because arbitrary stage boundaries could artificially inflate or deflate difficulty. The four-agent architecture (vitals, labs, notes, procedures) is clean and modular, though it’s unclear why notes and procedures are separate agents when both come from the clinical narrative. The ontology-grounded matching is the right choice, but the paper doesn’t discuss how it handles edge cases (off-label drug uses, rare procedures without CPT codes).

Experimental integrity: The baselines are fair—seven models spanning GPT-4 to Llama variants, all prompted identically. The 0.31 F1 ceiling is sobering and credible; the authors don’t cherry-pick high numbers. The outcome/process decoupling (models guess discharge diagnoses at 0.51 F1 but management decisions at 0.17 F1) is the paper’s strongest empirical contribution, and the result is robust across models. However, there’s a missing ablation: how much does performance drop if models are given all information upfront versus requiring active queries? This would isolate whether the difficulty is in decision-making or information-seeking.

Writing quality: Section 3 (dataset construction) is dense and would benefit from a worked example showing one patient’s journey from raw EHR to structured stages. The results section (Section 5) frontloads tables without enough narrative interpretation—readers need help understanding why medication F1 is 0.17 while diagnosis F1 is 0.51. The related work section is thorough but could be trimmed; two paragraphs on interactive environments vs. static benchmarks would suffice. The discussion section is unusually self-aware about limitations, which elevates the paper.

Verdict: Weak accept — Addresses a real gap with a well-executed benchmark and sobering results, but the dataset construction methodology needs more transparency before the community can fully trust the stage boundaries and ontology mappings.

Takeaways

Steal the decoupled evaluation idea: score both *what your model outputs and how it gets there. If you’re building agents that interact with APIs, databases, or tools, track not just final answers but query efficiency, redundancy, and sequencing. ClinEnv’s finding—that models can guess outcomes while flailing at process—likely generalizes beyond medicine to any domain where information must be gathered incrementally (legal discovery, financial auditing, investigative journalism).

Steal the ontology-grounded matching pattern: if your domain has messy natural language that maps to structured taxonomies (product catalogs, legal citations, chemical compounds), use the taxonomy as ground truth rather than string matching. This prevents false negatives from paraphrasing while enforcing semantic correctness.

Avoid the single-shot evaluation trap: if your task involves sequential decisions, don’t collapse it into one prediction. ClinEnv’s multi-stage structure reveals failure modes (late-stage degradation, redundant queries) that vanish in aggregated metrics. If you’re evaluating agents, break tasks into checkpoints and score each one.