VeriDx: A Correct Diagnosis Can Be Reached for the Wrong Reasons

Paper: 2609.14018 Authors: Zhong Cao, Shuying Chen Categories: cs.AI, cs.CL

The Gap

The paper opens with a sentence that states the whole problem: a correct diagnosis can still be reached for the wrong reasons. Getting the right answer and reasoning correctly are different achievements, and in clinical settings the difference is not academic — a right answer reached by luck carries no information about what happens on the next patient.

And the structure of the failure is specific to the domain. Every disease hypothesis creates obligations: key evidence must be checked, alternatives must be ruled out, contradictions must be resolved, useful tests must be considered, and closure must be justified. These are not stylistic preferences; they are what clinical reasoning consists of. A hypothesis that is entertained without its obligations being discharged has not been reasoned about — it has been asserted.

And existing evaluations miss exactly this. Current evaluations of medical LLMs mostly focus on final answers, local steps, or isolated facts. Each of the three looks past the obligations in a different way: final answers ignore the process, local steps do not track obligations accumulated across a trajectory, and isolated facts do not connect to any hypothesis at all. So the quantity that matters — whether commitments were honoured — is not measured in any of them.

   A CORRECT DIAGNOSIS CAN BE REACHED FOR THE WRONG REASONS
     GETTING THE RIGHT ANSWER and REASONING CORRECTLY are DIFFERENT
     ACHIEVEMENTS
       <- in CLINICAL settings the difference is NOT ACADEMIC: a right
          answer reached by LUCK carries NO INFORMATION about what
          happens ON THE NEXT PATIENT
        |
        v
   [AND THE STRUCTURE OF THE FAILURE IS SPECIFIC TO THE DOMAIN]
     EVERY DISEASE HYPOTHESIS CREATES OBLIGATIONS:
       key evidence MUST BE CHECKED
       alternatives MUST BE RULED OUT
       contradictions MUST BE RESOLVED
       useful tests MUST BE CONSIDERED
       closure MUST BE JUSTIFIED
       <- NOT stylistic preferences: this is WHAT CLINICAL REASONING
          **CONSISTS OF**
       -> a hypothesis entertained WITHOUT its obligations discharged has
          NOT BEEN REASONED ABOUT -- it has been ASSERTED

   [AND EXISTING EVALUATIONS MISS EXACTLY THIS]
     current evaluations of medical LLMs mostly focus on
       FINAL ANSWERS | LOCAL STEPS | ISOLATED FACTS
     <- each looks past the obligations in a DIFFERENT way:
        FINAL ANSWERS ignore the PROCESS
        LOCAL STEPS do NOT TRACK OBLIGATIONS ACCUMULATED ACROSS A
          TRAJECTORY
        ISOLATED FACTS do NOT CONNECT TO ANY HYPOTHESIS AT ALL
     -> the quantity that matters -- WHETHER COMMITMENTS WERE HONOURED --
        is NOT MEASURED in ANY of them

The Increment

One sentence: Before this paper, medical-LLM evaluation scored answers, steps or facts; after it, a disease-centric framework links free-form reasoning to structured disease profiles and shows many diagnostic errors are broken commitments made earlier, not isolated mistakes.

Core Mechanism

The design move is to give the reasoning something structured to be checked against: VeriDx links free-form diagnostic reasoning to structured disease profiles. The pairing is what makes obligations checkable. Free-form reasoning supplies the trajectory; the structured profile supplies what a hypothesis of that disease owes. Neither alone yields a verification: unstructured reasoning has nothing to compare to, and a profile alone says nothing about what the model actually did.

And the output is a three-way status per hypothesis, which is the right granularity: it tracks whether each hypothesis is satisfied, unresolved, or violated its clinical obligations. The distinction between the last two is the important one. Unresolved means an obligation was left open — the model never got to it. Violated means it was contradicted — the model did something inconsistent with its own hypothesis. Those have different remedies and different severities, and collapsing them into “wrong” would lose the distinction.

The taxonomy of what gets exposed is concrete, and each item is a different failure of commitment: missing critical tests, unresolved differentials, ignored contradictions, unsupported claims, and premature closure. Reading them together, they are all cases of a hypothesis-with-obligations where the obligations were not discharged — which is the paper’s thesis in list form. Premature closure is worth singling out because it is a recognised clinical failure mode with real consequences: settling on a diagnosis before the alternatives have been excluded.

The instantiation is appropriately narrow and demanding: complex respiratory diagnosis, using guideline-derived disease profiles and expert-annotated longitudinal cases. Three properties of that choice matter. Guideline-derived profiles give the obligations external authority rather than the authors’ opinion. Expert-annotated cases give ground truth for the annotation. And longitudinal cases are where commitments accumulated across a trajectory become visible — a single time point would not show a differential left unresolved over time.

And the headline finding is a reinterpretation, which is what makes the framework worth building: many diagnostic errors are not isolated mistakes, but broken commitments made earlier in the reasoning process. So the error is not located where the wrong answer appears. It was created earlier, when an obligation went undischarged, and the final wrong answer is a downstream consequence. That reframing changes what a fix would target — not the final decision, but the point in the trajectory where the commitment was dropped.

   [THE DESIGN MOVE: GIVE THE REASONING SOMETHING STRUCTURED TO BE
   CHECKED AGAINST]
     VERIDX LINKS FREE-FORM DIAGNOSTIC REASONING TO STRUCTURED DISEASE
     PROFILES
       <- the PAIRING is what makes OBLIGATIONS CHECKABLE:
          FREE-FORM REASONING supplies THE TRAJECTORY
          THE STRUCTURED PROFILE supplies WHAT A HYPOTHESIS OF THAT
            DISEASE **OWES**
       <- NEITHER ALONE YIELDS A VERIFICATION:
          UNSTRUCTURED REASONING has NOTHING TO COMPARE TO
          A PROFILE ALONE says NOTHING ABOUT WHAT THE MODEL ACTUALLY DID

   [AND THE OUTPUT IS A THREE-WAY STATUS PER HYPOTHESIS -- THE RIGHT
   GRANULARITY]
     it tracks whether each hypothesis is SATISFIED, UNRESOLVED, or
     VIOLATED ITS CLINICAL OBLIGATIONS
       <- the distinction between the LAST TWO is the IMPORTANT ONE:
          UNRESOLVED = an obligation LEFT OPEN -- the model NEVER GOT TO
            IT
          VIOLATED   = it was CONTRADICTED -- the model did something
            INCONSISTENT WITH ITS OWN HYPOTHESIS
       <- DIFFERENT REMEDIES and DIFFERENT SEVERITIES
       -> collapsing them into "WRONG" would LOSE THE DISTINCTION

   [THE TAXONOMY OF WHAT GETS EXPOSED IS CONCRETE -- each item a
   DIFFERENT FAILURE OF COMMITMENT]
     MISSING CRITICAL TESTS
     UNRESOLVED DIFFERENTIALS
     IGNORED CONTRADICTIONS
     UNSUPPORTED CLAIMS
     PREMATURE CLOSURE
     <- read together, ALL are cases of a hypothesis-with-obligations
        where the OBLIGATIONS WERE NOT DISCHARGED
        -> the paper's thesis IN LIST FORM
     <- PREMATURE CLOSURE deserves singling out: a RECOGNISED CLINICAL
        FAILURE MODE with REAL CONSEQUENCES -- settling on a diagnosis
        BEFORE the alternatives have been excluded

   [THE INSTANTIATION IS APPROPRIATELY NARROW AND DEMANDING]
     COMPLEX RESPIRATORY DIAGNOSIS
       GUIDELINE-DERIVED disease profiles
       EXPERT-ANNOTATED longitudinal cases
       <- GUIDELINE-DERIVED profiles give the obligations EXTERNAL
          AUTHORITY rather than THE AUTHORS' OPINION
       <- EXPERT-ANNOTATED cases give GROUND TRUTH for the ANNOTATION
       <- LONGITUDINAL cases are where COMMITMENTS ACCUMULATED ACROSS A
          TRAJECTORY become VISIBLE
          -> a SINGLE TIME POINT would NOT SHOW A DIFFERENTIAL LEFT
             UNRESOLVED OVER TIME

   [AND THE HEADLINE FINDING IS A REINTERPRETATION -- which is what makes
   the framework WORTH BUILDING]
     MANY DIAGNOSTIC ERRORS ARE NOT ISOLATED MISTAKES, BUT BROKEN
     COMMITMENTS MADE EARLIER IN THE REASONING PROCESS
       -> the ERROR IS NOT LOCATED WHERE THE WRONG ANSWER APPEARS
       -> it was CREATED EARLIER, when an OBLIGATION WENT UNDISCHARGED
       -> the FINAL WRONG ANSWER is a DOWNSTREAM CONSEQUENCE
       -> that REFRAMING CHANGES WHAT A FIX WOULD TARGET: NOT the final
          decision, but THE POINT IN THE TRAJECTORY WHERE THE COMMITMENT
          WAS DROPPED

Think of it as auditing a diagnosis the way you would audit a differential checklist rather than the discharge note. The note says the right thing, and reading it tells you nothing about how the decision was made. The checklist tells you what was owed: was the test ordered, was the alternative excluded, was the contradiction addressed, was the closure argued. Note that the checklist distinguishes “never got to it” from “contradicted yourself”, which matters because they are different problems — an omission to be filled versus an inconsistency to be resolved. And the reframing is the point: if most errors are items left unchecked earlier, then improving the final decision is treating a symptom, and the intervention belongs where the checklist was abandoned.

Key Concepts

• Obligations created by a hypothesis: evidence to check, alternatives to exclude, contradictions to resolve, tests to consider, closure to justify. They are what clinical reasoning consists of, so a hypothesis without them has been asserted rather than reasoned about.
• Why existing evaluations miss it: answers ignore process, steps miss accumulated obligations, facts connect to no hypothesis. Each looks past the commitment structure differently.
• Linking free-form reasoning to structured profiles: the pairing that makes obligations checkable. The trajectory supplies behaviour, the profile supplies what is owed.
• Satisfied, unresolved, violated: the three-way status, where the last two must stay separate. An omission and an inconsistency need different remedies.
• Broken commitments rather than isolated mistakes: the reinterpretation. The error is created earlier than the wrong answer appears, which changes where a fix belongs.
• Guideline-derived profiles and longitudinal cases: external authority for the obligations, and a time span long enough for accumulated commitments to be visible.

Framework Shift

Before (score answers, steps or facts):
  evaluate medical reasoning by final answer, local step, or fact recall
  -> a correct diagnosis reached by luck scores the same as one reasoned to
  -> obligations accumulated across a trajectory are not tracked
  -> errors appear where the wrong answer does

After (verify hypothesis obligations):
  free-form reasoning linked to structured disease profiles
  each hypothesis marked satisfied, unresolved, or violated
  -> missing tests, unresolved differentials, ignored contradictions,
     unsupported claims, premature closure
  -> on guideline-derived profiles and expert-annotated longitudinal cases
  -> many errors are broken commitments made earlier, not local mistakes

From asking whether the diagnosis came out right, to asking whether each hypothesis discharged the obligations it created, the core shift is that clinical correctness is a property of a trajectory’s commitments rather than of its final line.

Expert Assessment

Problem choice: Excellent, and the opening observation is the one that justifies the framework. “A correct diagnosis can still be reached for the wrong reasons” is exactly the failure a final-answer evaluation cannot see, and in a clinical setting the practical stakes of that blindness are high — a lucky right answer says nothing about the next patient.

Method maturity: The design’s logic is sound: obligations become checkable only when free-form reasoning is paired with something structured describing what is owed, which is why neither half suffices alone. The three-way status is the right granularity, and insisting on the unresolved-versus-violated distinction is what keeps the output diagnostic rather than merely evaluative. Deriving profiles from guidelines rather than from the authors’ judgement gives the obligations external authority, and using longitudinal cases is necessary for obligations that accumulate over time to be observable at all.

Experimental integrity: The headline finding is a reinterpretation rather than a score, and it is the more valuable form: locating errors earlier than they surface changes where an intervention belongs. The scope is narrow — one clinical domain, respiratory diagnosis — which is appropriate for a framework paper that needs guideline-derived profiles to exist. The limitation is that the verification depends on the quality of the structured profiles and the annotation, so disagreements about clinical obligations would propagate into the metric, and the paper’s use of guideline-derived profiles is the mitigation rather than a demonstration that obligation extraction is uncontroversial.

Writing quality: The obligation list is stated up front and every subsequent component maps onto it, which makes the framework easy to follow. Because the finding’s practical value is a change of target, a short example of one trajectory — the hypothesis, the obligation left unresolved, and the downstream error it produced — would make the reframing immediately concrete for a clinical reader.

Verdict: strong accept — it identifies the failure mode that answer-level evaluation cannot see, makes hypothesis obligations checkable by pairing free-form reasoning with structured profiles, and reports that errors are created earlier than they appear.

Takeaways

• Ask whether a right answer was reasoned to. A correct conclusion reached by luck carries no information about the next case.
• Track obligations, not just steps. Local step scoring misses commitments accumulated and abandoned across a trajectory.
• Separate omission from contradiction. “Never got to it” and “contradicted yourself” need different remedies, and one label loses the difference.
• Locate errors where they were created. If most failures are commitments broken earlier, fixing the final decision treats a symptom.