ClinFusion: A Vision-Centric Multimodal LLM for Holistic Medical Understanding

Paper: 2607.24743 Authors: Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han Categories: cs.CV, cs.AI, cs.CL

The Gap

Existing medical MLLMs have reached a frustrating boundary: they handle 2D images (X-rays, CT slices) and 3D volumes (MRI, full CT) through separate pipelines, forcing the field to maintain parallel architectures that never share knowledge.

Worse, the evaluation problem is equally broken. Models are typically assessed using generic NLP metrics like BLEU and ROUGE, which measure surface text similarity. A report can score well on these metrics while containing clinically incorrect findings or misattributing locations.

No prior system has tackled both problems together. Hulu-Med and Lingshu focus on model architecture but inherit standard evaluation. Meanwhile, evaluation papers propose better metrics but don’t reshape the underlying model design.

The logical path from gap to contribution:

[Problem]                  [Assumption]               [Method]
2D and 3D medical     -->  A unified encoder      -->  Cascade Spatial-Aware
images require              with smart fusion           Locality Fusion
separate pipelines          can bridge the gap          operator
        |                          |                          |
        v                          v                          v
[Evidence]                 [Conclusion]
20/24 benchmarks    -->   ClinFusion is SOTA
beaten vs open-source;    for unified medical
13/16 vs proprietary;     MLLM with
radiologists rank         clinically-aligned
reports highest           evaluation

[Three Disconnected Problems in Prior Work]

   2D Images              3D Images              Evaluation
  (X-rays, CT           (MRI volumes,          (BLEU, ROUGE
   slices)                full CT scans)         = text similarity)
      |                      |                        |
      v                      v                        v
  [Pipeline A]          [Pipeline B]           "Report reads OK
   = 2D encoder           = 3D encoder          on paper"
   + 2D processing        + 3D processing            |
      |                      |                       v
      v                      v                   [But is it
  [Partial               [Partial                clinically
   diagnosis]              diagnosis]              correct?]
      |                      |                       |
      v                      v                       v
  [Never shares          [Never shares          [NO GUARANTEE
   knowledge]              knowledge]              of accuracy]
                                                    |
   +------------------+-----------------------------+
                      |
                      v
            [[ THE GAP: Three disconnected
               problems, no unified solution ]]


=====================================================================

[ClinFusion's Unified Approach]

   2D Images              3D Images
  (X-rays, CT            (MRI volumes,
   slices)                 full CT scans)
      |                       |
      v                       v
  [2D Vision             [3D Vision
   Encoder]                Encoder]
      |                       |
      +---------- + ----------+
                  |
                  v
      [Cascade Spatial-Aware
       Locality Fusion]
       (unifies features,
        preserves 3D spatial info)
                  |
                  v
      [Unified Multimodal
       Embedding]
                  |
                  v
      [LLM generates
       medical report]
                  |
                  v
      [ROI-Grounded
       Evaluation]
       (checks: correct findings
        in correct anatomical
        regions, not just
        text similarity)
                  |
                  v
      [Validated by
       blinded radiologists]

The Increment

One sentence: Before this paper, 2D and 3D medical imaging lived in separate worlds with poor evaluation; after, there’s a single model that handles both and a way to check its work that actually matches clinical judgment.

Core Mechanism

ClinFusion’s architecture has two main stages: a compositional vision encoder and a language model head. The vision encoder is where the real work happens. It starts with two specialized encoders — one pre-trained on 2D medical images, another on 3D volumes — each producing feature representations at multiple spatial scales.

These multi-scale features then enter the Cascade Spatial-Aware Locality Fusion (CASALF) operator. “Cascade” means it happens in stages: features from the 2D encoder’s shallow layers meet 3D features early, then progressively fuse through deeper layers. “Spatial-Aware Locality” means the fusion respects spatial relationships — it doesn’t just average features globally but preserves where things are in the image. This is critical for 3D data, where collapsing to 2D would destroy depth information.

The fused features produce a single embedding that the LLM can consume, generating reports, answering questions, or following clinical instructions — all from a unified representation of whatever image type was provided.

[Internal Architecture: Data Flow]

Input Images
    |
    +---> [2D Vision Encoder] ---> 2D features at scales s1, s2, s3
    |                                     |
    |                                     v
    |                          [CASALF Stage 1]
    |                          (early fusion: shallow 2D + shallow 3D)
    |                                     |
    |                                     v
    +---> [3D Vision Encoder] ---> 3D features at scales s1, s2, s3
    |                                     |
    |                                     v
    |                          [CASALF Stage 2]
    |                          (mid fusion: refined 2D + refined 3D)
    |                                     |
    |                                     v
    |                          [CASALF Stage 3]
    |                          (deep fusion: final unified features)
    |                                     |
    |                                     v
    |                          [Unified Multimodal Embedding]
    |                                     |
    |                                     v
    +---------------------------------> [LLM]
                                           |
                                           v
                                    [Report / VQA / Instructions]

Think of it like a hospital’s radiology consultation rounds. Imagine a patient has both a chest X-ray (2D) and a lung CT scan (3D). In the old way, you’d have two separate radiologists — one reads only X-rays, the other only CTs — and they never talk to each other. Each gives a partial diagnosis. The final report is stitched together without coordination.

ClinFusion is like a modern tumor board. Both specialists present their findings in stages. At the first meeting (early cascade), the X-ray specialist points out a shadow, and the CT specialist says “I see that region in 3D — it’s actually a cyst, not a mass.” At the second meeting (mid cascade), they refine: “The cyst borders this vessel.” By the third meeting (deep cascade), they’ve converged on a unified understanding that neither could reach alone. The chief physician (LLM) writes the final report based on this synthesized, spatially-grounded understanding.

The spatial-awareness part is crucial: it’s like the specialists using anatomical coordinates (“left lower lobe, posterior segment”) rather than vague references (“somewhere in the chest”). Without it, you lose the precision that makes 3D imaging valuable.

Key Concepts

• Cascade Spatial-Aware Locality Fusion (CASALF): Imagine you’re assembling a 3D puzzle, but you also have the box cover (2D). Instead of finishing the puzzle and then comparing to the cover, you consult the cover at multiple stages — when you’re doing the edges, when you’re filling the middle, when you’re finishing the details. At each stage, the 2D reference helps you orient the 3D pieces correctly. CASALF works the same way: 2D features guide 3D feature refinement at multiple depths of the encoder, not just once at the end.

• ROI-Grounded Evaluation: Traditional metrics ask “does the generated text match the reference text?” This is like grading a student’s essay by checking if they used the same words as the answer key. ROI-grounded evaluation asks something different: “did the model identify the correct anatomical region and describe it correctly?” It’s like checking whether a geography student pointed to France on a map, not whether they spelled “France” the same way as the textbook. This matters enormously in medicine — a report that says “opacity in right lower lobe” when the finding is actually in the left lower lobe would score well on BLEU but could harm the patient.

• Vision-Grounded Evaluation Framework: This is the umbrella system ClinFusion introduces. It includes MedIF-Bench (a benchmark for testing whether models follow clinical instructions properly) and the ROI-grounded metric for report generation. The key insight is that evaluation should mirror how radiologists actually judge reports: looking at the image, reading the report, and checking whether the findings match what’s visible. ClinFusion validated this by having board-certified radiologists blindly rank reports and showing that the ROI-grounded metric correlates best with their judgments.

Framework Shift

Before (mainstream approach):         After (this paper):
                                      
  2D images    3D images               2D images    3D images
      |            |                       |            |
      v            v                       v            v
  [2D MLLM]   [3D MLLM]              [2D Enc.]    [3D Enc.]
  (separate)   (separate)                  |            |
      |            |                       +-----++-----+
      v            v                             ||
  [2D report]  [3D report]                 [CASALF Fusion]
      |            |                             |
      v            v                             v
  [BLEU/ROUGE  [BLEU/ROUGE               [Unified LLM]
   text-only]    text-only]                     |
                                               v
                                         [Report + VQA
                                          + Instructions]
                                               |
                                               v
                                         [ROI-Grounded
                                          Evaluation +
                                          Radiologist
                                          Validation]

From parallel pipelines with text-centric evaluation to a unified vision encoder with clinically-grounded assessment, the core shift is treating medical image understanding as a fusion problem rather than a scaling problem.

Expert Assessment

Problem choice: This is a genuine gap. The 2D/3D fragmentation in medical MLLMs is a real coordination problem, not manufactured. The evaluation critique is even stronger — using BLEU for medical reports has been criticized for years, but few systems have proposed and validated alternatives at this scale. The paper sits at the right moment: MLLMs are mature enough that the bottleneck is now domain-specific engineering and evaluation, not raw capability.

Method maturity: The cascade fusion approach is technically solid but not revolutionary. Multi-scale feature fusion with progressive refinement is well-established in computer vision (FPN, U-Net, etc.). The spatial-awareness mechanism adds genuine value for 3D data, but the authors don’t clearly explain what makes their cascade different from simply concatenating features at multiple scales. The real novelty is framing this as a unified 2D/3D problem and validating end-to-end.

Experimental integrity: The benchmark breadth is impressive (24 benchmarks for open-source comparisons, 16 for proprietary). The blinded radiologist evaluation is a strong signal of real-world validity. However, I’d want to see compute-normalized comparisons — is ClinFusion simply larger? The claims about outperforming GPT-5.2 and Gemini-3-Flash read like marketing copy and deserve scrutiny: which specific benchmarks, and are those fair comparisons? The paper should be more transparent about model size, training data, and computational cost.

Writing quality: The paper suffers from “SOTA chasing” language. Claims like “outperforming powerful proprietary models” without immediately qualifying the comparison dimensions feel promotional. The methods section could be clearer about the CASALF operator’s mathematical formulation and why it’s better than simpler alternatives. The evaluation framework section is the strongest writing — clear motivation, clean validation with radiologists.

Verdict: weak accept — The unified 2D/3D approach and ROI-grounded evaluation are genuine contributions that advance the field, but the paper oversells its novelty and needs more rigorous ablation of the cascade mechanism.

Takeaways

1. ROI-grounded evaluation is transferable: The idea of evaluating generated text by checking whether it correctly references structured regions (not just matching reference text) applies anywhere you generate descriptions of structured data. Think: automated code documentation that checks whether described functions match the actual code, or manufacturing reports that reference specific defect locations.

2. Cascade fusion for multi-modal/multi-resolution data: If you’re working with data that has natural 2D/3D structure (satellite imagery, manufacturing inspection, autonomous driving), the progressive fusion pattern is worth trying. The insight: don’t fuse once at the end; fuse repeatedly at multiple abstraction levels.

3. Domain-specific evaluation matters more than you think: The paper’s strongest contribution might be its evaluation framework, not its model. If you’re building AI for any expert domain, generic metrics will lie to you. Build evaluation that mirrors how experts actually judge quality, then validate that your metric correlates with expert judgment. It’s expensive but worth it.