How Seemingly Inconsequential Design Choices Dictate LLM Performance in Pathology

Paper: 2606.12407 Authors: Kian R. Weihrauch, Thomas A. Buckley, William Lotter, Arjun K. Manrai Categories: cs.CV

The Gap

Prior work evaluated general-purpose LLMs on whole-slide images (WSIs) by feeding small, high-magnification patches (e.g., 256×256 at 20×) independently and aggregating via majority voting. Across benchmarks, these models consistently scored far below specialized pathology systems, leading the community to believe that architectural changes or extensive domain fine-tuning were required for any serious WSI task.

This paper steps back and asks: What if the input configuration – not the model – is the bottleneck? The authors systematically vary four factors (inference mode, patch size, magnification, patch count) in a full factorial design.

Problem: LLMs appear weak on WSI pathology (e.g., 15% on TCGA)
    |
    v
Assumption: gap is due to lack of domain-specific training
    |
    v
Method: factorial analysis of 4 input factors (inference_mode, patch_size, mag, count)
    |
    v
Evidence: optimal config gives GPT-5 39.5% on TCGA, 62.9% on GTEx
    |
    v
Conclusion: the assumed gap is largely an artifact of poor input choices

The Increment

One sentence: Before, the field thought LLMs needed special training for pathology; after this paper, we know that simply picking the right patch size, magnification, and inference strategy can close most of the gap.

Core Mechanism

The paper does not introduce a new model. Instead, it treats the input pipeline as a hyperparameter space and performs a grid search over four dimensions:

1. Inference mode – majority vote (each patch gets an independent prediction) vs. joint inference (all patches fed together as a single prompt/sequence).
2. Patch size – from 128×128 up to 1024×1024 pixels.
3. Magnification – from 2.5× to 40× (simulated by downsampling the original 40× WSI).
4. Patch count – the number of randomly sampled patches per WSI (e.g., 16, 32, 64, 128).

The data flow: a WSI → select N patches with given size and magnification → feed them to the LLM (either one-at-a-time with majority vote or all-at-once as a joint sequence) → obtain a classification.

[Data flow of factorial experiment]
+------------------+     +----------------------+     +-----------+
| WSI at 40x       | --> | Downsample to target  | --> | Sample N  |
| (whole slide)    |     | x0.5, x1, x2, ...    |     | patches   |
+------------------+     +----------------------+     +-----------+
                                                        |
                                                        v
                                     +----------------------------------+
                                     | Inference Mode                   |
                                     | [A] Majority vote: each patch    |
                                     |     -> LLM -> vote               |
                                     | [B] Joint: all patches           |
                                     |     -> LLM as single context     |
                                     +----------------------------------+
                                                        |
                                                        v
                                                   prediction

Structural metaphor: Think of the WSI as a *giant jigsaw puzzle painting that you want to classify (e.g., “is this a landscape or a portrait?”).

• Patch size = the size of the cardboard piece you look at. Tiny pieces (128×128) capture only a brushstroke; large pieces (1024×1024) reveal objects.
• Magnification = how close you hold the piece to your eyes. High magnification (40×) shows paint cracks but loses the big picture; low magnification (5×) shows the whole composition.
• Patch count = how many pieces you examine. Too few, you may miss the key feature; too many, you flood your brain (or the LLM’s context).
• Inference mode = your strategy for combining the pieces. *Majority vote means you look at each piece independently, shout “landscape!” or “portrait!”, and then count the shouts. Joint inference means you lay all the pieces on a table and look at them together, letting their spatial arrangement inform your decision.

The paper finds that for this task, it’s better to take a few large pieces at low magnification and look at them all together rather than many tiny close-up pieces voted independently. That’s the “balanced configuration” that jumps performance from 15% to 39%.

Key Concepts

• Factorial analysis: Instead of tweaking one knob at a time, the authors run experiments for all combinations of the four factors (a full factorial design). This reveals interactions – e.g., the best patch size depends on the magnification, and joint inference only helps when patches are large enough. Simple but often overlooked.
• Context window utilization: LLMs have a fixed context length (e.g., 128k tokens for GPT-5). Small, high-magnification patches waste tokens on redundant detail; large, low-magnification patches pack more semantic content per token, and joint inference allows the model to attend to relationships across patches.
• Domain transfer without tuning: The optimal configuration found on TCGA (cancer-type classification) transfers to GTEx (organ classification) and CPTAC (a fully held-out breast cancer cohort) without any parameter adjustment. This suggests the configuration benefit is a property of the input structure, not the label space.

Framework Shift

Here is the napkin sketch of the old practice vs. the new practice:

Before (mainstream approach):
+-----------+     +-------------+     +---------+
| Whole     | --> | tiny 256x256| --> | LLM     |
| Slide     |     | patches at 20x    | (per    |
| Image     |     | (e.g., 64 patches)| patch)  |
+-----------+     +-------------+     +---------+
                                          |
                                          v
                                     majority vote
                                          |
                                          v
                                      prediction

After (this paper's recommendation):
+-----------+     +-------------+     +---------+
| Whole     | --> | large 1024x  | --> | LLM     |
| Slide     |     | at 5x (e.g., |     | (joint) |
| Image     |     | 32 patches)  |     |         |
+-----------+     +-------------+     +---------+
                                          |
                                          v
                                     single prediction

From “many isolated close-ups → majority vote” to “few wide-angle views → joint reasoning,” the core shift is feeding the model more informative, less redundant patches while enabling it to combine information across them.

Expert Assessment

Problem choice: Real gap. Many prior LLM baselines in pathology were thrown together without thought to input design, and the community accepted the low scores as inherent. This paper surfaces a subtle but high-impact confound.

Method maturity: Clever but not deep. The experiment is a grid search – brute force by design. No new algorithm, no model change. The insight is in asking the right question, not in technical novelty.

Experimental integrity: Baselines are fair: they compare against the exact configs used in previous papers (small high-mag, majority vote). The factorial design is complete, and the hold-out cohort validates generalizability. Minor concern: “optimal” configs may be dataset-specific (though transfer results are encouraging). Also, the paper does not explore non-uniform patch sizes or adaptive sampling – possible improvements.

Writing quality: Clear and direct. The introduction could be shorter; the real meat is in the results tables. The “implications” section is the weakest – it ends too soon. A paragraph on practical guidelines for future WSI benchmarks would elevate the paper.

Verdict: Weak accept – Important empirical finding that every pathology-AI researcher should be aware of, but the method itself is simple factorial analysis.

Takeaways

• Always test input configuration before blaming the model. If you run an LLM baseline, try large patches (512–1024 at 5×–10×) and joint inference. This simple check can save months of “we need a specialized model” thinking.
• Majority voting is usually suboptimal for low-shot WSI classification. Joint inference allows the LLM to use its attention mechanism across patches, which matters more when you have few patches.
• Factorial analysis is underused in deep learning. When evaluating a pipeline, do a small grid over plausible input choices – the interaction effects can dwarf the effect of architecture changes.
• For transfer learning, the optimal input pipeline may be more robust than you expect. The config found on TCGA worked on GTEx and CPTAC without tuning; this suggests the pipeline shapes the input distribution in a dataset-agnostic way.