Clinical LLM Safety Doesn't Scale Like Accuracy: Evidence Quality Matters More Than Model Size

Paper: 2605.04039 Authors: Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn Categories: cs.CL, cs.AI, cs.LG

The Gap

The field assumes that scaling clinical LLMs (bigger models, longer context, fancier RAG) produces safer behavior because accuracy improves. But in medicine, average performance hides what matters: a single confident misdiagnosis can kill. Prior work measures accuracy on benchmarks but doesn’t track whether errors are high-risk, contradict evidence, or concentrate in dangerous patterns. The gap: we scale for accuracy without measuring whether safety scales the same way.

Problem: Clinical LLM deployment assumes accuracy = safety
   |
   v
Assumption: Bigger models / better RAG -> fewer errors -> safer
   |
   v
Reality check: What if high-risk errors don't follow accuracy curves?
   |
   v
Method: SaFE-Scale framework + RadSaFE-200 benchmark
         (measure safety separately across scale dimensions)
   |
   v
Evidence: Clean evidence cuts high-risk errors 78%
          RAG improves accuracy but keeps dangerous failures
   |
   v
Conclusion: Safety is a deployment property, not a scaling byproduct

The Increment

One sentence: Before this paper, we scaled clinical LLMs assuming safety followed accuracy; after, we know evidence quality and retrieval design shape safety independently, and dangerous errors concentrate in ways scaling doesn’t fix.

Core Mechanism

SaFE-Scale is a measurement framework, not a new model. It evaluates how safety metrics change across five dimensions: model size, context length, evidence quality (clean vs conflicted), retrieval strategy (no RAG, standard RAG, agentic RAG), and inference-time compute. The authors instantiate this with RadSaFE-200, a radiology benchmark where each question has clinician-labeled clean evidence, conflict evidence, and option-level tags for high-risk error, unsafe answer, and evidence contradiction.

The evaluation pipeline works like this: take 34 LLMs, run them in six conditions (zero-shot, clean evidence, conflict evidence, standard RAG, agentic RAG, max-context), and measure four safety metrics alongside accuracy. High-risk error tracks whether the model picks a clinically dangerous option. Evidence contradiction checks if the answer contradicts provided evidence. Dangerous overconfidence flags high-confidence wrong answers on high-risk questions. Worst-case analysis identifies questions where most models fail dangerously.

Input: Question + Evidence (clean / conflict / retrieved)
   |
   v
LLM (34 models x 6 conditions = 204 configurations)
   |
   v
Output: Answer + Confidence
   |
   v
Evaluation:
  - Accuracy (standard metric)
  - High-risk error (picked dangerous option?)
  - Contradiction (answer vs evidence mismatch?)
  - Dangerous overconfidence (confident + wrong + high-risk?)
  - Worst-case (questions where >50% of models fail badly?)

Think of this like crash-testing cars. Accuracy is average fuel efficiency across all roads. Safety metrics are: does the airbag deploy in head-on collisions (high-risk error), does the car ignore road signs (contradiction), does the speedometer show 60 when you’re doing 90 (dangerous overconfidence), and which intersections cause the most fatal accidents (worst-case). You can improve fuel efficiency without fixing the airbag. The paper shows clinical LLMs have the same problem: scaling improves the average but leaves the airbag broken.

Key Concepts

• High-risk error: Not all wrong answers matter equally. If a model misses a subtle finding, a radiologist might catch it. If the model confidently recommends discharging a patient with an aortic dissection, someone dies. High-risk error measures whether the model picked an option clinicians labeled as dangerous. This isn’t about being wrong—it’s about being wrong in a way that causes harm. Example: Question asks about chest pain + widened mediastinum. Correct answer: “aortic dissection, urgent imaging.” High-risk wrong answer: “musculoskeletal pain, discharge home.” Standard wrong answer: “pneumonia, antibiotics.” The first kills, the second just delays treatment.

• Evidence contradiction: In medicine, you don’t just answer questions—you justify answers with evidence. A model that says “no fracture” when the radiology report explicitly describes a fracture is contradicting evidence, even if “no fracture” happens to be correct for a different reason. This metric checks whether the model’s answer aligns with the provided evidence, independent of ground truth. It’s the difference between a student who gets the right answer by ignoring the textbook versus one who reads and reasons correctly.

• Dangerous overconfidence: Confidence should correlate with correctness. A model that’s 95% confident in a high-risk wrong answer is worse than one that’s 60% confident in the same mistake, because humans trust and act on high-confidence outputs. This metric isolates the intersection of three failures: wrong answer, high confidence (>80%), and high-risk question. It’s the “confidently wrong about something that kills people” failure mode.

Framework Shift

Before (mainstream approach):        After (this paper):

Scale model size                     Measure safety separately
     |                                    |
     v                                    v
Accuracy improves                    Accuracy improves
     |                                    |
     v                                    v
Assume safety improves               Check: high-risk errors?
                                          contradictions?
                                          overconfidence?
                                          worst-case clusters?
                                          |
                                          v
                                     Safety != accuracy
                                     (evidence quality matters more)

From “scale and hope” to “measure and design,” the core shift is treating safety as an independent deployment property shaped by evidence quality and retrieval design, not a passive consequence of model scale.

Expert Assessment

Problem choice: Real gap. Clinical AI deployment is happening now, and the field genuinely assumes bigger models are safer because benchmarks improve. The authors correctly identify that medicine has a different error distribution than general tasks—tail risk matters more than average performance. This isn’t manufactured; it’s an urgent question as hospitals adopt LLMs.

Method maturity: The framework is straightforward measurement, not algorithmic innovation. The insight is in *what to measure (safety metrics decoupled from accuracy) and how to construct the benchmark (clinician-labeled evidence and risk levels). The execution is solid: 34 models, 6 conditions, 204 configurations. The weakness is that RadSaFE-200 is small (200 questions) and radiology-specific. Generalization to other specialties is assumed, not proven. The agentic RAG implementation is underspecified—hard to reproduce.

Experimental integrity: Baselines are fair. The authors compare zero-shot, clean evidence, conflict evidence, standard RAG, agentic RAG, and max-context on the same questions. The clean evidence condition is the key control: it shows what’s possible when retrieval works perfectly. The numbers are credible: clean evidence produces a 20.6 percentage point accuracy gain and cuts high-risk error from 12.0% to 2.6%. The worst-case analysis (identifying questions where most models fail) is a smart addition. One concern: the paper doesn’t report inter-rater reliability for clinician labels. If two radiologists disagree on what’s “high-risk,” the metric is noisy.

Writing quality: The abstract and introduction are clear. The results section is dense with tables—Figure 2 (safety vs accuracy scatter plots) does more explanatory work than three pages of text. The discussion is where the paper weakens: it lists implications but doesn’t deeply explore *why RAG fails to reproduce clean evidence safety. Is it retrieval precision? Chunk boundaries? Model instruction-following? The authors hint at “evidence quality” but don’t operationalize it. Rewriting Section 4.3 to dissect RAG failure modes would elevate the paper from “important measurement” to “actionable diagnosis.”

Verdict: Strong accept — this is the right question at the right time, with credible evidence that challenges a widespread assumption. The framework will be cited, and the finding (safety doesn’t scale like accuracy) will shape how people deploy clinical LLMs.

Takeaways

For practitioners: Don’t assume your RAG system is safe because it’s accurate. Measure high-risk errors and evidence contradictions separately. If you can’t curate clean evidence, you’re not ready to deploy in clinical settings—agentic RAG won’t save you.

For researchers: The worst-case analysis (questions where most models fail) is a transferable technique. Apply it to your domain: find the questions where scaling doesn’t help, then study why. That’s where the next algorithmic insight lives.

For evaluators: Accuracy is not enough for high-stakes domains. Build benchmarks with option-level risk labels and evidence-contradiction checks. The SaFE-Scale framework (measure safety across scale dimensions) generalizes beyond medicine—try it for legal, financial, or safety-critical engineering tasks.

Concrete steal: The “clean evidence vs RAG” comparison is a diagnostic tool. If clean evidence fixes the problem but RAG doesn’t, your issue is retrieval quality, not model capacity. Stop scaling the model and fix your retrieval pipeline.