Evidence-Augmented Machine Learning for Self-Harm Detection in Emergency Department Triage Notes

Paper: 2606.02545 Authors: Liuliu Chen, Gowri Rajaram, Eleanor Bailey, Katrina Witt, Michelle Lamblin, Jo Robinson, Mike Conway, Vlada Rozova Categories: cs.CL

The Gap

Hospitals track self-harm through diagnostic codes (ICD-10), but these codes miss most cases—low sensitivity means surveillance is blind to the majority of self-harm presentations. Emergency Department (ED) triage notes are recorded at first contact and contain the clinical picture before formal diagnosis, but traditional NLP struggles with their brevity and variability. Prior work either relies on brittle rule-based systems that don’t transfer across sites, or trains site-specific models that can’t generalize.

The gap: how do you build a model that’s both accurate and transferable, working across hospitals without site-specific retraining?

Problem: Diagnostic codes miss most self-harm cases
   |
   v
Opportunity: Triage notes captured at first contact
   |
   v
Challenge: Notes are brief, variable, site-dependent
   |
   v
Prior approaches: Rules (brittle) or site-specific ML (no transfer)
   |
   v
This paper: Evidence-augmented 3-stage pipeline
   |
   v
Result: 88% AUPRC with cross-hospital transfer

The Increment

One sentence: Before this paper, detecting self-harm from triage notes meant choosing between accuracy (site-specific models) and transferability (rule-based systems)—now you can have both through evidence extraction as an intermediate representation.

Core Mechanism

The method has three stages. Stage 1 uses an LLM (GPT-4o) as a coarse filter to screen out obviously irrelevant notes—think of it as a high-recall, low-precision net that catches all potential self-harm cases plus some noise. Stage 2 extracts evidence spans: the LLM identifies and quotes the exact phrases in the note that indicate self-harm (e.g., “cut wrists”, “overdose paracetamol”). Stage 3 takes these evidence spans and feeds them into a traditional classifier (logistic regression with TF-IDF features) to make the final binary decision.

The key architectural choice is that the traditional classifier operates on extracted evidence, not raw notes. This creates a two-layer abstraction: the LLM handles linguistic variability and contextual understanding, while the classifier learns decision boundaries in a cleaner, more consistent feature space.

Raw triage note
    |
    v
[Stage 1: LLM Screen]  <-- High recall filter
    |
    v
Candidate notes
    |
    v
[Stage 2: Evidence Extraction]  <-- LLM quotes relevant spans
    |
    v
Evidence spans: "cut left wrist", "suicidal ideation"
    |
    v
[Stage 3: Traditional Classifier]  <-- Logistic regression on TF-IDF
    |
    v
Binary prediction + method classification

Think of it like a legal case: the LLM is a paralegal who reads through case files, highlights relevant passages, and hands you a memo with direct quotes. You (the classifier) then make the judgment based on those highlighted facts, not the entire case file. The paralegal adapts to different jurisdictions’ language (hospital sites), but the decision logic you apply to the facts stays consistent. This is why the model transfers: site-specific linguistic variation is absorbed by the LLM layer, while the downstream classifier learns portable patterns in the evidence itself.

Key Concepts

• Evidence extraction as intermediate representation: Instead of asking the model to go from raw text → decision in one step, decompose it into raw text → evidence → decision. Evidence acts as a bottleneck layer that forces the model to ground its reasoning in explicit textual spans. This has two benefits: (1) the LLM’s linguistic flexibility handles site-specific phrasing, while (2) the downstream classifier learns from a more normalized feature space (quoted evidence spans), making it less sensitive to superficial site differences. It’s the difference between training on “patient reports cutting behavior on left forearm with razor blade” vs training on the extracted span “cutting…left forearm…razor blade”—the latter strips away filler words and syntactic variation while preserving semantic content.

• Transferability through architectural separation: Traditional transfer learning asks “how do I retrain this model on new data with minimal updates?” This paper asks a different question: “how do I design the pipeline so most components don’t need retraining?” The LLM stages (1 and 2) use zero-shot prompting, so they adapt to new sites automatically. Only the final classifier sees training data, but it operates on a cleaner feature space (evidence spans). When you move to a new hospital, the LLM absorbs linguistic shifts, and the classifier’s learned patterns over evidence still apply. It’s like building a translation layer—local dialects get normalized before the decision logic sees them.

• AUPRC over AUROC for imbalanced tasks: Self-harm is rare (1-2% of ED visits), making this a heavily imbalanced classification problem. AUROC can be misleadingly high because it credits the model for correctly identifying the abundant negative class. AUPRC (Area Under Precision-Recall Curve) focuses on how well the model handles the minority class—precision asks “of the cases you flagged, how many are real?” and recall asks “of all real cases, how many did you catch?” For surveillance, both matter: false positives waste clinical resources, false negatives mean missed cases. AUPRC of 0.88 means the model maintains high precision and recall simultaneously across the decision threshold range.

Framework Shift

Before (mainstream approach):        After (this paper):

Raw text                             Raw text
   |                                    |
   v                                    v
[End-to-end classifier]              [LLM: Screen]
   |                                    |
Site-specific patterns                 v
   |                                 [LLM: Extract evidence]
   v                                    |
Prediction                               v
                                      Evidence spans (portable)
No transfer: retrain                     |
for each hospital                        v
                                      [Classifier: Decide]
                                         |
                                         v
                                      Prediction
                                      
                                      Transfers: LLM adapts,
                                      classifier stays fixed

From monolithic pattern matching to staged abstraction, the core shift is separating linguistic understanding (LLM) from decision logic (classifier) through an evidence bottleneck.

Expert Assessment

Problem choice: This is a real gap. Self-harm surveillance has genuine public health stakes, and the ICD-10 coding gap is well-documented (30-40% sensitivity in some studies). The choice to target triage notes is smart—they’re the earliest structured data point and more complete than discharge diagnoses. Not a manufactured problem.

Method maturity: The three-stage design is thoughtful, not just “throw an LLM at it.” Evidence extraction as an intermediate representation is the key insight—it’s what enables transfer. That said, the paper doesn’t deeply explore failure modes of the LLM stages. What happens when evidence extraction hallucinates or misquotes? The reliance on GPT-4o for stages 1-2 introduces a dependency on a closed model, which limits reproducibility and cost-scalability for real deployment. A stronger paper would ablate the LLM choice or show that cheaper models (e.g., fine-tuned smaller LLMs) can handle stages 1-2.

Experimental integrity: Baselines are fair—they compare against diagnostic codes (ICD-10) and prior NLP approaches. The three-hospital validation with prospective testing is rigorous. The numbers are credible: 88% AUPRC holds across sites, and the external validation doesn’t cherry-pick. One concern: they don’t report compute cost or inference time for the LLM stages. For a surveillance system running on every ED visit, cost and latency matter. Also, the paper mentions “95% accuracy” for method classification (cutting vs overdose vs other) but doesn’t break down how errors distribute—are certain methods systematically confused?

Writing quality: The method section is dense but clear. The results section could be tightened—Table 3 has redundant metrics, and the discussion restates findings without much critical reflection. The limitation section is short (half a page) and defensive rather than probing. A rewrite of Section 5 that honestly explores when the model fails, what biases might exist in triage note language, and how clinicians should interpret model outputs would elevate the paper significantly.

Verdict: Weak accept — solid contribution to a real problem with demonstrated transferability, but the reliance on proprietary LLMs and thin exploration of failure modes limit impact. Would be stronger with open model ablations and cost analysis.

Takeaways

Steal the evidence extraction bottleneck pattern: If you’re building classifiers on noisy, domain-variable text (legal documents across jurisdictions, medical notes across hospitals, support tickets across companies), consider decomposing into [LLM extracts structured evidence] → [traditional classifier decides]. The LLM absorbs linguistic chaos, the classifier learns portable patterns. This is especially useful when you can’t afford to retrain for every new deployment site.

For imbalanced tasks, report AUPRC not just AUROC: If you’re working on rare event detection (fraud, adverse events, system failures), AUROC will flatter your model by crediting easy negatives. AUPRC forces you to show that precision and recall both hold, which is what actually matters for operational deployment.

Use LLMs for normalization, not just generation: Most applied LLM work treats them as text generators. This paper uses them as normalizers—extracting and quoting evidence spans creates a cleaner feature space for downstream models. If you’re struggling with domain adaptation, ask: can an LLM stage reduce the surface area of variation before your main model sees the data?