Paper: 2603.24562 Authors: Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu, Shih-Lun Huang, Long Chen, Gabe Schulman, Huizhen Jin, Shengduo Li, Yixuan Wang, Huidi Yang Categories: cs.LG
The Gap
Large language models have shown that pretraining on massive text corpora creates powerful foundation models. Healthcare has structured electronic health records (EHRs) with millions of patient histories, but existing approaches either use representation learning (BERT-style models that need fine-tuning for every task) or naive next-token prediction (treating medical events like words, ignoring that diseases recur). The core problem: chronic conditions like diabetes appear repeatedly in a patient’s record, but current models don’t distinguish “first diagnosis” from “routine follow-up.” This inflates metrics artificially and fails to capture clinical reality.
Problem: EHR foundation models lack clinical awareness
|
v
Observation: Diseases recur, but models treat all events equally
|
v
Method: Generative pretraining with recurrence regularization
|
v
Evidence: Zero-shot prediction rivals fine-tuned models
|
v
Conclusion: Explicit recurrence modeling enables better generalization
The Increment
One sentence: Before RAVEN, EHR foundation models either required task-specific fine-tuning or treated disease recurrence as noise; after RAVEN, a single pretrained model can predict new disease onsets zero-shot by learning the difference between first occurrence and repeat visits.
Core Mechanism
RAVEN is an autoregressive transformer that generates the complete set of clinical events for a patient’s next hospital visit. Given a patient’s history (sequence of visits, each containing diagnosis codes, procedures, medications), the model predicts what will happen at visit N+1. The key innovation: during pretraining, RAVEN applies stronger regularization (higher loss weight) to events that already appeared in the patient’s history. This forces the model to learn two distinct patterns—how new conditions emerge versus how existing conditions persist.
The architecture tokenizes clinical codes (ICD diagnoses, procedure codes, drug prescriptions) and feeds them through a standard transformer decoder. But unlike language models that predict one token at a time, RAVEN predicts an entire visit’s worth of events simultaneously, then moves to the next visit. The recurrence-aware loss function penalizes the model more heavily when it incorrectly predicts a repeated event, making it “pay attention” to what’s chronic versus acute.
Patient History RAVEN Model Next Visit Prediction
[Visit 1: A,B,C] ---> +-------------+ ---> [Visit 4: B,C,E,F]
[Visit 2: B,C,D] ---> | Transformer | ^ ^ ^ ^
[Visit 3: B,C] ---> | + Recurrence| | | | |
| Regularizer | Repeat | New events
+-------------+ events | (lower loss)
|
Higher loss weight
Think of RAVEN like a hospital’s scheduling system that learns from millions of patient trajectories. When a diabetic patient books their next appointment, the system knows diabetes will likely appear again (recurrence) but also watches for new complications (onset). A naive system treats every diagnosis equally—it might predict “diabetes” because it saw it before, scoring points without learning anything. RAVEN’s regularization is like telling the scheduler: “Yes, we know diabetes repeats—that’s easy. Show me you can predict the new kidney issue that’s about to emerge.” The model must learn causal patterns (diabetes → kidney disease) rather than just memorizing what repeats.
Key Concepts
-
Recurrence-aware regularization: In clinical records, most events are repeats—a diabetic patient gets “diabetes” coded at every visit. Standard next-token prediction gives the model free points for guessing repeated codes, inflating accuracy without learning meaningful patterns. RAVEN applies higher loss weights to repeated events, forcing the model to work harder on them. This shifts the learning signal toward predicting *new onsets, which requires understanding disease progression rather than memorization. Imagine training a weather forecaster: if you reward them equally for predicting “sunny” in a desert (trivial) versus predicting the first rainstorm (hard), they’ll just always say “sunny.” Recurrence regularization is like saying “we know it’s usually sunny—impress us by catching the rain.”
-
Zero-shot disease prediction: After pretraining, RAVEN can predict whether a patient will develop a specific disease at their next visit without any task-specific training. You simply run the model forward, generate the predicted event set, and check if the target disease code appears. This works because the model learned general patterns of disease progression during pretraining. Contrast this with BERT-style models that learn representations but need a classification head and fine-tuning for each disease. It’s the difference between a doctor who studied general medicine (can diagnose many conditions) versus a specialist who needs retraining for each new disease.
-
Data-constrained, compute-saturated regime: The paper finds that simply scaling up model size (more parameters) doesn’t help much when you have limited training data. A 100M parameter model performs similarly to a 300M model on the same 1M patient dataset. This is the opposite of language modeling, where bigger is usually better. The insight: EHR data is sparse and structured, not dense like text. Each patient has maybe 10-20 visits, each with 5-10 codes. Scaling model capacity without more patients is like hiring more chefs when you only have ingredients for one meal—they just get in each other’s way.
Framework Shift
Before (representation learning): After (RAVEN):
Patient History Patient History
| |
v v
[Encoder] [Autoregressive
| Generator]
v |
Representation v
Vector Full Visit
| Prediction
v |
[Fine-tune for v
each disease] [Check for
| target code]
v |
Prediction v
Prediction
Key shift: From "learn embeddings, Key shift: Generate complete
then train classifiers" to "generate visits, recurrence-aware"
complete visits directly"
[One sentence: From learning compressed representations that need task-specific decoders to directly generating clinical futures with built-in recurrence awareness, the core shift is treating prediction as generation rather than classification.]
Expert Assessment
Problem choice: Real gap. EHR foundation models have lagged behind NLP because researchers blindly copied language modeling techniques without adapting to clinical data’s unique properties (sparse, structured, recurrence-heavy). The observation about inflated metrics from repeated events is sharp—it exposes a measurement problem that likely affects many published results.
Method maturity: The core idea (regularize repeated events) is simple and well-motivated. However, the paper doesn’t deeply explore *why this works better than alternatives. Could you achieve similar results by simply masking repeated events during training? Or by using a two-stage model (first predict recurrences, then predict new onsets)? The scaling analysis is valuable but limited—only three model sizes, one dataset size sweep.
Experimental integrity: Baselines are reasonable (BERT-style models, simulation-based approaches), but the zero-shot comparison is slightly unfair—RAVEN is generative and can naturally do zero-shot, while representation models need fine-tuning by design. A fairer comparison would include few-shot fine-tuning of RAVEN. The external validation (different hospital system) is strong and addresses a real concern about generalization. However, no ablation study isolates the recurrence regularization’s contribution—we don’t know how much performance comes from the regularization versus just using a generative approach.
Writing quality: The abstract and introduction are clear, but the method section buries the key innovation (recurrence regularization) in notation-heavy paragraphs. The scaling analysis deserves more prominence—it’s one of the paper’s strongest contributions but feels like an afterthought. The related work section is thorough but could better position this work relative to recent clinical LLMs (Med-PaLM, etc.).
Verdict: weak accept — Solid contribution with a clever insight about recurrence, but the experimental design doesn’t fully isolate what’s working, and the writing doesn’t highlight the most interesting findings.
Takeaways
Recurrence regularization as a general technique: If you’re working with any sequential data where events repeat (user behavior logs, sensor data, financial transactions), consider weighting your loss function to penalize repeated events more heavily. This forces models to learn transitions and causality rather than memorization. The implementation is trivial (just multiply loss by a weight based on whether the event appeared before), but the effect is significant.
Zero-shot evaluation for generative models: Instead of fine-tuning a classifier for each task, generate the full output and check if it contains what you’re looking for. This is underused outside of language modeling. If your domain has structured outputs (medical codes, product recommendations, molecular structures), generative pretraining might give you a universal predictor.
Scaling laws don’t transfer blindly: The finding that model size doesn’t help much with limited data is a warning for anyone applying LLM techniques to specialized domains. Before scaling up, check if you’re data-constrained. If you are, invest in data collection or augmentation before buying bigger GPUs.
论文: 2603.24562 作者: Haresh Rengaraj Rajamohan,