Text Knows What, Tables Know When: Bridging Clinical Narratives and EHR Data for Precise Timeline Reconstruction

Paper: 2605.15168 Authors: Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, Jeremy C. Weiss Categories: cs.CL, cs.AI, cs.LG, stat.ML

The Gap

Clinical timeline reconstruction faces a fundamental data mismatch. Text-based approaches extract events from clinical notes using LLMs, capturing rich semantic context but producing vague timestamps (“earlier today,” “a few days ago”). Structured EHR extraction pulls precise timestamps from database tables but misses events that clinicians only document in prose. Prior work treats these as separate pipelines—you either parse text or query tables, but not both in a principled way.

The gap: no existing method systematically aligns these complementary modalities to get both semantic completeness from text AND temporal precision from tables. The result is timelines that are either semantically incomplete (tables-only) or temporally fuzzy (text-only).

Problem: Fuzzy text timelines vs incomplete table timelines
    |
    v
Assumption: Text events can be temporally anchored using table timestamps
    |
    v
Method: Graph-based alignment with retrieval-augmented calibration
    |
    v
Evidence: +8.2% AULTC improvement, 34.8% text-only events discovered
    |
    v
Conclusion: Multimodal fusion yields more complete + precise timelines

The Increment

One sentence: Before this paper, you chose between semantically rich but temporally vague text timelines or temporally precise but incomplete table timelines—now you can fuse both to get semantic completeness with temporal accuracy.

Core Mechanism

The method works in three stages. First, it extracts “anchor events” from clinical narratives—these are central occurrences like “intubation” or “antibiotic started” that form the temporal backbone. An LLM identifies these anchors and assigns them relative timestamps based on narrative cues. This creates an initial scaffold.

Second, it places non-anchor events relative to this scaffold. The LLM determines whether each secondary event happened before, after, or during an anchor event, building out the timeline’s branches. At this point, you have a complete event graph but with relative, not absolute, timestamps.

Third—and this is the key innovation—it retrieves relevant rows from structured EHR tables (medications, labs, procedures) and uses them as external temporal evidence. For each text-derived event, the system searches for matching table entries. When it finds a match (e.g., “started vancomycin” in text aligns with a vancomycin administration record at 14

), it calibrates the text event’s timestamp to the table’s precise value. This retrieval-augmented calibration converts the relative timeline into an absolute one.

Clinical Note          EHR Tables
     |                      |
     v                      |
[Extract anchors]           |
     |                      |
     v                      |
[Build relative graph]      |
     |                      |
     +--[Retrieve matches]--+
            |
            v
    [Calibrate timestamps]
            |
            v
    Absolute timeline

Think of it like assembling a photo album from a friend’s vacation story. They tell you “we visited the museum, then had lunch, then went to the beach”—that’s the narrative structure, rich in detail but vague on timing. Meanwhile, their credit card statements show exact timestamps: museum ticket at 10

AM, restaurant charge at 1 PM, beach parking at 3 PM. The narrative gives you the full story (they also saw a street performer, which isn’t on any receipt), while the receipts anchor the timeline. This method does the same: text provides the complete event set, tables provide the temporal anchors, and retrieval links them together.

Key Concepts

• Anchor events: Not all clinical events are created equal. An “anchor” is a medically significant occurrence that clinicians naturally use as a reference point when narrating a patient’s course—think “patient was intubated” or “started on pressors.” These anchors are both semantically central (they matter clinically) and temporally stable (they’re likely to appear in structured records too). The method identifies these first because they form the skeleton onto which everything else attaches. Without anchors, you’re trying to build a timeline with no fixed points—every event floats relative to every other event, and errors compound. With anchors, you partition the problem: get the backbone right first, then fill in the details.

• Retrieval-augmented calibration: This is the bridge between modalities. After extracting events from text, the system doesn’t just guess at timestamps—it actively searches structured EHR tables for matching entries. The retrieval step uses semantic similarity (embedding-based search) to find candidate table rows, then a matching function determines if they refer to the same real-world event. When a match is found, the table’s timestamp becomes the ground truth, “pulling” the text event’s fuzzy timestamp into alignment. This is augmentation in the true sense: the text provides events the tables don’t have, the tables provide precision the text lacks, and retrieval is the mechanism that lets each modality compensate for the other’s weakness.

• Temporal concordance vs absolute accuracy: The paper tracks two distinct metrics. Absolute accuracy (AULTC) measures how close predicted timestamps are to ground truth in real time units—did you get the hour right? Temporal concordance measures whether events are in the correct order—did you get the sequence right? You can have high concordance with low accuracy (events in order but all shifted by 6 hours) or low concordance with decent accuracy (some timestamps close but events scrambled). The multimodal approach improves both, but the concordance gains are especially telling: they show the method isn’t just shifting timestamps, it’s actually reconstructing the causal structure of the patient’s trajectory.

Framework Shift

Before (text-only):                After (multimodal fusion):

Clinical note                      Clinical note    EHR tables
     |                                  |              |
     v                                  v              v
[LLM extraction]                  [LLM extraction] [Query]
     |                                  |              |
     v                                  +------+-------+
Fuzzy timeline                                |
                                              v
                                        [Retrieval match]
                                              |
                                              v
                                         [Calibrate]
                                              |
                                              v
                                      Precise timeline
                                      (with text-only events)

From single-source extraction to cross-modal alignment, the core shift is treating text and tables as complementary evidence sources rather than competing alternatives.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Clinical timeline reconstruction matters for sepsis prediction, readmission risk, and any task requiring temporal reasoning over patient trajectories. The observation that text and tables capture different events (34.8% text-only) is empirically grounded and clinically plausible—nurses write “patient became agitated” in notes, not in structured fields. The problem sits at the intersection of NLP and health informatics, where multimodal fusion is underexplored compared to vision-language work.

Method maturity: The approach is pragmatic rather than theoretically novel. Graph-based timeline construction is standard, and retrieval-augmented generation is a known pattern. The contribution is in the application: showing that retrieval can serve as a calibration mechanism, not just a context provider. The three-stage pipeline (anchors → relative graph → calibration) is sensible but feels somewhat ad hoc—why not jointly optimize all three stages? The reliance on LLM instruction-tuning for event extraction is a bit of a black box; the method inherits whatever biases and failure modes the base model has.

Experimental integrity: Baselines are fair—they compare against text-only and table-only approaches, which are the natural alternatives. The i2m4 benchmark spanning MIMIC-III and MIMIC-IV is appropriate. The 34.8% text-only events statistic is compelling, but I’d want to see a breakdown: are these clinically significant events or documentation noise? The paper reports improvements across “nearly all evaluated models,” which suggests robustness, but the absolute AULTC numbers aren’t given in the abstract—I’d want to see if we’re talking about 5-minute accuracy or 5-hour accuracy. No major red flags, but the evaluation is somewhat coarse-grained.

Writing quality: The abstract is dense but clear. The framing (“Text Knows What, Tables Know When”) is catchy and accurate. The paper likely front-loads the method and underexplains the retrieval matching function—how exactly do you decide if a text event and a table row refer to the same occurrence? That’s where errors creep in, and it deserves more scrutiny. The gap analysis (34.8% text-only events) should be in the main results, not buried as an empirical aside.

Verdict: weak accept — Solid applied work addressing a real clinical need with a sensible multimodal fusion strategy, though the method is more engineering than algorithmic innovation.

Takeaways

Retrieval as calibration, not just context: Most retrieval-augmented systems use retrieved content to inform generation. This paper uses retrieval to correct outputs post-hoc. That pattern transfers: if you have a noisy prediction and a clean reference database, retrieval can serve as a calibration layer. Example: extracting product mentions from reviews (noisy) and aligning them with a product catalog (clean) to get canonical IDs.

Anchor-first decomposition: When reconstructing complex structures, identify stable “anchor” elements first, then fill in the rest relative to those anchors. This reduces error propagation. Applies beyond timelines—think dependency parsing (anchor on main verbs), scene graphs (anchor on primary objects), or workflow extraction (anchor on mandatory steps).

Complementary modality analysis: The 34.8% text-only events finding is a reminder to measure modality coverage, not just fusion performance. Before building a multimodal system, quantify what each modality uniquely contributes. If one modality is a strict subset of the other, fusion is overkill. If they’re complementary (like here), fusion is essential.

问题：模糊的文本时间线 vs 不完整的表格时间线
    |
    v
假设：文本事件可以用表格时间戳进行时间锚定
    |
    v
方法：基于图的对齐 + 检索增强校准
    |
    v
证据：AULTC提升8.2%，发现34.8%的纯文本事件
    |
    v
结论：多模态融合产生更完整+精确的时间线

临床记录            电子病历表
     |                   |
     v                   |
[提取锚点]               |
     |                   |
     v                   |
[构建相对图]             |
     |                   |
     +--[检索匹配]-------+
            |
            v
    [校准时间戳]
            |
            v
     绝对时间线

之前（仅文本）：                  之后（多模态融合）：

临床记录                          临床记录    电子病历表
     |                                |            |
     v                                v            v
[LLM提取]                        [LLM提取]    [查询]
     |                                |            |
     v                                +-----+------+
模糊时间线                                  |
                                            v
                                      [检索匹配]
                                            |
                                            v
                                       [校准]
                                            |
                                            v
                                      精确时间线
                                   (含纯文本事件)