Features With Footnotes: Auditable Heart-Failure Feature Engineering by Multi-Agent Pipeline

Paper: 2608.06366 Authors: Soorya Ram Shimgekar, Michelle Hu, Dorisa Shehi, Daniel Kang, Roy Ka-Wei Lee, Koustuv Saha, Christian Poellabauer, Christopher Lee, Sajeev Singh, Piyum Zonooz Categories: cs.AI, cs.LG

The Gap

The unglamorous truth of clinical ML is that the model is the easy part. Somewhere between 39% and 45% of a clinical data scientist’s time goes into feature engineering: figuring out which of nine-plus source tables holds the ejection fraction, deciding whether a loop-diuretic dose change counts as “escalation,” and encoding that decision as SQL nobody will be able to interpret in eighteen months. Heart failure makes this worse than average, because the clinically meaningful features are not columns — they are judgments defined in guideline documents (HFrEF vs HFpEF, NYHA class, congestion trajectory, GDMT optimization) that must be assembled from fragments scattered across labs, echo reports, medication orders, and encounters.

Two families of tooling exist, and both are partial. Rule-based extraction (hand-written SQL, phenotype libraries like eMERGE/OHDSI concept sets) is deterministic and auditable in principle, but it is written once, maintained never; when the guideline updates or the EHR schema shifts, the rules rot silently. LLM-based feature generation — prompt a model with a schema and ask for features — is fast and fluent, and produces exactly the failure mode you would expect: plausible feature definitions with no way to check whether the number attached to a patient actually came from that patient’s data, or from the model’s sense of what a heart-failure patient usually looks like. Neither line gives you the two things a clinical reviewer needs: *why does this feature exist (evidence link to a guideline) and where did this value come from (row-level provenance).

nMAS is the attempt to close that specific hole: keep LLM-scale coverage, but make every generated feature carry its rubric and its receipts, and add a verification layer that can mechanically reject features whose receipts do not resolve.

[Problem]  EHR feature engineering = 39-45% of DS workload,
           worst in HF where features are guideline judgments
              |
              v
[Prior art]  rule-based SQL ...... auditable but unmaintainable
             single-shot LLM ..... scalable but unverifiable
              |
              v
[Assumption] an HF feature can be decomposed into
             (a) a rubric derived from guideline text  +
             (b) a pointer set into source rows
             ... and (b) can be checked by machine
              |
              v
[Method]     nMAS: extraction agents -> rubric agents ->
             deterministic verifier -> restricted LLM auditor
              |
              v
[Evidence]   132 structured + 70 rubric-scored features
             AUROC HFrEF 0.895 -> 0.963
             AUROC HFpEF 0.870 -> 0.910
             independent LLM rubric: 81.5% of max
              |
              v
[Conclusion] auditable automated feature engineering is
             *feasible* -- single institution, dummy records,
             external validation still open

The Increment

One sentence: Before, an automated EHR feature was a number you had to trust; after, it is a number bundled with the guideline rule that defined it and the source rows that produced it, so a verifier can reject it without a human reading any code.

Core Mechanism

The pipeline runs in two generative stages plus two checking stages. The first generative stage is extraction: agents walk the nine source tables and emit 132 structured features — typed, single-concept fields (most recent LVEF, latest NT-proBNP, current beta-blocker, eGFR trend). The point of this layer is not cleverness; it is that each emitted value keeps a pointer back to the table, row, and timestamp it came from. That pointer is what makes everything downstream checkable.

The second generative stage is rubric-grounded aggregation: 70 features that are not extractions but scored judgments. Instead of asking a model “is this patient congested?”, the pipeline first has an agent write a rubric from guideline text — an ordinal scoring scheme with named criteria — and then applies that rubric to the structured features. The output is a score *plus the rubric version it was scored against plus the subset of structured features (and hence source rows) that triggered each criterion. This is the load-bearing design decision: the judgment is separated from the scoring rule, so the rule can be reviewed once and reused, rather than being re-invented inside every LLM call.

Then two checks. The deterministic verifier enforces three things that need no intelligence at all: structural integrity (types, ranges, cardinality), rubric compliance (did this score use the criteria the rubric declares?), and provenance resolution (does every cited row actually exist and support the claimed value?). Only what survives goes to the restricted LLM auditor, which reads feature-plus-evidence pairs and can flag or score but not write — a deliberate capability restriction, so the model that judges cannot quietly patch what it dislikes. Surviving features feed a standard supervised phenotyper, where the ablation (structured-only vs structured-plus-aggregated) produces the reported AUROC deltas.

  nine EHR source tables
  (labs, echo, meds, dx, encounters, vitals, orders, ...)
             |
             v
  +-------------------------------------+
  | EXTRACTION AGENTS                   |  --> 132 structured features
  |  table.column -> typed field        |      each = (value, table,
  |  attach (table, row_id, timestamp)  |               row_id, ts)
  +-------------------------------------+
             |
             v
  +-------------------------------------+     guideline text
  | RUBRIC AGENTS                       | <---- (HF staging, GDMT,
  |  step 1: write ordinal rubric       |        congestion, NYHA)
  |  step 2: score patient against it   |
  +-------------------------------------+  --> 70 aggregated features
             |                                each = (score, rubric_id,
             |                                        criteria_hit,
             v                                        evidence_rows)
  +-------------------------------------+
  | DETERMINISTIC VERIFIER              |
  |  . schema / type / range            |  reject ----> [ discard ]
  |  . rubric compliance                |
  |  . do evidence_rows resolve?        |
  +-------------------------------------+
             |
             v
  +-------------------------------------+
  | RESTRICTED LLM AUDITOR              |
  |  read-only: flag + score            |
  |  cannot rewrite features            |
  +-------------------------------------+
             |
             v
  feature table --> phenotyper --> AUROC (HFrEF / HFpEF)
  ablation: structured only  vs  structured + aggregated

The metaphor: this is a newsroom, not a writing assistant.

The extraction agents are reporters. They do not opine; they come back with quotes and, crucially, a note of who said it and when. A quote with no attribution is not filed.

The rubric agents are editors working from a style guide. The style guide here is the clinical guideline: it says what counts as “congestion,” in what gradations, using which named criteria. The editor’s job is a two-step: first codify the standard, then apply it. That separation is why the newsroom is consistent — the standard exists as an artifact you can pull off the shelf and argue about, rather than living in one editor’s head.

The deterministic verifier is the fact-checking desk. It is not smart and does not need to be. It calls every phone number in the footnotes. If a quote is attributed to a source that does not exist, the piece is killed — no judgment call required.

The restricted LLM auditor is the standards editor: reads the finished piece against the code of ethics, can write a memo, cannot touch the copy. Giving that role read-only access is not a technical accident; it is the whole reason its verdict means anything.

And the AUROC experiment? That is circulation numbers — evidence that the newsroom’s output is useful, not evidence that any individual story is true.

Key Concepts

• Row-level provenance: The difference between “this patient’s LVEF is 32%” and “this patient’s LVEF is 32%, from echo_reports row 88134, dated 2024-03-11, field lvef_pct.” The second version can be falsified by a script in milliseconds; the first requires a human with database access and an afternoon. Most LLM feature pipelines produce the first kind, which is why nobody deploys them in regulated settings. Provenance is not documentation — it is the thing that converts a claim into a testable claim.

• Rubric-grounded aggregation: Suppose you want a “diuretic escalation” feature. The naive approach asks a model per-patient and gets inconsistent answers, because the implicit definition drifts across calls. The rubric approach makes the definition an object first: 0 = no change; 1 = dose increase within class; 2 = added second agent; 3 = IV therapy, with criteria naming the specific order/med rows to look for. Then scoring is mechanical application. The gain is not accuracy per se — it is that disagreement moves from “the model is wrong” to “criterion 2 is too loose,” which is a fixable, versionable argument. Same trick as a grading rubric in a course: it does not make grading correct, it makes it consistent and contestable.

• Restricted auditor (LLM-as-judge with reduced capability): LLM-as-judge is usually weak evidence because the judge shares the generator’s blind spots. nMAS mitigates one axis of this by removing the judge’s ability to act — it can only flag. That kills a specific failure (the model quietly fixing what it should have reported) but does not fix the deeper one: an LLM scoring LLM-written rubrics at 81.5% of maximum is largely a coherence measurement, not a clinical validity measurement. Useful as a smoke test, not as a substitute for cardiologist review.

Framework Shift

Before (mainstream approach):        After (this paper):

  EHR tables                          EHR tables + guideline text
      |                                       |
      v                                       v
 [ analyst or LLM             ]        [ agents emit:              ]
 [ writes extraction logic    ]        [   value                   ]
      |                                [ + rubric_id / criteria    ]
      v                                [ + evidence row pointers   ]
   feature = number                            |
      |                                        v
      v                              [ deterministic verifier ]
 [ trust the pipeline ]              [   provenance resolves? ]
 [ (or re-derive it   ]                       |
 [  by hand)          ]                       v
      |                              [ read-only LLM auditor ]
      v                                       |
   model                                      v
                                     feature = auditable claim
                                              |
                                              v
                                           model

From feature-as-number to feature-as-claim-with-citations, the core shift is that verification stops requiring a human who understands the clinical domain and starts requiring only a script that can follow a pointer.

Expert Assessment

Problem choice: Real, and under-served relative to its economic weight. The field spends enormous effort on modeling architectures for clinical prediction and almost none on the 40% of labor that precedes them. The specific framing — that the bottleneck is not *generating features but trusting them — is the right diagnosis, and it lines up with where regulated clinical AI is heading (traceability requirements, model cards, provenance mandates). This is a sensible place to be working.

Method maturity: Solid engineering, modest novelty. Every ingredient exists elsewhere: multi-agent decomposition, rubric-based LLM scoring, provenance tracking from data-lineage tooling, capability-restricted judges from agent-safety work. The contribution is the assembly and the insistence that the deterministic verifier gate the LLM auditor rather than the reverse. That ordering is genuinely the right instinct — cheap mechanical checks first, expensive fuzzy checks on survivors. What is being overlooked is how far a much simpler system gets: a versioned rubric library written *once by clinicians plus templated SQL would deliver most of the auditability with none of the agent complexity. The paper does not tell us what the agents buy over that, and I suspect the honest answer is “coverage speed,” which is a real but smaller claim than the framing suggests.

Experimental integrity: This is where I would push hard, and there are three separate problems.

First, the cohort. The abstract says “500 dummy patient records” and then says evaluation was “limited to a single-institution cohort.” Those are not the same thing, and the difference matters enormously. If the records are synthetic, then the AUROC gain measures how well the pipeline recovers the synthetic generator’s structure — which is close to vacuous as clinical evidence. If they are real records from one site, then “dummy” is a misleading word choice. Either way, 500 patients is small for a two-phenotype study.

Second, leakage. HFrEF is *defined by reduced ejection fraction, typically LVEF below 40%. If any of the 70 aggregated features encodes EF, EF trajectory, or a guideline rubric that references EF thresholds — and it would be strange if none did — then the jump from 0.895 to 0.963 is partly definitional rather than predictive. The smaller HFpEF gain (0.870 to 0.910) is mildly reassuring, since HFpEF is the harder, more inference-dependent phenotype, but the paper needs an explicit accounting of which features touch the label definition. Without it, the headline number is uninterpretable.

Third, the missing baseline. The comparison is an ablation of nMAS against itself (structured-only vs full). There is no rule-based phenotyping baseline and no single-shot-LLM baseline, despite both being named as the prior art this work improves on. So the claim “better than existing approaches” is asserted, not measured. And the 81.5% rubric score is an LLM grading LLM output with no reported human expert agreement — that is a consistency metric wearing a validity metric’s clothes.

Writing quality: The provenance and verification design is clearly the intellectual core and, judging by the abstract’s proportions, gets less space than the results. The section that would elevate the paper most is a failure analysis of the verifier: how many candidate features were generated, how many were rejected, and for what reason (bad types, rubric violations, dangling provenance). Those rejection statistics are the actual evidence that the auditability machinery does work — right now we are told features were “verified,” which tells us the check ran, not that it caught anything. A single table of rejection rates by category would do more for credibility than the AUROC numbers do. Second priority: state plainly whether the data is synthetic, and stop using “dummy” and “cohort” interchangeably.

Verdict: borderline — the design instinct (mechanical verification gating fuzzy audit, features as claims with citations) is genuinely worth publishing, but with synthetic-or-unclear data, likely label leakage in the headline metric, no comparison against the baselines it critiques, and an LLM grading its own family’s work, the evidence currently supports “we built this” rather than “this is better.”

Takeaways

Things worth stealing regardless of whether you care about cardiology:

• Make the emitted artifact a claim, not a value. Any generated feature, label, or annotation should carry (value, rule applied, pointer to source). This one change converts review from “an expert reads the logic” to “a script follows the pointers,” which is a difference of two orders of magnitude in cost. Applies directly to LLM data labeling, document extraction, and compliance pipelines.

• Order your checks by cost, and let the cheap one gate the expensive one. Deterministic verifier first, LLM auditor only on survivors. Most agentic pipelines do the reverse or run them in parallel, wasting model calls on things a type check would have killed.

• Split the rule from the application of the rule. Have the model write a versioned rubric as a standalone artifact, then apply it mechanically. Disagreements become edits to a shared object instead of arguments about individual outputs — the same reason grading rubrics and coding manuals exist in human workflows.

• Restrict your judge’s capabilities on purpose. A read-only auditor that can flag but not fix produces a signal you can count. An auditor that can also edit destroys its own evidence trail.

• A caution to steal too: when you report gains from generated features on a phenotype defined by a threshold, publish the mapping between your features and the label definition. Otherwise your reviewers will assume leakage, and here they would probably be partly right.