SymptomAI: Why Structured Interviews Beat Chat for Medical Diagnosis

Paper: 2605.04012 Authors: Joseph Breda, Fadi Yousif, Beszel Hawkins, Marinela Cotoi, Miao Liu, Ray Luo, Po-Hsuan Cameron Chen, Mike Schaekermann, Samuel Schmidgall, Xin Liu Categories: cs.AI

The Gap

Language models perform well on curated medical vignettes—complex cases with rich context written for teaching. But real patients don’t arrive with a paragraph of symptoms. They say “my throat hurts” or “I’ve been tired lately.” Existing benchmarks (PubMedQA, MedQA, USMLE-style cases) test diagnostic reasoning on complete information. They don’t test the interview itself: what questions to ask, when to probe deeper, how to build a complete symptom picture from fragments.

The gap: we don’t know if LLMs can conduct the messy, iterative process of symptom gathering that precedes diagnosis in everyday healthcare. Prior work evaluates the destination (correct diagnosis given full context) but ignores the journey (getting that context through conversation).

Prior work:                    This paper:
                              
Rich vignette --> LLM --> Dx   Patient: "throat hurts"
                                    |
[Tests reasoning only]              v
                              LLM interview (iterative)
                                    |
                                    v
                              Complete symptom set --> Dx
                              
                              [Tests interview + reasoning]

The Increment

One sentence: Before this paper, we knew LLMs could diagnose given complete information; now we know they can gather that information through structured interviews better than humans do in the same setting.

Core Mechanism

SymptomAI deploys five conversational agents that differ in how they structure the interview. The baseline agent lets users guide the conversation—ask whatever, in whatever order. The agentic variants impose structure: they conduct a dedicated symptom interview before offering a diagnosis. This means asking about symptom onset, duration, severity, associated symptoms, and relevant medical history in a systematic way, regardless of what the user initially volunteers.

Each agent uses the same underlying language model but with different prompting strategies. After the interview, all agents produce a ranked list of differential diagnoses. The study randomized 13,917 Fitbit users to interact with these agents when reporting symptoms. A subset of 1,228 later provided clinician-confirmed diagnoses, creating ground truth. Independent clinicians also reviewed 517 conversations in a blinded comparison—they saw the dialogue transcript and made their own diagnosis without knowing what the AI suggested.

User reports symptom
        |
        v
    [Agent type]
        |
    +---+---+
    |       |
Baseline  Agentic
    |       |
    |   [Structured interview]
    |       |
    |   - Onset/duration?
    |   - Severity scale?
    |   - Associated symptoms?
    |   - Medical history?
    |       |
    +---+---+
        |
        v
   [LLM reasoning]
        |
        v
  Ranked DDx list

Think of it like the difference between a customer walking into a store and browsing versus a salesperson conducting a needs assessment. The baseline agent is the browser: the user picks what to mention, the AI responds to what’s offered. The agentic agent is the salesperson: it has a checklist of information it needs and systematically works through it. The user still provides all the information, but the agent controls the sequence and ensures nothing critical is skipped.

The key insight: medical interviews have known structure (review of systems, OPQRST for pain, etc.). Letting users guide the conversation means they might forget to mention the rash that started yesterday or the family history of autoimmune disease. A structured interview ensures completeness. The LLM’s job isn’t just to reason about symptoms—it’s to actively elicit them.

Key Concepts

• Differential Diagnosis (DDx): A ranked list of possible conditions that could explain a patient’s symptoms. Not a single answer but a probability distribution over diseases. For “fever + cough,” the DDx might be: (1) viral URI, (2) influenza, (3) COVID-19, (4) bacterial pneumonia. The goal is to rank the true condition highly, not necessarily first. In this study, accuracy means the true diagnosis appears in the top-k predictions, where k varies by analysis (top-1, top-3, top-5).

• Agentic Strategy: The AI takes initiative in structuring the conversation rather than passively responding. In a baseline chat, if the user says “I have a headache,” the AI might ask “how long?” and wait for the next user input. In an agentic strategy, the AI says “I’m going to ask you a series of questions to understand your headache better” and then systematically covers onset, location, quality, radiation, severity, timing, and exacerbating factors—even if the user didn’t think to mention them. The agent has a goal (complete symptom profile) and a plan (interview protocol) rather than reacting turn-by-turn.

• Blinded Randomized Comparison: Clinicians reviewed conversation transcripts without knowing which diagnosis came from the AI and which came from another clinician. They also provided their own independent diagnosis. This setup isolates diagnostic accuracy from other factors (bedside manner, trust in AI). If the AI’s diagnosis is more accurate than the human clinician’s diagnosis when both are working from the same transcript, it suggests the AI’s reasoning is stronger, not that it has better rapport or more time.

Framework Shift

Before (mainstream approach):        After (this paper):

User: "I feel sick"                  User: "I feel sick"
  |                                    |
  v                                    v
AI: "Tell me more"                   AI: [Interview protocol]
  |                                    |
User: [volunteers info]              AI: "When did it start?"
  |                                  User: "Two days ago"
  v                                    |
AI: [responds to info]               AI: "Rate severity 1-10?"
  |                                  User: "7"
  v                                    |
[Iterates until user                 AI: "Any fever?"
 stops volunteering]                 User: "Yes, 101F"
  |                                    |
  v                                    v
Diagnosis on incomplete data         [Continues until complete]
                                       |
                                       v
                                     Diagnosis on structured data

One sentence: From reactive chat that depends on user knowledge of what’s medically relevant, to proactive interview that systematically ensures completeness.

Expert Assessment

Problem choice: Real and underexplored. The medical AI literature is heavy on benchmark performance (USMLE, MedQA) and light on actual patient interaction. Deploying to 14,000 real users via Fitbit is a significant step toward ecological validity. The problem isn’t manufactured—clinicians do spend significant time on symptom gathering, and patients do underreport relevant information.

Method maturity: The core idea (structured prompting for interviews) is straightforward, almost obvious in hindsight. The contribution is empirical: showing it works at scale with real users and real diagnoses. The agentic strategies aren’t novel algorithms—they’re prompt engineering plus conversation management. That’s fine; sometimes the insight is “do the obvious thing systematically” rather than “invent a new technique.” The comparison to clinicians is valuable but limited by the blinded transcript setup, which removes clinical examination and follow-up.

Experimental integrity: The randomization is solid. The ground truth (self-reported clinician diagnosis) is weak but acknowledged. The blinded clinician comparison is well-designed. One concern: the 1,228 participants who provided diagnoses are a self-selected subset of the 13,917 who used the app. Were they sicker? More engaged? The auxiliary validation on a general US panel (1,509 conversations) helps but doesn’t fully address this. The wearable data analysis (500,000 days across 400 conditions) is exploratory and hypothesis-generating, not confirmatory—the authors are appropriately cautious here.

Writing quality: The abstract and introduction are clear. The methods section is dense and would benefit from a figure showing the study flow (recruitment → randomization → interaction → ground truth collection → clinician review). The results section jumps between the main study and auxiliary analyses without clear signposting. The discussion oversells the wearable findings—those are correlational and underpowered for rare conditions. The core message (agentic interviews improve accuracy) gets diluted by the breadth of analyses.

Verdict: weak accept — Solid empirical work on an underexplored problem, but the method is incremental (prompt engineering) and the ground truth is noisy. The scale and ecological validity elevate it above a typical benchmark paper.

Takeaways

For practitioners building medical AI: Don’t default to open-ended chat. If there’s a known structure to the information you need (and in medicine, there usually is), encode it in the conversation flow. Use the LLM to execute the protocol, not to invent one on the fly.

For conversational AI more broadly: User-guided conversations feel natural but often leave gaps. If your task has a completeness requirement (gathering all relevant info before making a decision), consider agentic strategies that take initiative. The tradeoff is user experience—some people will find structured interviews robotic—but for high-stakes decisions, completeness beats naturalness.

For evaluation: Benchmarks that provide complete context test reasoning but not information gathering. If your system will face incomplete or user-provided information in deployment, test it in that setting. The gap between “diagnose this vignette” and “interview this patient” is larger than it looks.

Specific technique: The paper doesn’t detail the prompts, but the pattern is clear: (1) explicit interview phase with a checklist, (2) transition signal (“Now that I have this information…”), (3) reasoning phase. This two-phase structure (gather, then decide) is transferable to other domains where decisions depend on systematically collecting information—insurance claims, technical support, legal intake.