When Patient Chatbots Leak Everything: A Security Autopsy of Medical RAG Systems

Paper: 2605.00796 Authors: Alfredo Madrid-García, Miguel Rujas Categories: cs.CR, cs.AI, cs.CL

The Gap

Medical RAG chatbots are being deployed to patients with promises of privacy and security. Prior work focuses on adversarial prompts (jailbreaking, prompt injection) or theoretical attack surfaces. What’s missing: empirical evidence of how badly real-world deployments fail basic security hygiene—not through sophisticated exploits, but through architectural mistakes visible in plain sight.

Problem: Patient-facing RAG chatbots deployed without security audit
    |
    v
Assumption: Sensitive config/data lives server-side, not client-side
    |
    v
Method: Two-stage assessment (LLM-assisted exploration + manual verification)
    |
    v
Evidence: System prompt, embeddings, 1000 patient conversations exposed via browser
    |
    v
Conclusion: Deployment failures are trivial to find; governance is broken

The Increment

One sentence: Before this paper, we worried about prompt injection attacks; after, we know some medical chatbots are shipping with their entire backend exposed in the browser’s network tab.

Core Mechanism

The assessment uses a two-stage process. Stage one: Claude Opus 4.6 acts as a hypothesis generator, exploring the chatbot through conversational probing to identify potential vulnerabilities. The LLM suggests what to look for based on common RAG architecture patterns. Stage two: manual verification using Chrome Developer Tools. The researcher inspects network traffic, API payloads, JavaScript objects, and browser storage—no authentication bypass, no exploit code, just reading what the browser already receives.

Stage 1: LLM-Assisted Exploration
    User <--prompt--> Claude Opus 4.6 <--query--> Target Chatbot
                           |
                           v
                   Vulnerability Hypotheses
                           |
                           v
Stage 2: Manual Verification
    Chrome DevTools --> Network Tab --> API Payloads
                    --> Application --> LocalStorage
                    --> Sources --> JS Config Objects
                           |
                           v
                   Confirmed Exposures:
                   - System prompt
                   - Model configs
                   - RAG parameters
                   - 1000 patient conversations

Think of this like a building inspector checking a hospital. Stage one is walking through with an experienced contractor who points out suspicious cracks and odd wiring. Stage two is pulling out the inspection tools—moisture meter, voltage tester—to confirm the walls are actually wet and the circuits are actually live. The contractor (LLM) accelerates hypothesis generation; the inspector (human with DevTools) confirms what’s real. The shocking finding: you don’t need to break through walls. The hospital left blueprints, patient records, and security codes taped to the front door.

Key Concepts

• Retrieval-Augmented Generation (RAG): Instead of relying solely on an LLM’s training data, RAG systems retrieve relevant documents from a knowledge base and inject them into the prompt. For a medical chatbot, this means fetching relevant medical literature or guidelines, then asking the LLM to answer based on those sources. The promise: grounded, verifiable answers. The risk: if the retrieval pipeline is exposed, attackers see what documents exist, how they’re chunked, and what metadata tags them.

• Client-side vs Server-side Architecture: In secure web apps, sensitive logic runs on the server; the browser only receives what the user should see. In this chatbot, the opposite happened: system prompts, embedding models, retrieval parameters, and conversation history were sent to the browser as JavaScript objects or API responses. It’s like a bank ATM that displays your PIN, account balance, and the vault combination on the screen instead of keeping them in the backend.

• LLM-Assisted Security Testing: The paper uses Claude to generate test prompts and hypothesize vulnerabilities. This is a double-edged sword: it lowers the barrier for auditors (good) but also for adversaries (bad). The LLM doesn’t execute exploits; it suggests what to look for. The human verifies using standard tools. The authors tested this under a “false developer persona”—the LLM helped even when told the user was building a similar system, not auditing one.

Framework Shift

Before (assumed secure deployment):        After (this paper's findings):

  Patient                                    Patient
    |                                          |
    v                                          v
  Browser (UI only)                          Browser (UI + full backend state)
    |                                          |
    | encrypted HTTPS                          | encrypted HTTPS (but exposes everything)
    v                                          v
  Server (system prompt, RAG config,         Server (just forwards data)
          embeddings, conversations)           |
    |                                          v
    v                                        Browser receives:
  Knowledge Base                             - System prompt
                                             - Model configs
                                             - RAG parameters
                                             - 1000 patient conversations
                                             - Knowledge base metadata

From “trust the HTTPS lock icon” to “HTTPS encrypts the pipe, not the payload”—the core shift is recognizing that encryption in transit means nothing if sensitive data is intentionally sent to an untrusted client.

Expert Assessment

Problem choice: This is a real gap. The AI safety community obsesses over prompt injection and model alignment, but deployment security gets less attention. The paper picks a high-stakes domain (patient-facing medical advice) where failures have immediate consequences. It’s not manufactured—the chatbot was live and publicly accessible.

Method maturity: The method is deliberately simple, which is the point. Using an LLM to accelerate hypothesis generation is clever but not novel; the contribution is demonstrating that trivial inspection reveals catastrophic failures. The two-stage approach (LLM + manual verification) is sound and reproducible. No sophisticated tooling required.

Experimental integrity: The paper is an anonymized case study, not a controlled experiment. The authors don’t name the chatbot (ethical choice), but this limits reproducibility. The findings are documented with screenshots and network traces, which is good. The “false developer persona” test is interesting but feels like a side note—it doesn’t change the main finding that the system was insecure by design.

Writing quality: The paper is clear and well-structured. The anonymization is appropriate but makes it hard to assess generalizability—is this one bad apple or a systemic problem? The discussion of governance lessons is strong, but the paper could have included a checklist or threat model for RAG deployments. The related work section is thin; more comparison with prior security audits of AI systems would strengthen the framing.

Verdict: weak accept — Important empirical evidence of a real problem, but limited to a single case study. The findings are alarming and the governance recommendations are sound, but the paper doesn’t establish how widespread these failures are.

Takeaways

If you’re deploying a RAG system, audit what your browser receives. Open DevTools, check the Network tab, inspect API responses and JavaScript objects. If you see system prompts, model configs, or user data, you have a problem. The paper’s implicit checklist: (1) system prompts and RAG configs should never leave the server, (2) conversation history should require authentication and be scoped per user, (3) knowledge base metadata should be sanitized before exposure, (4) assume adversaries have LLM assistance. For security researchers: LLMs are useful for generating test cases, but manual verification is non-negotiable. For governance: independent security review should be mandatory before deploying patient-facing AI, and “we use HTTPS” is not a security posture.