Words Speak Louder Than Code: How Cognitive Heuristics Trick LLMs Into Missing Vulnerabilities

Paper: 2606.30587 Authors: Asif Shahriar, Hongyu Cai, Hadjer Benkraouda, Gang Wang, Z. Berkay Celik Categories: cs.CR, cs.AI

The Gap

Prior work established two separate facts, but never connected them. On one side, researchers like Zheng et al. and others demonstrated that LLMs can detect code vulnerabilities with decent accuracy — feeding code in, getting “vulnerable” or “safe” out. On the other side, work in cognitive science and NLP showed that LLMs inherit the same cognitive heuristics that bias human judgment: they’re swayed by irrelevant framing, authority cues, and anchoring information.

Nobody asked the natural follow-up question: when LLMs judge code for security vulnerabilities, does surrounding context — who wrote it, how the task is framed, what a previous analysis concluded — override what the code actually says?

This gap matters because vulnerability detection isn’t a trivia benchmark. It’s a security-critical task where a missed bug means an exploitable system. If context noise can flip verdicts, that’s not just an academic curiosity — it’s an attack surface.

Problem: LLMs do vuln detection.
         But are they actually reading the code?

   Assumption: "Context is neutral; model learns code semantics."
                      |
                      v
   Prior evidence: LLMs *do* exhibit cognitive biases in NLP tasks
                   But nobody tested this in code security settings
                      |
                      v
   Method: Hold code FIXED. Vary ONLY surrounding context.
           Three levers: author halo, task framing, prior verdict
                      |
                      v
   Evidence: 8 LLMs, 3 languages, ~1000 controlled pairs.
             All models shift verdicts based on context.
             Semantic-hard bugs are most affected.
                      |
                      v
   Conclusion: Cognitive susceptibility is an EXPLOITABLE property.
               Black-box attack suppresses 97% of detections.

The Increment

One sentence: Before this paper, we evaluated LLM vulnerability detectors by asking “can they find bugs?” — after this paper, we must ask “can someone make them forget what they found?”

Core Mechanism

The experimental design is elegantly simple: build a controlled test where the code never changes but everything around it does. The authors construct triplets — same code snippet, three different contextual wrappers — and observe whether the LLM’s verdict flips.

Three cognitive levers are pulled independently. The halo effect is triggered by author attribution: prepending lines like ”// Author: Senior security engineer at Google” or ”// Author: CS freshman project” before the code. The framing effect is triggered by rephrasing the task objective: one version says “find vulnerabilities to fix them,” another says “evaluate whether this code is production-ready,” a third says “this code was flagged by users as suspicious.” The anchoring effect is triggered by injecting a prior analysis result: “a static analyzer found 0 vulnerabilities” versus “a static analyzer found 3 vulnerabilities.”

Each lever is pulled in isolation and in combination. The authors test eight LLMs (including GPT-4, Claude, Gemini, Llama, and CodeLlama variants) across C, Python, and Java. They measure verdict flip rates — how often a model changes from “vulnerable” to “safe” or vice versa — and perform code-level analysis to understand which vulnerability types are most affected.

[Same Code Snippet]
       |
       +---> [Baseline: no context]        ---> Verdict A
       |
       +---> [Halo wrapper: author info]    ---> Verdict B  (compare to A)
       |
       +---> [Framing wrapper: task goal]   ---> Verdict C  (compare to A)
       |
       +---> [Anchor wrapper: prior result] ---> Verdict D  (compare to A)
       |
       +---> [Combined wrappers]            ---> Verdict E  (compare to A)

   Data flow:
   ---------

   Code DB  --+--> Context Generator (3 lever types)
              |         |
              |         v
              |    [Controlled Pairs: same code, diff context]
              |         |
              |         v
              |    [8 LLMs x 3 Languages]
              |         |
              |         v
              +---> Verdict Comparator ---> Flip Rates
                                         ---> Code-level Analysis
                                         ---> Attack PoC

Think of it like a restaurant critic’s blind taste test — except someone keeps slipping notes under the door. The food on the plate (the code) never changes. But between tastings, the critic receives a note: “This dish was prepared by a Michelin-starred chef” (halo), or “Your editor wants a harsh review to generate clicks” (framing), or “The previous critic gave it 5 stars” (anchoring). The question is: does the critic’s written review change? The paper’s answer is yes, every time, and sometimes wildly — the critic might call the same dish “exquisite” one moment and “inedible” the next.

The metaphor holds structurally: the code is the food, the context wrappers are the notes, the LLM is the critic, and the verdict flip is the changed review. Crucially, the authors also check whether the critic, when it does change its verdict, points to the right reason (actually identifies the real vulnerability) or just echoes the note. Most of the time, it just echoes the note. That’s the scary part.

Key Concepts

• Cognitive Heuristics: These are mental shortcuts your brain uses to make quick judgments, often at the cost of accuracy. The halo effect is when you assume someone is good at everything because they’re good at one thing — like assuming a Google engineer writes secure code. The framing effect is when the way a question is asked changes your answer — “is this code safe to deploy?” versus “does this code have bugs?” feels different even though they’re nearly identical. The anchoring effect is when an initial piece of information disproportionately influences your judgment — if someone says “the static analyzer found 0 bugs,” you’re more likely to agree it’s safe, even if the static analyzer is irrelevant to what you’re evaluating. In this paper, LLMs exhibit all three, without any code changing.

• Verdict Flip Rate: This is the paper’s core metric. Take a code snippet, get a verdict from the LLM. Now change only the context wrapper, get a new verdict. Did it flip from “vulnerable” to “safe” or vice versa? The flip rate is the percentage of times this happens across all tested pairs. A 33% framing susceptibility means that one in three times you reframe the task, the model changes its mind about whether the code is vulnerable — regardless of whether the code actually changed. It’s a direct measure of how much the model is listening to you instead of reading the code.

Framework Shift

Before (mainstream approach):          After (this paper):

  Code ---> [LLM] ---> Vuln/Safe       Code + Context ---> [LLM] ---> Vuln/Safe
                                                      |
  Eval metric: accuracy                             Context is NOT neutral.
  Assumption: context is                             Context is an attack surface.
              neutral noise.                         Eval must include context
                                                    robustness testing.

  Threat model:                   -->    Threat model:
  Adversary modifies code                Adversary modifies only context
                                         (names, comments, framing)

From “evaluate the model on code alone” to “evaluate the model on code plus whatever noise surrounds it,” the core shift is that context itself becomes a first-class attack vector in vulnerability detection.

Expert Assessment

Problem choice: This is a genuine and timely gap. The security community has been racing to deploy LLM-based code analysis tools, but the evaluation methodology has been frozen on accuracy benchmarks that assume context-neutral inputs. Real-world codebases are full of context noise — comments, commit messages, author metadata, CI/CD output — and if models are swayed by this noise, that’s a real blind spot. The paper sits at the intersection of adversarial ML and software security in a way that neither community has fully explored.

Method maturity: The controlled-pair design is clean and well-motivated — it’s not brute force, it’s a principled ablation of context effects. The three heuristics are well-chosen as representatives. That said, the framework is somewhat narrow in scope: the halo, framing, and anchoring manipulations are a curated subset of possible context perturbations. The authors acknowledge this but don’t deeply explore what other contextual features (e.g., code comments inside the function, repository popularity, dependency graphs) might trigger additional biases. The proof-of-concept attack is compelling but somewhat expected — if context can flip verdicts, an attacker can obviously exploit that. The 97% suppression number is eye-catching but deserves scrutiny: it’s on a specific model (GPT-4) with aggressive prompting, and generalization across models would strengthen the claim.

Experimental integrity: The baselines are fair — they compare context-manipulated runs against clean baselines using the same models and code. The statistical approach (flip rates across controlled pairs) is sound. One concern: the paper uses ~1000 code snippets, which is reasonable but not huge, and the vulnerability ground truth comes from existing benchmarks (D2A, Big-Vul, etc.) that have their own noise. The cross-language comparison (C, Python, Java) adds breadth, but the depth of analysis varies — C gets the most attention. The authors do report confidence intervals and ablate individual levers, which is good practice.

Writing quality: The paper is well-structured and readable. The code-level analysis section (Section 5.4) is where it shines — the breakdown of semantic-hard vs. pattern-easy vulnerabilities is the most novel and actionable finding. The related work section could be tighter; it spends too much time cataloging LLM vulnerability detection papers and not enough framing the cognitive psychology angle. The threat model discussion feels somewhat superficial — the PoC attack is presented as a demo rather than a fully developed threat scenario with real-world plausibility analysis.

Verdict: weak accept — The core insight is important and the evidence is solid, but the framework’s narrow scope and the somewhat surface-level attack analysis prevent it from being a definitive work. It opens a door more than it walks through it.

Takeaways

Three things a practitioner can steal right now:

1. Strip context before evaluating LLM-based code review tools. If you’re benchmarking or deploying an LLM for vulnerability detection, test it with and without comments, author metadata, and task descriptions. If verdicts shift significantly, your tool is listening to noise, not code. This is a concrete evaluation checklist item.

2. Semantic-hard vulnerabilities need different detection strategies than pattern-matching ones. The paper shows that simple pattern-match bugs (e.g., obvious buffer overflows) are robust to context noise, but vulnerabilities requiring semantic reasoning (e.g., logic errors, race conditions) are heavily influenced by surrounding text. If you’re relying on LLMs for the hard cases, you’re relying on them where they’re most fragile.

3. Context sanitization is a new defensive primitive. Just as input sanitization prevents injection attacks, context sanitization — stripping or neutralizing surrounding metadata before feeding code to an LLM — becomes a security-relevant preprocessing step. This is cheap to implement and could be a standard recommendation in any LLM-integrated code review pipeline.