When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

Paper: 2602.14161 Authors: Max Fomin Categories: cs.LG

Abstract

As LLM-based agents increasingly handle untrusted data from diverse sources, detecting prompt injection and jailbreak attacks becomes critical for safe deployment. This paper presents a comprehensive analysis revealing fundamental limitations in current evaluation practices and production systems. Using a diverse benchmark of 18 datasets spanning harmful requests, jailbreaks, indirect prompt injections, and extraction attacks, the authors propose Leave-One-Dataset-Out (LODO) evaluation to measure true out-of-distribution generalization. The findings are striking: standard train-test splits from the same dataset sources severely overestimate performance, with aggregate metrics showing 8.4 percentage point AUC inflation and per-dataset gaps ranging from 1% to 25% accuracy. Through Sparse Auto-Encoder (SAE) analysis, the research identifies that 28% of top classifier features are dataset-dependent shortcuts rather than semantic signals. Production guardrails like PromptGuard 2 and LlamaGuard fail dramatically on indirect attacks (7-37% detection) and cannot evaluate agentic tool injection due to architectural constraints.

Key Contributions

• LODO Evaluation Protocol: Introduces Leave-One-Dataset-Out cross-validation as the gold standard for measuring true generalization in prompt attack detection, exposing severe performance overestimation in standard evaluation practices
• Comprehensive Benchmark: Assembles 18 diverse datasets covering harmful requests, jailbreaks, indirect prompt injections, and extraction attacks to enable rigorous out-of-distribution testing
• Dataset Shortcut Analysis: Uses Sparse Auto-Encoder feature analysis to identify that 28% of top classifier features are dataset-dependent artifacts rather than genuine semantic signals
• Production System Evaluation: Systematically benchmarks PromptGuard 2, LlamaGuard, and LLM-as-judge approaches, revealing critical failures on indirect attacks and architectural limitations for agentic scenarios
• Open-Source Framework: Releases evaluation framework at https://github.com/maxf-zn/prompt-mining to establish reproducible standards for the research community

The Evaluation Crisis in Prompt Security

The paper begins by highlighting a critical disconnect between evaluation practices and real-world deployment scenarios. Current research typically evaluates prompt injection classifiers using random train-test splits from the same dataset sources. While this approach yields impressive metrics, it fundamentally misrepresents how these systems perform when encountering novel attack patterns in production.

The LODO evaluation protocol addresses this by training classifiers on $n-1$ datasets and testing on the held-out dataset, repeating this process for each dataset. This methodology mirrors the actual deployment challenge: classifiers must generalize to attack patterns they haven’t seen during training. The results are sobering—aggregate AUC drops by 8.4 percentage points under LODO evaluation, but this masks heterogeneous failure modes where individual datasets show accuracy gaps ranging from 1% to 25%.

This performance degradation isn’t uniform across attack types. The research identifies that classifiers struggle most with indirect prompt injections targeting agentic systems, where attackers embed malicious instructions within seemingly benign content that agents process from external sources. This finding has immediate implications for the rapidly growing ecosystem of LLM-based agents that interact with emails, documents, and API responses.

Mechanistic Analysis: Why Classifiers Fail

To understand the root causes of generalization failure, the authors employ Sparse Auto-Encoder (SAE) analysis to decompose classifier decision-making into interpretable features. SAEs learn to represent neural network activations as sparse linear combinations of learned features, enabling researchers to identify which features drive classification decisions.

The analysis reveals a troubling pattern: 28% of the top features used by classifiers are dataset-dependent shortcuts. These features exhibit strong class signal (correlation with malicious vs. benign labels) within specific datasets but fail to generalize across datasets. In other words, classifiers learn to exploit statistical artifacts of dataset construction rather than semantic properties of attacks.

For example, a feature might strongly correlate with malicious prompts in Dataset A because that dataset’s collection methodology introduced systematic differences in prompt length, formatting, or vocabulary. When the classifier encounters Dataset B with different collection artifacts, this feature becomes unreliable or even misleading.

The paper introduces the concept of LODO-stable features—features that maintain consistent class signal across all LODO folds. These features represent genuine semantic patterns of malicious prompts rather than dataset artifacts. By filtering explanations to only include LODO-stable features, the research demonstrates more reliable interpretability for classifier decisions.

Production Systems Under Scrutiny

The systematic evaluation of production guardrails reveals critical gaps in current defensive capabilities. PromptGuard 2 and LlamaGuard, two widely deployed prompt security systems, show detection rates of only 7-37% on indirect attacks targeting agents. This failure rate is particularly concerning given that indirect attacks represent the primary threat vector for agentic systems processing untrusted external data.

The architectural analysis exposes fundamental limitations: both PromptGuard 2 and LlamaGuard are designed to evaluate single prompts in isolation. They cannot assess whether a prompt attempts to manipulate an agent’s tool usage or decision-making process because they lack context about the agent’s capabilities, current state, and intended workflow. This architectural constraint means these systems are blind to an entire class of attacks specifically designed for agentic scenarios.

LLM-as-judge approaches, which use large language models to evaluate prompt safety, show somewhat better performance on indirect attacks but still fail to achieve reliable detection rates. The paper suggests this reflects a fundamental challenge: distinguishing between legitimate instructions for an agent and malicious attempts to hijack agent behavior requires deep understanding of context, intent, and potential consequences—capabilities that current systems lack.

Implications for AI Safety

The research carries significant implications for AI safety practices and deployment strategies. First, it establishes that current evaluation methodologies provide false confidence in prompt security systems. Organizations deploying LLM-based agents based on published benchmark results may be operating under dangerous misconceptions about their actual security posture.

Second, the identification of dataset-dependent shortcuts suggests that simply collecting more training data won’t solve the generalization problem. Without careful attention to dataset construction and evaluation methodology, larger datasets may simply encode more sophisticated shortcuts that fail in novel deployment contexts.

Third, the failure of production systems on indirect attacks highlights an urgent need for architectural innovation. Effective prompt security for agentic systems requires context-aware evaluation that understands agent capabilities, current state, and potential attack surfaces. This likely requires tight integration between security systems and agent architectures rather than bolt-on classifiers.

The paper’s release of the evaluation framework and benchmark datasets provides the research community with tools to develop and validate more robust approaches. By establishing LODO as the standard evaluation protocol, the work creates accountability for claims about prompt security system performance.

Takeaways

1. Standard evaluation practices using same-dataset train-test splits overestimate prompt classifier performance by 8.4 percentage points AUC on average, with individual dataset gaps reaching 25% accuracy
2. Leave-One-Dataset-Out (LODO) evaluation provides a more realistic measure of out-of-distribution generalization and should become the standard protocol for prompt security research
3. 28% of top classifier features are dataset-dependent shortcuts that exploit statistical artifacts rather than semantic attack patterns, explaining poor generalization
4. Production systems PromptGuard 2 and LlamaGuard achieve only 7-37% detection on indirect attacks and cannot evaluate agentic tool injection due to architectural limitations
5. LODO-stable SAE features enable more reliable interpretability by filtering dataset artifacts from classifier explanations
6. Effective prompt security for agentic systems requires architectural innovation beyond current single-prompt evaluation approaches
7. The open-source evaluation framework at https://github.com/maxf-zn/prompt-mining enables reproducible research and standardized benchmarking for the community