Semantic Pareto-DQN: Multi-Objective RL Breaks the Fraud Collapse Trap

Paper: 2607.09641 Authors: Cláudio Lúcio do Val Lopes, Lucca Machado da Silva Categories: cs.LG, cs.AI

The Gap

Financial anomaly detection has a nasty structural problem: fraud cases are rare (often <1% of transactions), and the cost of a missed fraud is wildly different from the cost of a false alarm. Traditional approaches hit a wall here. Standard classifiers collapse into “fraud collapse” — they learn that predicting “not fraud” every time minimizes overall error. Oversampling methods like SMOTE distort the data distribution. Single-objective reward formulations in RL either over-prioritize recall (annoying legitimate customers) or over-prioritize precision (letting fraudsters through). Nobody has cleanly decoupled these competing objectives while also dealing with the heterogeneous, messy nature of transaction data (timestamps, amounts, merchant categories, user histories — apples and oranges).

This paper’s insight: (1) convert messy features into coherent natural-language narratives, (2) encode them with an LLM for scale-invariant representations, and (3) use multi-objective RL to explicitly map the Pareto frontier of trade-offs rather than collapsing everything into a scalar.

+-------------------+     +-----------------------+     +---------------------------+
| Class imbalance   |     | Single-objective RL   |     | Heterogeneous features    |
| + asymmetric costs|---->| collapses to majority |---->| resist simple encoding    |
+-------------------+     | class ("fraud trap")  |     +---------------------------+
                          +-----------+-----------+
                                      |
                                      v
                     +----------------------------------+
                     |  ASSUMPTION:                      |
                     |  Multi-objective formulation +    |
                     |  semantic state representation    |
                     |  can decouple competing goals     |
                     +----------------+-----------------+
                                      |
                                      v
                          +---------------------------+
                          |  METHOD:                  |
                          |  Pareto-DQN + LLM-encoded |
                          |  transaction narratives   |
                          +----------------+----------+
                                      |
                                      v
                      +---------------------------------+
                      |  EVIDENCE:                      |
                      |  E-Commerce fraud + UCI Credit  |
                      |  datasets. Breaks zero-recall   |
                      |  trap. Superior minority recall.|
                      +----------------+----------------+
                                      |
                                      v
                      +---------------------------------+
                      |  CONCLUSION:                    |
                      |  Bounded operational friction   |
                      |  buys genuine anomaly discovery |
                      |  without data resampling.       |
                      +---------------------------------+

The Increment

One sentence: Before this paper, deploying RL for fraud detection meant picking a single reward scalar and praying the trade-off was right; after this paper, you can map the full Pareto frontier and let stakeholders choose their operating point — while the system handles the messy feature heterogeneity via natural-language abstraction.

Core Mechanism

The pipeline has two major stages: semantic state construction and multi-objective policy optimization.

First, heterogeneous transaction features (amount, time, merchant category, user history, etc.) are serialized into a coherent natural-language narrative — something like “A customer who typically spends $50 at grocery stores on weekday afternoons just made a$2,000 purchase at an electronics retailer in another country at 3 AM.” This narrative is then fed into a pre-trained LLM (frozen, no fine-tuning) which produces a dense embedding. This embedding becomes the state representation for the RL agent. The key advantage: LLMs are already trained to understand semantic relationships, so the representation is naturally scale-invariant and captures contextual nuance that raw feature vectors miss.

Second, a DQN agent receives this semantic state and outputs action values. But instead of a single scalar reward, the agent receives a vectorial reward with three decoupled components: (1) financial efficacy (did we catch fraud? what’s the monetary impact?), (2) operational friction (did we block a legitimate customer? how annoying was the verification step?), and (3) semantic discovery (did we surface a genuinely novel fraud pattern?). The agent trains to approximate the full Pareto frontier — the set of policies where you can’t improve one objective without hurting another. At deployment time, a human stakeholder picks where on that frontier to operate.

+-------------------+
| Transaction Data  |
| (heterogeneous)   |
+--------+----------+
         |
         v
+-------------------+
| Narrative         |
| Serializer        |  "Customer who usually spends $50
|                   |   at groceries just bought $2000
+--------+----------+   electronics abroad at 3AM"
         |
         v
+-------------------+
| LLM Encoder       |
| (frozen backbone) |
| -> dense embedding|
+--------+----------+
         |
         v
+-------------------+     +---------------------------+
| DQN Agent         |---->| Vectorial Reward          |
| (state -> Q-val)  |     | [efficacy, friction,      |
+--------+----------+     |  discovery]               |
         |                +---------------------------+
         v                         |
+-------------------+              |
| Pareto Frontier   |<-------------+
| Approximation     |
+--------+----------+
         |
         v
+-------------------+
| Dynamic Policy    |
| Selection         |
| (stakeholder      |
|  picks operating  |
|  point)           |
+-------------------+

Structural Metaphor: Airport Security Screening

Think of this system as a modern airport security checkpoint with multiple independent screeners, each optimizing for a different goal.

The narrative serializer is like the check-in agent who summarizes your trip in plain English to the security team: “This passenger has a one-way ticket purchased yesterday, is carrying unusual items, and has no checked luggage.” That’s much more informative than raw data fields (ticket number: 847291, purchase date: 2026-07-12, bags: 0). The LLM encoder is the experienced security officer who reads that summary and instantly grasps the contextual risk — they’ve seen thousands of cases and know what “unusual” means in context.

The multi-objective reward is like having three separate supervisors at the checkpoint: one who cares about catching contraband (efficacy), one who cares about passenger throughput and satisfaction (friction), and one who cares about identifying novel smuggling techniques (discovery). Each gives their own score for every screening decision. The Pareto frontier is the set of screening strategies where you can’t get better at catching smugglers without making passengers angrier, and vice versa.

At deployment, the airport manager picks the operating point: “We’re on high alert today, tolerate 15-minute average wait times” or “It’s a holiday weekend, we need smooth flow, accept slightly lower detection rates.” The system doesn’t force a single fixed trade-off — it maps the full menu and lets humans choose.

Key Concepts

• Semantic State Representation: Instead of feeding raw numbers and categories into an RL agent, this paper converts them into natural language first, then uses an LLM to encode that language into a dense vector. Why does this matter? Raw features are heterogeneous — a dollar amount, a timestamp, a merchant category code, and a user’s historical spending pattern live in completely different mathematical spaces. Concatenating them into a feature vector is like mixing apples, calendars, and barcodes. Language is a universal solvent here: it naturally integrates disparate information into a coherent narrative that an LLM can interpret. Concrete example: “$2,000 electronics purchase” and “3 AM local time” are just numbers in raw form, but in narrative form — “unusually large purchase at an unusual hour” — an LLM immediately grasps the semantic significance because it’s been trained on millions of texts describing exactly this kind of contextual anomaly.

• Pareto Frontier in RL: In single-objective RL, you get one optimal policy. In multi-objective RL, there’s a whole frontier of policies, each representing a different trade-off between objectives. The Pareto frontier is the set of all policies where you can’t improve one objective without making another worse. It’s like the efficient frontier in portfolio theory: you wouldn’t choose a portfolio that’s dominated (same risk but lower return), but among the efficient portfolios, which one you pick depends on your risk tolerance. The paper maps this frontier so that the deployment decision (how much friction is acceptable?) becomes a post-hoc human choice rather than a pre-hoc modeling assumption.

• Fraud Collapse: This is the failure mode where a fraud detection system learns to predict “not fraud” for everything because fraud cases are so rare that this strategy minimizes standard loss metrics. Imagine a student who knows 99% of the test answers are “B” — they’ll score 99% by just answering “B” every time, but they’ve learned nothing. The paper’s vectorial reward and Pareto approach structurally prevents this because one of the objectives (financial efficacy) explicitly rewards catching fraud, and the frontier mapping ensures you can’t optimize overall performance by sacrificing minority-class recall entirely.

Framework Shift

Before (mainstream):                After (this paper):

+----------------+                  +----------------+
| Raw Features   |                  | Raw Features   |
+-------+--------+                  +-------+--------+
        |                                   |
        v                                   v
+----------------+                  +----------------+
| Feature        |                  | Narrative      |
| Engineering    |                  | Serializer     |
| (manual,       |                  | (automatic)    |
|  brittle)      |                  +-------+--------+
+-------+--------+                          |
        |                                   v
        v                           +----------------+
+----------------+                  | LLM Encoder    |
| Classifier or  |                  | (semantic      |
| Single-obj RL  |                  |  embedding)    |
+-------+--------+                  +-------+--------+
        |                                   |
        v                                   v
+----------------+                  +----------------+
| Scalar Reward  |                  | DQN Agent      |
| (one number,   |                  +-------+--------+
|  fixed trade-  |                          |
|  off, collapse)|                          v
+----------------+                  +----------------+
                                    | Vectorial      |
                                    | Reward         |
                                    | [eff, friction,|
                                    |  discovery]    |
                                    +-------+--------+
                                            |
                                            v
                                    +----------------+
                                    | Pareto Frontier|
                                    | (dynamic       |
                                    |  operating     |
                                    |  point choice) |
                                    +----------------+

From scalarized single-objective optimization to vectorized Pareto frontier exploration with semantic state representations, the core shift is: the trade-off between catching fraud and avoiding friction is no longer baked into the model — it’s exposed as a navigable decision surface for humans to operate on.

Expert Assessment

Problem choice: This is a genuine gap. Class imbalance and asymmetric costs in fraud detection are well-known headaches, and the standard playbook (SMOTE, cost-sensitive loss, focal loss) has real limitations. The insight to combine multi-objective RL with LLM-based semantic encoding is non-obvious and addresses a real pain point — most practitioners I know would rather have a Pareto frontier to explore than argue about what weight to put on false positives vs. false negatives. It sits at an interesting intersection of three hot areas (LLMs, RL, financial ML) and the combination makes sense rather than feeling like a paper chasing buzzwords.

Method maturity: The LLM-as-state-encoder trick is clever but raises scalability concerns. Running every transaction through an LLM (even a frozen one) at inference time is expensive — I’d want to see latency and cost numbers. The Pareto-DQN approach itself is well-established in the multi-objective RL literature; the novelty is more in the application and semantic encoding than in the RL algorithm. There might be simpler baselines they’re overlooking: would a well-tuned XGBoost with cost-sensitive learning and careful threshold calibration get you 80% of the benefit at 5% of the complexity? The paper doesn’t explore this “practical baseline” adequately.

Experimental integrity: Two datasets (E-Commerce fraud and UCI Credit) is acceptable but thin. UCI Credit is a benchmark that’s been beaten to death — I’d want to see results on at least one more realistic, large-scale dataset. The baselines include scalarized RL variants, which is appropriate, but I’d want to see strong supervised baselines (XGBoost, LightGBM with SMOTE and cost-sensitive thresholds) to understand the absolute lift. The numbers on minority-class recall look promising, but I’d need to see the full precision-recall trade-off curves, not just cherry-picked operating points.

Writing quality: The paper reads reasonably well but the related work section could be sharper — it should more clearly delineate what’s been tried in multi-objective RL vs. what’s been tried specifically in fraud detection, and where the novelty sits relative to each stream. The experiments section would benefit from ablation studies: how much of the gain comes from semantic encoding vs. multi-objective formulation? Without this, it’s hard to attribute credit.

Verdict: weak accept — The combination of LLM-encoded semantic states with Pareto multi-objective RL is a genuinely interesting framework for a real problem, but the experimental scope is too narrow and the ablation analysis is missing to be fully convincing.

Takeaways

Three things worth stealing from this paper:

1. LLM-as-universal-feature-encoder for heterogeneous tabular data: Even if you don’t use RL, serializing messy tabular features into natural language and encoding with a frozen LLM is a powerful trick for feature integration. It sidesteps the brittle manual feature engineering that plagues many production ML systems. I could see this being useful in any domain where you have mixed numeric, categorical, and temporal features — insurance claims, medical records, HR analytics.

2. Expose trade-offs as a frontier, not a scalar: The general idea of training multi-objective models and letting stakeholders pick the operating point post-hoc is widely applicable. Any system where precision/recall trade-offs matter (content moderation, medical diagnosis, anomaly detection broadly) could benefit from mapping the Pareto frontier rather than pre-committing to a single F1 score.

3. Decoupled reward design: Breaking a complex objective into explicitly separated components (efficacy, friction, discovery) forces you to articulate what you actually care about. This is valuable even outside RL — it’s a good discipline for any ML project to enumerate the distinct objectives and think about how they interact rather than lumping them into one loss function.