Learning User Simulators with Turing Rewards

Paper: 2606.19336 Authors: Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P. Levy, Yoon Kim Categories: cs.CL

The Gap

Existing user simulators are trained as LLMs to match a single ground-truth response per context. The dominant approaches are:

• MLE (Maximum Likelihood Estimation): maximize the log-probability of the reference response.
• SimRL (Similarity-based RL): use a reward like BLEU or BERTScore on the distance to the reference.

Both assume that matching one correct output is sufficient. This fails because human responses are diverse—there are many plausible responses to the same context. Training to match a single target either averages over possibilities (producing bland outputs) or overfits that one surface form. The resulting simulator cannot faithfully capture the full distribution of real user behavior.

The paper identifies this distributional mismatch as the core limitation. Their key insight: instead of rewarding fidelity to a *specific response, reward indistinguishability from the set of all possible real responses. They operationalize this through a Turing test: a judge (discriminator) tries to tell real from simulated responses, and the simulator is rewarded when it fools the judge. This shifts the training signal from “how close is this to the right answer?” to “could this have been said by a real user?”

[ Problem: Simulators match one reference               ]
[  - MLE: maximize log P(ref | context)                  ]
[  - SimRL: maximize similarity(ref, sample)             ]
[    |                                                    ]
[    v                                                    ]
[ Limitation: Ignores diversity of real responses;        ]
[   produces outputs that are either bland or overfit     ]
[    |                                                    ]
[    v                                                    ]
[ Key Insight: Use Turing test as reward function         ]
[   - Reward = P(judge classifies sample as real)         ]
[    |                                                    ]
[    v                                                    ]
[ Method: Turing-RL (PPO with discriminator reward)       ]
[    |                                                    ]
[    v                                                    ]
[ Evidence: Outperforms MLE / SimRL on LLM and human eval ]
[    |                                                    ]
[    v                                                    ]
[ Conclusion: Optimizing for indistinguishability >       ]
[   optimizing for response matching                      ]

The Increment

One sentence: Before this paper, user simulators were trained by squeezing the model toward one predetermined answer; after this paper, they are trained to blend into the crowd of real users, making the entire distribution of possible responses fair game.

Core Mechanism

The method has two main components:

1. User Simulator LLM (the policy) – an autoregressive language model that generates a response given a conversation history. This is the model we want to train.
2. Discriminator LLM (the judge) – another LLM that, given the same history and a proposed response, outputs a scalar score: the probability that the response is from a real human (as opposed to the simulator).

Data flow during training:

• For each training example (history + real human response), the simulator generates a synthetic response.
• The discriminator scores both the real response (should be high) and the synthetic response (should be low). The discriminator is trained via binary classification to maximize this gap.
• The synthetic response’s discriminator score is used as the reward for the simulator. The simulator is updated with PPO (Proximal Policy Optimization) to maximize this reward.
• Periodically, the discriminator is fine-tuned on fresh real + simulated data to prevent it from being exploited.

This creates a symbiotic min-max game: the simulator tries to fool the discriminator; the discriminator tries to not be fooled. The reward is dynamic and adapts as the simulator improves.

+-----------------------+          +-----------------------+
| User Simulator (RL)  |          | Discriminator (Judge) |
| Policy: LLM          |--------->| LLM that scores       |
| Input: history       |          | response = real?      |
| Output: response     |          | Output: score [0,1]   |
+-----------+----------+          +----------+------------+
            |                                ^
            | reward = score                 |
            v                                |
+--------------------------------+          |
| Reinforcement Learning (PPO)   |          |
| Update simulator to maximize   |          |
| reward on generated responses  |          |
+--------------------------------+          |
         |                                   |
         | (periodically)                    |
         v                                   |
+--------------------------------+          |
| Update Discriminator:          |----------+
| binary classification: real vs |
| simulated responses            |
+--------------------------------+

Structural metaphor: The Art Forger and the Expert Appraiser

Imagine a forger (user simulator) who wants to sell paintings that pass as originals by a master (real human user). The forger has studied the master’s entire portfolio (training data). The old way: the forger picks one specific painting and tries to copy it exactly (reference matching). The forger ends up being good at that painting but can’t paint anything else in the master’s style.

The new way: the forger paints something new, then brings it to a world-class appraiser (discriminator). The appraiser examines the painting and gives a score: “This looks 90% real” or “this looks 30% real.” The forger practices (RL updates) to increase that score. Meanwhile, the appraiser also gets better by studying real paintings and previous forgeries. Over time, the forger learns the master’s entire style—not just one canvas. The forger can now produce any new painting that the appraiser cannot distinguish from a real masterwork.

Mapping:

• Forger → User Simulator LLM
• Appraiser → Discriminator LLM
• Style of master → Distribution of real user responses
• Forgery practice → RL episode
• Appraiser’s score → Reward
• The appraiser also training → Discriminator fine-tuning

Key Concepts

• Turing Test as a Reward Signal: A Turing test decides if an agent is indistinguishable from a human. Here it’s applied *locally: given a context, can a judge tell whether a response came from a real user or a machine? The score from the judge becomes the reinforcement signal. This is fundamentally different from supervised loss because the judge learns what “real” looks like across the whole dataset, and the simulator learns to match that distribution rather than a single point.

• Reinforcement Learning from Learned Rewards: When the true reward (human judgment) is too expensive to query at every step, a learned proxy can be used. Here the discriminator is that proxy. The simulator is free to explore any response; as long as the discriminator finds it realistic, it gets high reward. This encourages diversity and naturalness.

• Adversarial Min-Max Dynamics: The objective can be thought of as a two-player game: the simulator maximizes the discriminator’s error, the discriminator minimizes its error. This drives both models toward a Nash equilibrium where the simulator’s output distribution matches the real user’s distribution. Intuitively, the discriminator only gets better by seeing harder fakes, and the simulator only gets better by facing a tougher judge.

Framework Shift

The core shift is from point-wise matching to distribution-wise indistinguishability.

Before (mainstream approach):
+-----------------------------------+
| Training:                          |
|   Input: (history, real_response)  |
|   Objective: P(simulator_output)   |
|   close to real_response           |
|   Method: MLE or similarity reward |
+-----------------------------------+
| Simulator learns to reproduce      |
| the single most "correct" token    |
| sequence for a given history       |
+-----------------------------------+

After (this paper):
+------------------------------------------+
| Training:                                  |
|   Input: (history, real_response)          |
|   Objective: simulator_output should      |
|   be indistinguishable from real by judge |
|   Method: RL with discriminator reward     |
+------------------------------------------+
| Simulator learns to sample from the       |
| full distribution of plausible human      |
| responses for that history                |
+------------------------------------------+

One sentence: From “make this output look like the label” to “make your outputs look like they came from the label’s author.”

Expert Assessment

Problem choice: Real and timely. As LLM-based assistants become widespread, better user simulators are needed for evaluation and training. The existing approach of SFT/MLE is clearly insufficient for capturing human variability. This paper identifies a concrete boundary and attacks it with a clean idea.

Method maturity: Clever insight rather than brute force. The core idea (Turing test reward) is elegant and well-motivated. The implementation uses standard RL (PPO) and a standard discriminator, so the novelty is in the *framing and reward design, not in complex machinery. A simpler approach might be to use GAN-style training directly, but that often suffers instability; the authors’ two-stage pretrain+RL scheme is a pragmatic middle ground.

Experimental integrity: Baselines (MLE, SimRL) are fair and representative. Results are consistent across two domains (conversational chat and Reddit) and are confirmed by both automated LLM-based evaluation and human judgment. Sample sizes in human evaluation are modest (~100-200 per condition) but sufficient for a clear trend. One minor red flag: the SimRL baseline uses BLEU+distinctness, which is not the strongest similarity reward (e.g., BERTScore might be better). However, the paper still shows a clear win, and the comparison is defensible.

Writing quality: Clear and well-organized. The abstract sets up the gap properly. The figures are helpful. The discussion of limitations is a bit thin—they could elaborate on when the discriminator might fail (e.g., with very long contexts or rare user types). That section, if expanded, would strengthen the paper’s credibility.

Verdict: strong accept — A well-motivated, cleanly executed paper that offers a genuine improvement over a widely used approach. The idea is portable and likely to influence future user simulation work.

Takeaways

• Use a learned discriminator as a reward function for free-form generation tasks where “correctness” is ill-defined but human-likeness is desired.
• Pretrain the simulator with supervised learning before RL to give the discriminator a good starting point; otherwise the discriminator trivially rejects everything.
• Periodically update the discriminator to prevent mode collapse and maintain a meaningful reward landscape.
• The Turing test framing can be applied beyond user simulation: any task that asks “does this output sound like it came from a specific distribution?” (e.g., style transfer, persona grounding).
• If you are building a user simulator for evaluation of dialog systems, switching from MLE to a Turing-reward RL training is a concrete, low-risk upgrade that yields more realistic synthetic users.