Move by Move: How LLMs Actually Conduct Therapy, and How to Steer It

Paper: 2608.21325 Authors: Afonso Baldo, Hugo Pitorro, Areti Vassilopoulos, Anabela C. Areias, Maya D’Eon, Fabíola Costa, Ricardo Rei, Nuno M. Guerreiro Categories: cs.CL

The Gap

People are already using language models for emotional support, at scale, whether or not the models were designed for it. But almost nothing is known about how these models actually conduct a therapeutic interaction. The gap is not one of outcome measurement — whether users feel better is hard to attribute and slow to measure. It is a gap of mechanism: nobody had a vocabulary for describing what a model does, turn by turn, as a therapist.

That absence has a cost, and it is a specific one. Without a shared set of categories, you cannot compare a model’s behaviour to a human clinician’s, because “supportive response” is not a behaviour — it is an impression. You also cannot audit a model for a systematic stylistic bias, and you cannot steer it, because steering requires naming the thing you want to change.

   [REALITY] users already use LLMs for emotional support
        |
        v
   [WHAT WE MEASURE] helpfulness / safety / user satisfaction
        |
        v
   [WHAT WE CANNOT SAY] what the model actually DOES
        - "supportive response" is an impression, not a behaviour
        - no shared categories -> no comparison to clinicians
        - no names -> no auditing, no steering
        |
        v
   [GAP] no vocabulary for the move-by-move conduct of
         a model-led therapeutic interaction

The Increment

One sentence: Before this paper, there was no vocabulary for what a model does in a therapy conversation and therefore no way to compare it to a clinician or steer it; after it, a validated ten-move ontology turns therapeutic conduct into a measurable distribution, and handing that ontology to the model as tools halves its deviation from human behaviour.

Core Mechanism

The ontology is the contribution, and its construction is what earns it credibility. Ten therapeutic moves are defined as compact, function-based categories grounded in the MULTI-60 inventory — a therapist’s repertoire, not a generic taxonomy of sentiment. “Function-based” is the key word: moves are classified by what they are trying to accomplish in the interaction, which is what makes them applicable to model output without asking whether the model intended anything.

Validation is two-stage. Five licensed psychologists annotated interactions, establishing expert agreement on the categories; the ontology was then scaled using a judge-based approach that was verified to match that expert agreement. So the measurement instrument is anchored to clinicians before it is applied at scale — the ordering matters, because it means the later findings rest on a category scheme that practitioners of the discipline endorsed.

Applying it produces three findings about frontier models, and they are behavioural rather than evaluative.

Models over-use inquiry at up to three times the human rate. The default therapeutic reflex of these models is to ask another question. That is a recognisable conversational tic, and at three times the human rate it is a strong stylistic signature rather than a subtle tendency.

They neglect psychoeducation. Explaining, teaching, normalising — giving the person something to take away — is under-represented relative to clinician behaviour.

They are strongly context-anchored. This is the most nuanced finding. Models carry forward strategies a human clinician initiated, but rarely initiate those strategies themselves. The model is capable of the move; it just does not reach for it unprompted. That distinction between capacity and propensity is only visible because the ontology makes the moves nameable.

The intervention follows directly. Exposing the ontology to the model as a set of tools — making the ten moves concrete, callable options rather than latent tendencies — roughly halves the mean deviation from the human move distribution and improves turn-level alignment with human therapists by 7 to 9 percentage points, with no fine-tuning at all.

   ONTOLOGY CONSTRUCTION
   MULTI-60 inventory
        |
        v
   10 therapeutic moves (function-based, compact)
        |
        v
   5 licensed psychologists annotate  --> expert agreement
        |
        v
   judge-based scaling verified to MATCH expert agreement
        |
        v
   apply to real counseling transcripts + model-led sessions
        |
        v
   FINDINGS
     - inquiry over-used: up to 3x human rate
     - psychoeducation neglected
     - context-anchored: follows human-initiated strategies,
       rarely initiates them
        |
        v
   INTERVENTION: expose ontology as tools
     -> mean deviation from human distribution roughly HALVED
     -> turn-level alignment +7-9 points
     -  no fine-tuning

Think of it as a chess coach reviewing a transcript with a notation for tactics. Watching two games, you can say “one player was more aggressive” — an impression, unfalsifiable, unactionable. Give the coach a notation that names the actual tactical motifs, and three things become possible at once: you can count how often each motif appears in each game, you can compare a student to a grandmaster and say “you play pins three times as often and never play a fork”, and — the important one — you can put that list of motifs in front of the student as a checklist. The last step is what the paper’s intervention does, and the result is that the student starts playing the moves they were always capable of, simply because the moves now have names.

Key Concepts

• Function-based therapeutic moves: categories defined by what a move tries to accomplish in the interaction rather than by its surface form or the speaker’s intent. This is what makes the scheme applicable to model output, where intent is not observable, and it is what allows comparison across human and model transcripts.
• Context anchoring: the observed tendency to continue a strategy a human initiated while rarely initiating it oneself. It separates capability from propensity, and it is a failure mode that aggregate helpfulness scores cannot detect, because a model that follows well still looks helpful.
• Ontology-as-tools: the intervention of exposing the category scheme to the model as callable tools. It is a form of prompting, but a structurally different one from instruction-tuning: rather than teaching the behaviour, it makes an existing capability selectable, which is why it works without any weight updates.

Framework Shift

Before (impression-level evaluation):
  conversation -> "helpful? safe? empathetic?"
  output: a rating
  - cannot compare to a clinician (no shared units)
  - cannot see systematic bias
  - cannot steer: nothing to name

After (move-level measurement):
  conversation -> distribution over 10 therapeutic moves
  output: a measurable profile
  - compare model vs clinician distributions
  - audit for specific over/under-use
  - steer by exposing the moves as tools
    (no fine-tuning, 7-9 pt alignment gain)

From asking whether a model’s support feels helpful, to counting which therapeutic moves it makes and comparing that distribution to a clinician’s, the core shift is replacing an impression with a repertoire you can name, count, and hand back to the model.

Expert Assessment

Problem choice: Excellent, and it takes the right half of the problem. Outcome evaluation of AI therapy is a swamp — user-reported wellbeing is confounded by everything. Choosing instead to measure conduct — what the model does, move by move — makes the question tractable, and it happens to be the question a supervising clinician would actually ask.

Method maturity: The two-stage validation is the paper’s methodological backbone and it is done properly: experts first, judge-model second, with the judge explicitly verified against the expert baseline. The intervention is elegant in its economy — exposing the ontology as tools is a prompting change, and getting a 7-9 point alignment gain without touching weights suggests the capability was already present and merely unprompted.

Experimental integrity: The findings are stated as distributions against a human reference, which is the right frame, and the comparison set includes real counseling transcripts rather than only synthetic sessions. The residual concern is the well-known one for judge-based measurement at scale: the judge was validated to match expert agreement on the annotated sample, but how that agreement holds across the full distribution — particularly on the under-used moves where data may be sparse — is not fully characterised.

Writing quality: The finding that models are context-anchored rather than incapable is the paper’s most interesting observation, and it is presented with the right emphasis. What would lift the paper is a concrete example: one short transcript excerpt annotated with its move sequence, human and model side by side, would make the three findings immediately legible to a clinician reader in a way the distribution statistics do not.

Verdict: strong accept — it builds a validated instrument for a domain that was being evaluated by impression, and then shows the instrument itself is the intervention.

Takeaways

• Measure conduct, not just outcomes, when evaluating any model that talks to people. Naming the moves is what makes comparison, auditing and steering possible at all.
• Distinguish capability from propensity. A model that rarely initiates a strategy may be perfectly capable of it — the fix is making the capability selectable, not training it in.
• Exposing a category scheme as tools can be a cheaper intervention than fine-tuning, and here it was equally effective. If a behaviour is absent but nameable, try naming it first.
• Watch for the inquiry reflex in any conversational agent. Asking another question is the path of least resistance, and at three times the human rate it becomes a style rather than a choice.