Inducing Task Models from Computer-Use Traces: Turning Raw Clicks into Auditable Workflows

Paper: 2608.20319 Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang Categories: cs.AI, cs.CL

The Gap

Computer-use agents are moving into real work, and that creates a second problem beyond capability. Organisations need to know how work is actually performed — to audit it, to hand it to an agent, and to reuse it. Passively recorded traces (screenshots plus mouse and keyboard events) already contain that knowledge. Nobody can extract it.

Two obstacles stand in the way, and the second is the one that matters. First, activity is observed only as low-level events: a click at a coordinate, a keypress, a frame. Nothing in the recording says “this was the invoicing step”. Second — and this is where existing methods fail outright — real work is multi-threaded. A person drafting a reply, checking a dashboard, and returning to the draft produces a single interleaved stream of events, and no boundary marker separates the threads. A trace is one timeline containing several overlapping goals.

Existing approaches sidestep this by assuming it away. They take either a given task or a single workflow as input, and they output step-level summaries — a sentence per action, in order. That produces a readable log, not a model. You cannot audit a log against intent, and you cannot reuse a log as a skill, because it never says what the goal was or how the steps were allowed to branch.

   [RAW TRACE] screenshots + mouse/keyboard events, one stream
        |
        |   "click (412, 88) ... type 'Dear...' ... click tab
        |    ... scroll ... click (90, 14) ... type 'Total' ..."
        |
        v
   WHAT IS MISSING
        |
        +-- which events belong to which GOAL?     (no markers exist)
        +-- real work interleaves several goals    (multi-threaded)
        +-- no structure, only ordered events
        |
        v
   [PRIOR WORK] assumes a given task, or one workflow
        -> emits step-level summaries (a readable log)
        -> not auditable (no goal), not reusable (no control flow)

The Increment

One sentence: Before this paper, a passively recorded work trace could only be summarised step by step, and only if you already knew the task; after it, Task Model Induction recovers the latent tasks hiding in an unconstrained trace and gives each one a structured model you can audit and hand to an agent.

Core Mechanism

Task Model Induction (TMI) does two things, in the order the problem demands.

It discovers the latent tasks. Given an unconstrained trace with interleaved, concurrent activity, TMI disentangles the threads — deciding, for every event, which goal it belongs to. This is the step prior work assumed away, and it is what makes the rest possible: without it, everything downstream is summarising noise.

It induces a task model per latent task, and the model has two parts that answer different questions. A hierarchical objective model captures recursive goal decomposition — the top-level goal, and the subgoals beneath it, and the subgoals beneath those. A procedure model captures the control flow that organised the actual execution: the ordering, the branches, the loops, the points where the work could legitimately go several ways. Together these are what turn a sequence of events into something a person can audit and a system can reuse.

The results are separated by the authors into intrinsic and extrinsic, which is the right split for this problem. Intrinsically, on controlled human and agent trajectories, TMI recovers the interleaved tasks with 0.974 agreement against ground-truth groupings, and reconstructs 74.9% of the observed execution steps — far more than the strongest workflow-induction baseline. The 0.974 is the load-bearing number: it says the hard part (disentangling) is solved on trajectories where ground truth exists.

Extrinsically, the test is whether the recovered model is useful. Skills derived from TMI’s task models improve held-out task accuracy by 30.0% over the strongest baseline. This matters because it closes the loop: the induced model is not merely a faithful description, it is a productive one.

   TASK MODEL INDUCTION

   [unconstrained trace: interleaved events, several goals]
                          |
                          v
   +-------------------------------------------------------+
   | STEP 1: DISCOVER LATENT TASKS                         |
   |   disentangle concurrent activity                     |
   |   assign every event to a goal                        |
   |   -> 0.974 agreement vs ground-truth groupings        |
   +--------------------------+----------------------------+
                              v
   +-------------------------------------------------------+
   | STEP 2: INDUCE A TASK MODEL PER LATENT TASK           |
   |                                                       |
   |   HIERARCHICAL OBJECTIVE MODEL                        |
   |     goal -> subgoal -> subgoal   (recursive)          |
   |                                                       |
   |   PROCEDURE MODEL                                     |
   |     ordering, branches, loops, control flow           |
   +--------------------------+----------------------------+
                              v
   +-------------------------------------------------------+
   | USES                                                  |
   |   audit: does execution match intent?                 |
   |   reuse: derive a skill -> +30.0% held-out accuracy   |
   |   reconstruct 74.9% of observed steps                 |
   +-------------------------------------------------------+

Think of it as a restaurant manager reconstructing a single busy cook’s shift from the security footage. The footage is one continuous stream: the cook chops, checks the oven, answers the phone, returns to chopping, plates, and starts a second dish. Nothing in the footage announces “now beginning dish two”. A naive summary produces a list of actions in time order, which is useless for training anyone. What the manager actually needs is: there were three dishes in flight; dish one’s goal decomposed into prep, cook, plate; dish two was started only when the oven freed up, which is a branch in the procedure, not a step. That is the difference TMI goes after — it hands you the recipes and the reason the cook did them in that order, recovered from footage that never labelled either.

Key Concepts

• Multi-threaded work: the property that real activity interleaves multiple concurrent goals within one observed stream. This is what defeats step-level summarisation, and it is emphatically not an edge case — it is what ordinary work looks like.
• The two-part task model: pairing a hierarchical objective model (what the work was trying to achieve, decomposed recursively) with a procedure model (the control flow that organised execution). Both are required: the objective alone cannot be executed, and the procedure alone cannot be audited, because there is nothing to audit it against.
• Intrinsic versus extrinsic evaluation: measuring the recovery of structure against ground truth (intrinsic), and measuring whether derived skills actually work on held-out tasks (extrinsic). For this problem the intrinsic number is the harder bar and the extrinsic number is the one that justifies the exercise.

Framework Shift

Before (assume the task, summarise the steps):
  input:  a given task OR a single workflow
  output: step-level summary
          "clicked X, then typed Y, then clicked Z"
  - cannot audit: no goal to compare against
  - cannot reuse: no branches, no decomposition
  - breaks when the trace interleaves goals

After (Task Model Induction):
  input:  an unconstrained trace, goals unknown
  step 1: disentangle concurrent activity   -> latent tasks
  step 2: per task, induce a model
            objective model (recursive goals)
          + procedure model (control flow)
  - auditable against intent
  - reusable as a skill (+30.0% held-out)

From producing a readable log of what was done, to recovering a structured model of what was being attempted and how the steps were allowed to unfold, the core shift is treating the label as something to recover rather than something to be given.

Expert Assessment

Problem choice: Excellent, and it will age well. As computer-use agents enter workplaces, the scarce resource stops being screenshots and becomes structure — the ability to say what a body of recorded work was for. Choosing to solve the unlabelled, multi-threaded version rather than the convenient single-workflow version is what makes this a real contribution rather than a summarisation demo.

Method maturity: The decomposition into discover-then-induce is the right architecture, and it maps onto the two genuine difficulties in order. The reported 0.974 grouping agreement is impressively high, and it is worth asking why: the number comes from controlled trajectories with ground truth, which are by construction cleaner than open-ended real work. The gap between that setting and the multi-threaded mess the motivation describes is the main open question the paper leaves.

Experimental integrity: Splitting results into intrinsic and extrinsic is exactly the right evaluation design, and the extrinsic test — derive skills, then measure held-out accuracy — is the one that could have embarrassed the method, so it is good that it is reported. The 74.9% step reconstruction is a useful honesty check: a model that recovers three-quarters of steps is not claiming to be a complete account of what happened, and the paper does not overstate it.

Writing quality: The motivation lands because multi-threading is stated as the normal case rather than a complication. Where the paper could do more is the procedure model: it is the half most directly usable as a skill, yet the objective model gets cleaner exposition. A worked example of one induced procedure model, side by side with the raw trace it came from, would make the contribution concrete for practitioners.

Verdict: strong accept — it identifies the obstruction that was actually blocking useful work on traces, and demonstrates recovery and reuse on the version of the problem worth solving.

Takeaways

• Before summarising any recorded activity, decide which goal each event belongs to. Step-level summaries of multi-threaded work are misleading no matter how fluent they are.
• Model the objective and the procedure separately. A goal hierarchy without control flow cannot be executed; control flow without goals cannot be audited.
• Prefer recovering structure over being given it. Requiring a task label as input is precisely what limited the previous generation of methods to single-workflow traces.
• Test induced models extrinsically. If skills derived from the model do not improve held-out task accuracy, the model is a description rather than knowledge — and only the extrinsic test can tell you which you have.