How AI Assistance Affects Skill Development: Short-Term Gains, Longer-Term Losses

Paper: 2608.23543 Authors: Shang Wu, Catarina G Belem, Shuyuan Fu, Mark Steyvers, Padhraic Smyth Categories: cs.AI

The Gap

The promise of AI assistance is straightforward — better performance now. The worry is equally straightforward: that assistance substitutes for the very effort through which skill is built, so the performance gain is rented rather than owned. Both claims are widely asserted, and the second is stated far more often than it is measured.

Measuring it requires a design that most studies do not have. You need to observe the same people before assistance exists, while it is available, and after it is taken away. That third phase is the one that matters and the one usually missing, because it is the only phase in which you can see whether the skill stayed. And you need to separate three quantities that a naive before-and-after comparison conflates: how good someone was to begin with, how good they are afterward, and how much they individually changed.

Without that separation, the obvious analysis is also the wrong one. If you predict post-AI performance from AI-assisted performance, you will be reading a number that was produced partly by the AI.

   THE THREE PHASES (and why most studies skip the third)

   [1] BEFORE AI     baseline ability
        |
   [2] DURING AI     performance goes UP      <- easy to measure
        |                                     <- the "promise"
   [3] AFTER AI      assistance REMOVED       <- the phase that matters
        |                                     <- usually missing
        v
   [THE CONFOUND] predict post-AI ability from
                  AI-assisted performance?
        -> that score was partly produced by the AI
        -> overestimates what the person can do alone
        |
        v
   [GAP] the skill-retention question needs all three
         phases AND a decomposition of ability

The Increment

One sentence: Before this study, the claim that AI assistance erodes skill was plausible but largely unmeasured; after it, a three-phase controlled experiment with a Bayesian latent ability model shows that AI users perform worse after assistance is removed, that extrapolating from assisted scores overestimates them, and that independent reasoning effort predicts the size of the gain.

Core Mechanism

The design is a logic-puzzle experiment with on-demand AI assistance, and it has two features that make it work.

Three phases with the same participants. Tasks are completed before, during and after AI is available. The “after” phase is the whole point: it is the only measurement taken when the crutch is gone, and it is what separates a rented gain from an owned one.

Experimental variation in the cost of asking. The AI request cost is varied across participants, which turns “did they use it” from a self-selected choice into something closer to an assigned condition. The finding is unsurprising and important: lower-cost assistance induces more frequent AI use. Friction is the lever on usage, which means the amount of assistance is a design parameter rather than a fixed property of the tool.

Three results follow.

Participants who requested AI assistance during the access phase performed worse on the task after assistance was removed. This is the direct measurement of the concern, and it is a performance deficit, not merely an absence of gain.

Their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. This is the finding with the widest practical reach. The natural way to estimate a person’s ability is to look at what they recently achieved — but recent achievement was co-produced by the AI. Any downstream decision that extrapolates from assisted scores inherits a systematic optimism.

Greater independent problem-solving effort during the access phase is associated with larger gains in latent ability. This is the mechanism, and it is where the Bayesian latent ability model does its work. By separating initial ability, post-AI ability, and participant-specific skill change, the analysis can ask what predicts the change. The answer is independent effort — consistent with the interpretation that skill development is weaker precisely when assistance substitutes for independent reasoning.

   ANALYSIS STRUCTURE (Bayesian latent ability model)

   observed: puzzle outcomes across three phases
        |
        v
   decompose into:
     [ initial ability ]      who they were at the start
     [ post-AI ability ]      who they are at the end
     [ skill change ]         the participant-specific delta
        |
        v
   PREDICTOR OF THE DELTA:
     independent problem-solving effort during the AI phase
        -> more independent effort  =>  larger latent gain
        -> assistance that REPLACES reasoning => weaker gain
        |
        v
   ALSO: predicting post-AI ability from AI-assisted
         scores systematically OVERESTIMATES it

Think of it as learning to navigate versus following a navigator. Someone driving a new city with turn-by-turn directions arrives fine, on time, and learns almost nothing about the street layout — the navigation replaced the wayfinding rather than supporting it. Take the phone away and they are worse off than a driver who spent the same trips getting lost and correcting. The two telling details from the study map onto this cleanly: cheap directions mean you ask for directions constantly, and the passenger’s confidence after the trip is a bad predictor of whether they could drive it alone tomorrow.

Key Concepts

• Rented versus owned performance: a gain that persists only while the aid is present versus one that survives its removal. The three-phase design exists solely to distinguish these, and the “after” phase is the only place the distinction is observable.
• Overestimation from assisted scores: predicting unassisted ability from assisted performance inherits the AI’s contribution as if it were the person’s. It is a measurement error with policy consequences, because assessment decisions are typically made from recent achievement.
• Effort as the mediator: independent problem-solving effort predicts the size of the latent ability gain. This matters because it identifies a variable a person can actually control, and it gives a concrete reading of what “substituting for reasoning” means behaviourally.
• Friction as the usage lever: varying request cost changes how often assistance is used. If the goal is to preserve skill development, the cost of asking is a design parameter, not an incidental UI detail.

Framework Shift

Before (assistance evaluated by the gain it produces):
  measure performance with and without the tool
  -> assistance helps (true, and short-sighted)
  -> ability estimated from recent assisted output

After (assistance evaluated by what survives removal):
  measure before / during / after
  -> during: gain            (real)
  -> after:  DEFICIT vs those who did not use it
  -> ability from assisted scores: OVERESTIMATED
  -> predictor of retained gain: independent effort

  assistance that REPLACES reasoning  -> skill loss
  assistance that SUPPORTS reasoning  -> retained gain

From judging an aid by the performance it produces while present, to judging it by what the user can still do when it is gone, the core shift is that the interesting measurement is taken after the tool has been removed.

Expert Assessment

Problem choice: Excellent, and this is the right moment for it. The skill-erosion worry is now a standard objection in discussions of AI assistance, and it was being argued from intuition. The three-phase design is the minimum needed to test it, and it is notable how few studies implement the third phase.

Method maturity: The Bayesian latent ability model is doing real work rather than decorating the result. Without it, the natural analysis — predict unassisted performance from assisted performance — produces a biased answer, which the paper demonstrates rather than merely asserts. Varying request cost to make usage less self-selected is a sensible design choice, though it governs usage rather than randomising the decision to use AI, so the causal reading of the deficit still rests on the assumption that cost variation is as-good-as-random with respect to unmeasured ability.

Experimental integrity: The honesty of stating both a performance deficit and an overestimation effect — the two results most likely to be contested — is to the authors’ credit, and the direction of the deficit is consistent across the analysis. The scope caveat is the logic-puzzle setting: a task with short horizons, immediate feedback and no transfer requirement. Whether the same erosion appears in open-ended, long-horizon work where assistance may scaffold rather than replace is genuinely unknown, and the paper does not overclaim it.

Writing quality: The distinction between “skill development is weaker when assistance substitutes for independent reasoning” and the stronger claim “AI makes people worse” is maintained carefully, which is exactly the discipline this topic needs. The paper would be more useful with one concrete illustration of the overestimation — a plot of predicted-versus-actual unassisted performance against the diagonal — since that is the finding most likely to be actioned by anyone doing assessment.

Verdict: strong accept — it converts a widely repeated intuition into a measured effect with an identified mechanism, and it names a measurement error that has consequences well beyond this experiment.

Takeaways

• Evaluate any assistance tool on the “after” phase. A gain measured while the aid is present is not evidence of retained capability, and that phase is usually the one nobody runs.
• Do not extrapolate unassisted ability from assisted performance. The score was co-produced by the tool, and the resulting estimate is systematically optimistic.
• Treat the cost of asking as a design parameter. Friction reliably changes how often assistance is used, which makes it a lever on how much independent reasoning is displaced.
• Track independent effort, not just outcomes. Effort during the assisted phase predicted the latent ability gain here, which means it is both a diagnostic and the thing to protect if you want the skill to survive.