When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Paper: 2605.00817 Authors: Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh Categories: cs.CL

The Gap

Existing LLM evaluation focuses on final-answer accuracy on reasoning benchmarks like GSM8K or MATH. Models score 80-90% on these tasks, leading to claims about emergent reasoning abilities. But final correctness doesn’t reveal whether models actually follow the procedure specified in the prompt, or just pattern-match their way to plausible answers.

Prior work on chain-of-thought and step-by-step reasoning assumes that if a model produces intermediate steps and gets the right answer, it must be executing the procedure. No one systematically tested whether models can faithfully execute explicit, step-by-step algorithms when the procedure itself is the constraint, not just a scaffold for reasoning.

Problem: High benchmark scores hide execution failures
   |
   v
Assumption: Final accuracy != procedural faithfulness
   |
   v
Method: Controlled diagnostic with explicit algorithms
   |         (simple ops, variable length, dependencies)
   v
Evidence: 61% -> 20% accuracy as steps increase 5 -> 95
   |         Missing answers, premature stops, hallucinations
   v
Conclusion: Reasoning ability masks instruction-following weakness

The Increment

One sentence: Before this paper, we measured whether LLMs get the right answer; after, we know they systematically fail to execute the procedure that produces it.

Core Mechanism

The benchmark constructs algorithmic procedures from simple arithmetic operations (add, subtract, multiply, divide, modulo). Each algorithm takes two numeric inputs and produces a final value through a sequence of steps. Steps reference intermediate variables from earlier in the procedure, creating look-back dependencies.

Complexity scales along two axes: algorithm length (5 to 95 steps) and dependency depth (how far back a step must look to find its operands). The operations themselves remain trivial—third-grade arithmetic—so failure cannot be blamed on mathematical difficulty. The challenge is pure procedural execution: read the algorithm, track variables, apply operations in order, return the final value.

The authors test 14 models across 55 datasets, analyzing not just final accuracy but generation-level failures: missing answers (model stops without output), premature answers (returns a value mid-procedure), self-correction (gives wrong answer, then corrects it), under-execution (skips steps), and hallucinated extra steps.

Input Algorithm:
  Step 1: a = input1 + input2
  Step 2: b = a * 3
  Step 3: c = b - input1
  Step 4: return c
         |
         v
    [LLM processes]
         |
         v
   Generation trace:
   "Step 1: a = 5 + 3 = 8
    Step 2: b = 8 * 3 = 24
    Step 3: c = 24 - 5 = 19
    Answer: 19"
         |
         v
   Diagnostic analysis:
   - Correct final answer?
   - All steps present?
   - Steps in order?
   - Correct intermediate values?
   - Premature termination?
   - Hallucinated steps?

Think of this like testing a sous chef. You don’t just taste the final dish and declare them competent. You hand them a recipe with explicit steps—dice onions, sauté for 3 minutes, add garlic, deglaze with wine—and watch whether they follow it. The recipe uses basic techniques (chopping, heating), so failure means they can’t follow instructions, not that the techniques are too hard.

The benchmark is the recipe. The arithmetic operations are the basic techniques. The model is the sous chef. And the diagnostic reveals that as recipes get longer, the chef starts skipping steps, adding random ingredients, or serving the dish before it’s done—even though each individual technique is trivial.

Key Concepts

• Procedural faithfulness: The model’s ability to execute the exact sequence of operations specified in a prompt, independent of whether the final answer is correct. A model can get the right answer through shortcuts or pattern matching without following the procedure. Conversely, it can follow every step correctly but make an arithmetic error. Faithfulness measures adherence to process, not outcome. Example: If the prompt says “multiply by 3, then add 5,” a faithful model must perform those operations in that order, even if it could reach the same result by adding 5 first and then multiplying.

• Look-back dependencies: When a step in an algorithm references variables computed in earlier steps, it creates a dependency chain. Step 5 might use the result from Step 2 and Step 4. As dependency depth increases (referencing steps further back), the model must maintain a longer working memory of intermediate values. This is distinct from algorithm length—a 10-step procedure where each step only uses the previous step has shallow dependencies, while a 10-step procedure where Step 10 uses Step 1 has deep dependencies.

• Generation-level failure modes: Beyond binary correct/incorrect, the paper categorizes how models fail during generation. Missing answers: the model produces reasoning steps but never outputs a final value. Premature answers: it returns a value before completing all steps. Self-correction: it gives an answer, then continues generating and changes it. Under-execution: it skips steps in the middle. Hallucinated steps: it invents operations not in the algorithm. These patterns reveal that models don’t just make calculation errors—they lose track of the procedural structure itself.

Framework Shift

Before (mainstream evaluation):        After (this paper):

Prompt: "Solve this problem"          Prompt: "Execute this algorithm"
   |                                      |
   v                                      v
[LLM generates reasoning]              Step 1: a = x + y
   |                                   Step 2: b = a * 2
   v                                   Step 3: return b
Final answer: 42                          |
   |                                      v
   v                                   [LLM must follow exactly]
Metric: Is 42 correct?                    |
   |                                      v
   v                                   Metrics:
Conclusion: Model can reason           - Correct final value?
                                       - All steps executed?
                                       - Correct order?
                                       - No hallucinations?
                                          |
                                          v
                                       Conclusion: Model struggles
                                       with procedural control

From outcome-based evaluation to process-based diagnosis, the core shift is separating “getting the right answer” from “following the instructions.”

Expert Assessment

Problem choice: This is a real gap. The field has been measuring reasoning through final-answer accuracy on increasingly complex benchmarks, but no one isolated procedural execution as a distinct capability. The problem sits at a critical juncture—as we deploy LLMs for tasks requiring faithful instruction-following (code execution, tool use, multi-step workflows), we need to know whether high benchmark scores reflect genuine procedural competence or clever pattern matching. The authors didn’t manufacture this gap; they exposed an assumption everyone was making.

Method maturity: Elegant in its simplicity. Using trivial arithmetic operations removes confounding factors (mathematical difficulty, domain knowledge) and isolates procedural control. The two-axis scaling (length and dependency depth) is well-motivated. However, the paper could have explored whether failure modes differ across operation types or whether certain dependency patterns are harder than others. The diagnostic categories (missing answers, premature answers, etc.) are useful but somewhat ad-hoc—a more principled taxonomy of failure modes would strengthen the analysis.

Experimental integrity: Baselines are fair—14 models spanning different architectures and sizes, 55 datasets with controlled variation. The numbers are stark and consistent across models, which increases confidence. One concern: all algorithms are deterministic and have a single correct answer. Real-world procedures often involve conditional logic, loops, or ambiguous specifications. The paper’s scope is narrow by design, but this limits generalization. The generation-level analysis is thorough, though manual inspection of failure modes on a subset of outputs introduces some subjectivity.

Writing quality: The paper is clear and well-structured. The introduction motivates the problem effectively, and the results section presents findings without overselling. The weakest section is the discussion—it gestures at implications for reasoning and instruction-following but doesn’t deeply engage with why models fail this way or what architectural changes might help. Rewriting the discussion to propose mechanistic hypotheses (attention span limitations? training data biases toward shortcuts?) would elevate the paper from diagnostic to generative.

Verdict: weak accept — Identifies a real and underexplored weakness in LLMs with a clean experimental design, but stops short of explaining why the weakness exists or how to fix it.

Takeaways

If you’re building systems that rely on LLMs to follow multi-step procedures—agents, workflow automation, code interpreters—don’t trust final-answer accuracy as a proxy for faithful execution. Test explicitly for procedural adherence, especially as task length increases.

The failure modes (premature answers, under-execution, hallucinations) suggest that LLMs lose track of procedural state as context grows. If your application requires long procedures, consider breaking them into smaller sub-procedures with explicit checkpoints, or use external state management (a scratchpad, a database) rather than relying on the model’s context window.

The diagnostic framework itself is portable: take a capability you care about, design a minimal test that isolates it from confounding factors, and scale complexity along controlled axes. This paper’s value isn’t the specific finding about arithmetic procedures—it’s the template for exposing gaps between benchmark performance and actual capability.