Stop Breaking Things: How Graph-Based Impact Analysis Cuts AI Coding Regressions by 70%

Paper: 2603.17973 Authors: Pepe Alonso Categories: cs.SE, cs.AI

The Gap

SWE-bench has become the de facto leaderboard for AI coding agents, but it measures one thing: did the agent resolve the issue? It says nothing about whether the agent broke something else in the process. Prior work on agents like SWE-agent, Devin, and similar systems optimizes hard for resolution rate while treating regressions as an acceptable side effect. The benchmark itself has no regression penalty — you get credit for fixing the bug even if you snap three other tests in half.

The deeper problem is that existing agent workflows treat test suites as oracles to consult after the fact, not as a map to navigate during development. TDD as a concept exists, but nobody had seriously asked: does telling a smaller model to “do TDD” actually help, or does it just add noise to the prompt?

Problem: AI agents fix bugs but break other tests
    |
    v
Assumption: Agents need to KNOW which tests are at risk
    |           (not just be told to "write tests first")
    v
Method: Build AST graph of code<->test relationships,
        rank tests by change impact, feed as context
    |
    v
Evidence: 70% regression drop (6.08% -> 1.82%),
          resolution up 24% -> 32% on SWE-bench Verified
    |
    v
Conclusion: Context (what to verify) beats procedure
            (how to do TDD) for smaller models

The Increment

One sentence: Before this paper, AI coding agents had no principled way to know which tests their changes would break; after it, a graph-based impact score can surface that context automatically and measurably reduce regressions.

Core Mechanism

TDAD works in three stages. First, it parses the repository using AST analysis to extract every function, class, and method definition then traces which test files import or call which production code symbols. This produces a bipartite graph: one side is production code nodes, the other is test nodes and edges carry weights based on call depth and coupling tightness.

Second, when an agent proposes a diff, TDAD computes an impact score for each test by walking the graph from the changed symbols outward. Tests that directly call a modified function score highest; tests that call something that calls something modified score lower. The top-K tests by impact score get surfaced as context to the agent before it finalizes the change.

Third, this context feeds into a GraphRAG-style retrieval step inside the agent’s workflow. Instead of the agent blindly running the full test suite (expensive) or guessing which tests matter (unreliable), it gets a ranked shortlist. An optional auto-improvement loop then lets the agent iterate: run the shortlisted tests, observe failures, patch repeat — up to N rounds.

Repository Source Code
        |
        v
   AST Parser
   /         \
Functions   Test Files
   \         /
    Bipartite
      Graph
        |
   Proposed Diff
        |
        v
  Impact Scorer
  (graph walk from
   changed nodes)
        |
        v
  Top-K Tests  <-- fed as context to agent
        |
        v
  Agent patches + verifies
        |
   [auto-loop?]
   yes |    | no
       v    v
   iterate  done

Think of it like a building’s electrical panel. When you’re rewiring one circuit, a good electrician doesn’t test every outlet in the building — they look at the panel diagram, trace which rooms share that circuit, and test only those outlets. TDAD is the panel diagram. The AST graph is the wiring map; the impact scorer is the electrician reading which breakers are downstream of your change; the top-K list is the clipboard of rooms to check. Without the diagram, you’re either testing everything (slow) or guessing (dangerous). With it, you test exactly what’s at risk.

Key Concepts

• AST-based code-test graph: An Abstract Syntax Tree is just a structured representation of source code — instead of raw text, you get a tree where each node is a meaningful unit (function call, class definition, import statement). TDAD uses this to find edges between production code and tests without running anything. Concretely: if test_login.py imports auth.validate_token and calls it, there’s an edge from validate_token to test_login. Change validate_token, and that edge lights up.

• Weighted impact analysis: Not all edges are equal. A test that directly calls your changed function is more at risk than a test that calls a wrapper that calls your function. TDAD assigns weights based on call depth and coupling — closer in the graph means higher impact score. It’s the same intuition as PageRank but for risk propagation instead of link authority.

• GraphRAG workflow: RAG (Retrieval-Augmented Generation) normally retrieves text chunks from a vector store. GraphRAG does retrieval over a graph structure instead, letting you exploit relational structure (A calls B which calls C) rather than just semantic similarity. Here it means the agent’s context window gets populated with structurally relevant tests, not just textually similar ones.

Framework Shift

Before (mainstream approach):        After (this paper):

Agent gets issue description          Agent gets issue description
         |                                     |
         v                                     v
   Agent writes fix                      AST graph built
         |                               from repo
         v                                |
   Run full test suite                         v
   (or skip entirely)                  Diff -> impact scores
         |                                |
         v                                     v
   Hope nothing broke                  Top-K tests surfaced
                               |
                                               v
                         Agent writes fix
                                         + verifies shortlist
                               |
                                               v
                                         Targeted confidence

From reactive to proactive, the core shift is: stop treating the test suite as a post-hoc oracle and start treating it as a navigable map that tells you where to look before you commit.

Expert Assessment

Problem choice: This is a real gap. The regression blind spot in SWE-bench is well-known but under-addressed — most teams just accept it as noise. Framing it as a first-class problem and building a benchmark methodology around it is genuinely useful. The field needed someone to just do this.

Method maturity: The AST graph + impact scoring idea is not exotic — it’s essentially a lightweight version of what static analysis tools like pytest-cov or dependency trackers already do. The novelty is in the integration: wiring it into an agent’s context window as a GraphRAG skill rather than a post-hoc check. That’s a reasonable engineering insight, not a theoretical breakthrough. The auto-improvement loop is the more interesting piece, but it’s also the least rigorously evaluated (10 instances is a toy sample).

Experimental integrity: The numbers are promising but the setup raises questions. 100 instances for the main experiment is thin for SWE-bench claims. The 60% resolution on10 instances for the auto-loop is essentially anecdote-level — you can’t draw conclusions from that. The comparison between TDD prompting and GraphRAG is the most interesting result, but the TDD baseline isn’t described in enough detail to know if it was a fair prompt. The finding that TDD prompting *increased regressions to 9.94% is surprising and deserves more investigation than it gets.

Writing quality: The paper is readable and the open-source commitment is a genuine plus. The weakest section is the experimental setup — it reads like a lab notebook rather than a controlled study. If the author rewrote the evaluation section with clearer ablations (what exactly changes between conditions, what’s held constant), the paper’s credibility would jump significantly. Right now you have to take several comparisons on faith.

Verdict: weak accept — the problem framing and the TDD-vs-context finding are worth publishing, but the experimental rigor needs another pass before the numbers can be trusted at face value.

Takeaways

The most transferable idea here isn’t the full TDAD system — it’s the design principle the paper accidentally discovers: for smaller models, giving them the right context beats giving them the right procedure. If you’re building agent workflows with sub-70B models, this is worth internalizing. Don’t write elaborate chain-of-thought prompts telling the model how to think; instead, invest in retrieval infrastructure that surfaces what the model needs to know.

The AST-to-graph pipeline is also directly stealable. If you’re building any kind of code review tool, change impact estimator, or CI triage system, the bipartite code-test graph with weighted edges is a clean primitive that’s cheap to build and immediately useful. You don’t need the full agent loop to get value from it.

Finally, the regression-aware benchmark methodology is something the community should adopt. If you’re evaluating any coding agent, tracking what percentage of previously-passing tests break is a one-line addition to your eval harness and it tells you something resolution rate completely hides.