RESTestBench: When LLMs Generate API Tests from Requirements, How Do We Know They Actually Work?

Paper: 2604.25862 Authors: Leon Kogler, Stefan Hangler, Maximilian Ehrhart, Benedikt Dornauer, Roland Wuersching, Peter Schrammel Categories: cs.SE, cs.AI

The Gap

Traditional REST API testing tools measure success through code coverage and crash detection. But when LLMs generate tests from natural language requirements, these metrics miss the point entirely — a test can cover 90% of the code and still fail to validate what the requirement actually asked for. Prior work like Bartocci et al.’s property-based mutation testing exists, but no standardized benchmark lets us reproducibly measure whether generated tests catch requirement violations.

Problem: LLM test generators lack requirement-focused evaluation
   |
   v
Assumption: Traditional metrics (coverage, crashes) are weak proxies
            for requirement validation effectiveness
   |
   v
Method: RESTestBench = 3 REST services + manual requirements
        + requirement-based mutation testing metric
   |
   v
Evidence: Test effectiveness drops 20-40% when generator sees
          mutated code, especially for vague requirements
   |
   v
Conclusion: Interacting with actual SUT during generation may be
            unnecessary when requirements are detailed

The Increment

One sentence: Before this paper, we evaluated LLM-generated API tests by how much code they touched; after, we can measure whether they actually catch violations of the specific requirements they were supposed to validate.

Core Mechanism

RESTestBench consists of three REST services (a pet store, a library system, and a task manager) paired with manually verified natural language requirements in two variants: precise (detailed, unambiguous) and vague (underspecified, open to interpretation). For each requirement, the benchmark includes a set of mutants — modified versions of the service that violate that specific requirement while remaining syntactically valid.

The evaluation flow works like this: an LLM generates test cases from a requirement, those tests run against both the correct implementation and the requirement-specific mutants, and effectiveness is measured by the mutation score — the percentage of requirement-violating mutants the generated tests successfully detect. The benchmark tests two generation strategies: non-refinement (generate tests from requirements alone) and refinement (generate, run against the SUT, observe responses, refine tests iteratively).

Requirement (precise/vague)
        |
        v
   LLM Generator  <--[optional: observe SUT responses]
        |
        v
   Test Cases
        |
        +---> Run on correct implementation
        |
        +---> Run on mutants (each violates requirement)
              |
              v
         Mutation Score = mutants killed / total mutants

Think of this like a driving test for self-driving cars. Traditional metrics are like measuring “did the car move forward?” or “did it crash?” But what you really want to know is: when the requirement says “stop at red lights,” does the car actually stop at red lights? RESTestBench creates a controlled environment with traffic lights (requirements), correct driving behavior (the original service), and specific violations (mutants that run red lights), then measures whether the generated tests catch those violations. The refinement approach is like letting the car observe traffic patterns before finalizing its driving algorithm — but the paper finds that if the traffic it observes is already chaotic (mutated), the car learns bad habits.

Key Concepts

• Requirement-based mutation testing: Traditional mutation testing randomly breaks code to see if tests catch it. Requirement-based mutation testing surgically breaks code in ways that violate a specific requirement. If a requirement says “users can only delete their own posts,” a mutant might allow deleting anyone’s posts. A good test for that requirement should fail on the mutant but pass on the correct code. This measures whether the test validates the requirement, not just whether it achieves coverage.

• Refinement-based generation: Instead of generating tests from requirements in one shot, the LLM generates initial tests, runs them against the actual running service, observes the responses (status codes, payloads, errors), and iteratively refines the tests based on what it learns. The intuition is that seeing real behavior helps the LLM write more accurate tests. But the paper’s key finding is that if the service you’re observing is buggy, you learn buggy behavior — garbage in, garbage out.

Framework Shift

Before (mainstream approach):        After (this paper):

Requirements                         Requirements (precise/vague)
     |                                    |
     v                                    v
LLM generates tests                  LLM generates tests
     |                                    |
     v                                    +---> Mutants (violate req)
Run tests on SUT                          |
     |                                    v
Measure: coverage, crashes           Measure: mutation score
                                     (% of req violations caught)

Focus: Did tests execute code?       Focus: Did tests validate
                                             the requirement?

From measuring test execution breadth to measuring requirement validation depth, the core shift is from proxy metrics to direct measurement of what we actually care about.

Expert Assessment

Problem choice: This is a real gap. The field has been generating tests from requirements for years without a principled way to evaluate whether those tests actually validate the requirements. The problem sits at the intersection of two trends: LLMs as code generators and specification-based testing. It’s timely and well-motivated.

Method maturity: The benchmark design is solid — manually verified requirements, controlled mutants, reproducible setup. The requirement-based mutation approach extends prior work (Bartocci et al.) appropriately. However, the paper doesn’t deeply explore why refinement fails on mutated code. Is it prompt design? Model limitations? The lack of ablation studies (e.g., what if we show the LLM only correct responses, or mix correct and mutated?) leaves mechanistic questions unanswered.

Experimental integrity: Baselines are fair — they compare non-refinement vs refinement across multiple LLMs (GPT-4, Claude, etc.). The drop in effectiveness when interacting with mutated code is striking (20-40% in some cases), and the finding that vague requirements amplify this effect is believable. One concern: the three services are relatively simple (pet store, library, task manager). Do results generalize to complex enterprise APIs with hundreds of endpoints and intricate business logic?

Writing quality: The paper front-loads motivation well but buries key results in tables. Section 5 (results) would benefit from visual summaries — line plots showing mutation score vs requirement precision, or heatmaps of LLM performance across services. The discussion of why refinement backfires is too brief; this deserves a dedicated subsection with examples of how mutated responses mislead the generator.

Verdict: weak accept — Solid benchmark contribution with surprising findings, but lacks depth in explaining the mechanisms behind refinement failure and could use broader evaluation on complex APIs.

Takeaways

If you’re building LLM-based test generators, steal the requirement-based mutation testing metric. It’s a direct way to measure whether your tests validate what they’re supposed to, not just whether they run. The paper’s finding about refinement is a cautionary tale: letting your generator interact with the system under test sounds smart, but if that system is buggy, you’re training on bad data. This suggests a design principle: if you’re going to refine based on observed behavior, validate the correctness of that behavior first, or at least use multiple implementations as cross-checks. For practitioners in other domains (not just API testing), the broader lesson is: when evaluating generative models, measure the thing you care about directly, not proxies. Coverage is easy to measure but often meaningless; requirement validation is hard to measure but actually matters.