ClassEval-Pro: Exposing the Class-Level Code Generation Gap

Paper: 2604.26923 Authors: Yeheng Chen, Chaoxiang Xie, Yuling Shi, Wenhao Zeng, Yongpan Wang, Hongyu Zhang, Xiaodong Gu Categories: cs.SE, cs.CL

The Gap

Existing code generation benchmarks cluster at two extremes: function-level tasks (HumanEval, MBPP) where models write isolated 10-20 line functions, and repository-level tasks (SWE-bench) where models edit existing codebases. Between these sits an underserved capability: compositional code creation—building a complete class from scratch with multiple interdependent methods, internal state, and cross-method coordination.

Function-level benchmarks don’t test whether models can manage state across methods or handle dependency chains. Repository-level benchmarks assume existing structure and test modification, not creation. The gap: can models architect and implement a self-contained, multi-method class where methods call each other and share state?

Prior class-level datasets (ClassEval) are small (100 tasks), manually curated (expensive to scale), and increasingly contaminated (pre-2024 data). This paper addresses all three: automated pipeline, 300 tasks across 11 domains, post-Jan-2025 GitHub code.

Problem: Function-level ----gap----> Repository-level
         (isolated logic)            (edit existing)
                    |
                    v
         Missing: Class-level composition
                    |
                    v
         Assumption: Multi-method coordination is the bottleneck
                    |
                    v
         Method: Automated pipeline + coverage validation + LLM judge
                    |
                    v
         Evidence: Best model 45.6% Pass@1, 17.7pt model gap
                   Logic errors 56.2%, dependency errors 38.0%
                    |
                    v
         Conclusion: Cross-method coordination is indeed the core challenge

The Increment

One sentence: Before this paper, we knew LLMs could write functions and edit repos but had no systematic evidence they struggle with class-level composition; after, we have a 300-task benchmark showing even frontier models fail 54% of the time, primarily on cross-method logic.

Core Mechanism

ClassEval-Pro constructs tasks through a three-stage automated pipeline. Stage 1 takes seed classes and enhances complexity by adding methods, introducing dependencies between methods, and expanding state management. Stage 2 performs cross-domain composition: it samples classes from different domains (e.g., data structures + algorithms) and merges them into hybrid classes that require coordinating logic from multiple domains. Stage 3 integrates real-world GitHub classes contributed after January 2025 to prevent contamination.

Every generated task passes through dual validation. First, an LLM Judge Ensemble (three models voting) verifies that the specification is clear, the reference solution is correct, and test cases cover edge cases. Second, automated test suite validation ensures over 90% line coverage and that tests actually exercise cross-method interactions, not just individual methods in isolation.

The benchmark evaluates models under five generation strategies: direct (one-shot class generation), step-by-step (method-by-method with reflection), bottom-up (dependencies first, then dependents), top-down (interface first, then implementation), and compositional (generate components separately, then integrate). Each strategy tests a different hypothesis about how to manage complexity.

Pipeline Flow:

Seed Classes --> [Stage 1: Complexity Enhancement] --> Enhanced Classes
                      - Add methods
                      - Introduce dependencies
                      - Expand state
                           |
                           v
                 [Stage 2: Cross-Domain Composition] --> Hybrid Classes
                      - Sample from multiple domains
                      - Merge logic patterns
                           |
                           v
                 [Stage 3: GitHub Integration] --> Final Task Set
                      - Post-Jan-2025 code
                      - Prevent contamination
                           |
                           v
                 [Dual Validation]
                      - LLM Judge Ensemble (3 models)
                      - Test Coverage >90%
                           |
                           v
                 300 Tasks x 5 Strategies x 5 Models --> Results

Think of this like building a cooking exam for chefs. Stage 1 takes simple recipes (boil pasta) and makes them complex (pasta carbonara with timing dependencies—eggs must go in after heat is off). Stage 2 creates fusion dishes (Italian-Japanese: miso carbonara) that require coordinating techniques from different cuisines. Stage 3 adds brand-new recipes from 2025 cookbooks so contestants can’t have memorized them.

The validation is like having three master chefs taste-test every exam question to confirm it’s fair, then checking that the answer key actually demonstrates all the required techniques (coverage). The five strategies are different approaches to cooking: direct (make the whole dish at once), step-by-step (one component at a time with tasting between), bottom-up (make the sauce before the pasta), top-down (plate presentation first, then fill in), compositional (prep all ingredients separately, then combine).

The key insight: class-level code isn’t just “more functions”—it’s about managing state that flows through method calls, handling dependencies where method A’s output feeds method B’s logic, and coordinating error handling across the call chain. The benchmark specifically targets these coordination challenges.

Key Concepts

• Cross-method coordination: When a class has methods that call each other or share mutable state, the correctness of one method depends on the behavior of others. For example, a BankAccount class where withdraw() must check the balance set by deposit(), and getBalance() must reflect all prior transactions. This isn’t just writing multiple functions—it’s ensuring they form a coherent system. The challenge: LLMs generate each method somewhat independently, so they often miss implicit contracts (e.g., withdraw() assumes deposit() initializes balance to zero, but the generated deposit() doesn’t). The paper finds 38% of failures stem from these broken dependencies.

• Compositional generation collapse: One strategy the paper tests is generating class components (methods, helper functions) separately, then integrating them. Intuition says this should work—divide and conquer. Reality: it achieves only 1.3% Pass@1 for weaker models. Why? Because when you generate methods in isolation, you lose the context of how they’ll interact. It’s like writing chapters of a novel separately without an outline—each chapter might be fine, but they don’t form a coherent story. The integration step tries to stitch together pieces that were never designed to fit, leading to mismatched assumptions about state, calling conventions, and error handling.

• LLM Judge Ensemble: Instead of one model deciding if a task is valid, three models vote. Each judges whether the specification is unambiguous, the reference solution is correct, and test cases are comprehensive. A task passes only if at least two models agree. This catches edge cases where a single model’s biases (e.g., overly permissive on ambiguous specs) would let bad tasks through. It’s like peer review: one reviewer might miss something, but three independent reviewers rarely all miss the same flaw. The paper shows this reduces invalid tasks from ~15% (single judge) to ~3% (ensemble).

Framework Shift

Before (mainstream approach):        After (this paper):

Function-level:                      Class-level composition:
  Input: "write fibonacci"             Input: "write LRU cache class"
  Output: one function                 Output: complete class with:
  Test: does it return                   - __init__, get, put methods
        correct numbers?                 - internal dict + doubly-linked list
                                         - cross-method state updates
         +                                      +
         |                                      |
Repository-level:                              |
  Input: "fix bug in line 47"                  |
  Output: edit existing code                   |
  Test: does the bug disappear?                |
                                               |
  [Gap: no test for building          [Tests: does get() reflect put()?
   multi-method classes from                   does eviction work?
   scratch]                                    do methods coordinate?]

One sentence: From testing isolated logic (functions) or modification skills (repo edits) to testing architectural coordination—can the model design and implement a system where parts depend on each other?

Expert Assessment

Problem choice: Real gap. The jump from function-level to repo-level is too large—practitioners need models that can scaffold new modules, not just tweak existing ones. Class-level sits at the sweet spot: complex enough to test coordination, scoped enough to evaluate automatically. The timing is right: function-level benchmarks are saturated (90%+ Pass@1), so the field needs harder targets.

Method maturity: The automated pipeline is clever—cross-domain composition is a genuinely novel way to generate hard tasks without manual curation. But the LLM Judge Ensemble feels like a patch over a deeper problem: if you need three models to validate tasks, maybe the generation process is too noisy. A tighter generation algorithm (e.g., constraint-based synthesis) might produce cleaner tasks without needing ensemble validation. The five strategies are well-chosen, though “compositional” seems like a strawman—no practitioner would generate methods in total isolation.

Experimental integrity: Baselines are fair—five frontier models, consistent prompting, multiple strategies. The 90% coverage threshold is rigorous. One concern: the paper doesn’t report inter-annotator agreement for the 500 manually labeled failures. If two annotators disagree on whether an error is “logic” vs “dependency,” the 56.2% / 38.0% split might be less meaningful. Also, the post-Jan-2025 GitHub code is good for contamination, but only 100 of 300 tasks use it—the other 200 are synthetic, so contamination risk remains for models trained on similar synthetic data.

Writing quality: The method section is dense—three stages, dual validation, five strategies all crammed into two pages. A diagram showing one example task flowing through the entire pipeline would clarify more than three paragraphs of description. The error analysis (Section 5.3) is the paper’s strongest part—concrete failure modes with examples—but it’s buried at the end. Leading with “here’s what models get wrong” would hook readers better than leading with “here’s our pipeline.”

Verdict: Weak accept — Addresses a real gap with a scalable method and rigorous evaluation, but the automated pipeline’s complexity raises questions about whether simpler approaches were explored, and the writing doesn’t do justice to the strong empirical findings.

Takeaways

For benchmark designers: Cross-domain composition (merging classes from different domains) is a cheap way to generate hard tasks. You don’t need more data—you need to combine existing data in ways that force models to coordinate disparate logic patterns.

For prompt engineers: Structured strategies (bottom-up, step-by-step) help weaker models significantly (+9.4 points) but barely help strong models (+1.2 points). If you’re using a frontier model, one-shot generation is often better than elaborate scaffolding. If you’re using a smaller model, invest in dependency-aware prompting.

For model developers: The 56.2% logic error rate suggests models struggle with maintaining invariants across method calls. Training on more class-level code won’t fix this—you need data where methods explicitly depend on each other, not just collections of independent methods that happen to live in the same class. Look for codebases with high cyclomatic complexity per class, not just high method count.

For researchers: The 17.7-point gap between strongest and weakest models is unusually large for a 2026 benchmark (most recent benchmarks show 5-10 point gaps). This suggests class-level composition is still a frontier capability, not a solved problem. There’s room for architectural innovations (e.g., explicit dependency graphs in the model’s reasoning) rather than just scaling.