DARE-bench: A Ground-Truth Benchmark for Data Science Instruction Following

Paper: 2602.24288 Authors: Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu, Zhewei Yao, Yuxiong He, Feng Yan Categories: cs.AI, cs.CL

The Gap

Current LLM benchmarks for data science suffer from two critical flaws. First, they rely on subjective human or model-based judges to evaluate outputs, making results non-reproducible and vulnerable to evaluator bias. Benchmarks like DS-1000 or MATH check only final answers, ignoring whether the model followed the correct process—like grading a math exam by answer key alone, missing whether the student used valid reasoning. Second, there’s a chicken-and-egg problem: we lack large-scale, accurately labeled training data for multi-step data science tasks, yet models need exactly this data to improve. Existing benchmarks like HumanEval focus on code correctness but don’t capture the full complexity of real data science workflows involving data exploration, feature engineering, model selection, and iterative refinement.

The Increment

Before: Benchmarks evaluate data science tasks with subjective judges and sparse training data. After: DARE-bench provides 6,300 Kaggle-derived tasks with verifiable ground truth for both objective evaluation and large-scale training.

Think of DARE-bench as a driving test with telemetry. Traditional benchmarks are like checking if you arrived at the destination (outcome-only evaluation). DARE-bench is like a driving instructor with a dashboard monitoring your speed, lane position, turn signals, and brake timing throughout the journey. The benchmark has three components that map to this analogy: (1) Task specification (the route and traffic rules)—clear instructions derived from real Kaggle competitions; (2) Process checkpoints (telemetry sensors)—intermediate verification points that check if you’re following correct procedures, not just reaching the end; (3) Ground truth labels (the instructor’s answer key)—objectively verifiable outcomes from actual Kaggle solutions, eliminating subjective judgment.

The authors extracted tasks from Kaggle competitions where ground truth is inherently available—competition leaderboards provide objective performance metrics. Each task includes the dataset, problem description, evaluation metric, and verifiable solution. This creates a closed-loop evaluation system: models generate code, execute it, and produce predictions that can be scored against held-out test sets using the competition’s own metric (accuracy, F1, RMSE, etc.).

Key Concepts

Process Fidelity vs. Outcome Accuracy

Most benchmarks ask: “Did you get the right answer?” DARE-bench asks: “Did you get there the right way?” Imagine two students solving $\int x^2 dx$. Student A writes $\frac{x^3}{3} + C$ immediately. Student B writes $\frac{x^3}{3}$ but forgets the constant. Traditional benchmarks might mark both wrong if they check only the final symbolic form. Process-aware evaluation would recognize that Student A understands integration while Student B made a minor notation error.

In data science, this distinction matters enormously. A model might achieve 85% accuracy by accidentally overfitting to test data leakage, or by correctly implementing cross-validation and feature engineering. DARE-bench tracks intermediate steps: Did the model split data properly? Did it handle missing values before training? Did it use appropriate evaluation metrics? This is “instruction fidelity”—following the recipe, not just producing something edible.

Verifiable Ground Truth

Why is “verifiable” important? Consider asking an LLM to “analyze customer churn.” A human judge might score the response based on vibes—does it sound insightful? A model-based judge (like GPT-4-as-a-judge) might prefer verbose explanations. Neither is reproducible. Verifiable ground truth means: run the model’s generated code, make predictions on a held-out test set, compute a metric (say, AUC), and compare to a threshold. No ambiguity. No drift in evaluation standards over time.

DARE-bench achieves this by sourcing tasks from Kaggle, where competitions already define success objectively. If the task is “predict house prices,” success is measured by RMSE on a hidden test set. The benchmark inherits this objectivity. It’s like using a standardized ruler instead of asking people to estimate lengths by eye.

Kaggle-Derived Task Diversity

Kaggle competitions span tabular data, time series, NLP, computer vision, and recommendation systems. By deriving 6,300 tasks from this ecosystem, DARE-bench covers a representative slice of real-world data science. This isn’t synthetic data or toy problems—these are tasks that practitioners actually cared enough about to compete on.

The diversity matters for generalization. A model that excels on tabular classification but fails on time series forecasting hasn’t truly mastered data science instruction following. DARE-bench’s breadth prevents overfitting to narrow task distributions, similar to how ImageNet’s 1,000 classes prevent vision models from memorizing a handful of categories.

Expert Assessment

Problem significance: This is a high-impact problem. The LLM-for-data-science space is exploding—tools like GitHub Copilot, Cursor, and specialized agents are being deployed widely. Yet we’re flying blind without rigorous benchmarks. The affected community includes ML practitioners, AI researchers, and enterprises investing in LLM-powered analytics tools. The lack of objective evaluation has real consequences: companies might deploy underperforming models, researchers can’t reliably compare methods, and progress is hard to measure.

Method maturity: This is deployment-ready as a benchmark, but the paper reveals the field itself is immature. Even GPT-4-mini achieves mediocre performance, suggesting current models aren’t ready for autonomous data science work. The benchmark’s design is solid—using Kaggle tasks is clever because ground truth comes for free. However, there’s a limitation the authors don’t emphasize enough: Kaggle tasks are competition-style, often with clean datasets and clear objectives. Real-world data science involves ambiguous requirements, messy data, and stakeholder negotiation—aspects DARE-bench doesn’t capture. It’s a controlled environment, like testing self-driving cars on closed tracks rather than chaotic city streets.

Another limitation: the benchmark focuses on single-task episodes. Real data science involves iterative refinement—you train a model, analyze errors, engineer new features, retrain. DARE-bench’s tasks are more atomic. The authors acknowledge this implicitly by providing “agentic tools,” but the evaluation doesn’t deeply probe multi-turn iterative workflows.

Experimental rigor: The baselines are fair and comprehensive—testing models from GPT-4 to smaller open-source options. The evaluation is objective by design (verifiable ground truth), which is the paper’s main strength. The Kaggle-derived dataset is representative of competition-style tasks, though as noted, it may not fully represent production data science.

One red flag: the paper shows massive improvements from fine-tuning (8x for RL on Qwen3-4B), which is impressive but raises questions. Is the benchmark leaking information? Are the training and test sets sufficiently separated? The authors claim proper splits, but such dramatic gains warrant scrutiny. It’s possible the tasks share structural similarities that make transfer learning unusually effective—not necessarily a flaw, but worth investigating.

Verdict: Weak accept — DARE-bench fills a real gap with a well-executed benchmark, but the paper oversells the generality (Kaggle tasks ≠ all data science) and doesn’t deeply probe why fine-tuning gains are so extreme.

Takeaways

Verifiable evaluation beats vibes: The shift from human/model judges to objective metrics is transferable to any domain where ground truth can be established. If you’re building benchmarks for code generation, theorem proving, or even creative tasks with measurable outcomes (e.g., “generate a recipe that uses these ingredients and stays under 500 calories”), prioritize verifiable checks over subjective scoring.

Process matters as much as outcomes: Instruction fidelity—tracking whether intermediate steps are correct—is underutilized in LLM evaluation. This applies beyond data science. In legal document generation, did the model cite relevant statutes? In medical diagnosis, did it consider differential diagnoses? Outcome-only evaluation misses these process failures.

Real-world task sources reduce synthetic bias: Sourcing tasks from Kaggle rather than hand-crafting them ensures the benchmark reflects actual practitioner needs. This principle applies elsewhere: use GitHub issues for software engineering benchmarks, Stack Overflow for debugging tasks, or customer support tickets for dialogue systems. Real data has a texture that synthetic tasks lack.

Fine-tuning on process-rich data yields outsized gains: The 8x improvement from RL fine-tuning suggests that models have latent capabilities unlocked by task-specific training. For practitioners, this means: if your LLM struggles on a domain, don’t just prompt-engineer—consider fine-tuning on high-quality, process-annotated examples. The gains may be larger than expected, especially for smaller models where the capability exists but isn’t surfaced by pretraining alone.