From Vibes to Metrics: Formalizing How Users Really Evaluate LLMs

Paper: 2604.14137 Authors: Itay Itzhak, Eliya Habba, Gabriel Stanovsky, Yonatan Belinkov Categories: cs.CL, cs.AI, cs.LG

The Gap

Existing LLM evaluation relies on standardized benchmarks (HumanEval, MMLU, etc.) that measure aggregate performance across generic tasks. But there’s a disconnect: users don’t pick models based on benchmark scores. They run informal tests on tasks that matter to their specific workflow—what the community calls “vibe-testing.” A developer might test code completion on their own codebase style, or a writer might check tone consistency for their genre. This practice is widespread but completely ad hoc: no one knows what makes vibe-testing work, whether it’s reproducible, or how to systematize it.

The gap isn’t that benchmarks are wrong—it’s that they measure the wrong thing. They answer “how good is this model on average?” when users actually care about “how good is this model for me?”

Problem: Benchmark scores =/= real-world usefulness
   |
   v
Observation: Users rely on informal "vibe-testing"
   |
   v
Gap: Vibe-testing is ad hoc, unstructured, not analyzable
   |
   v
Method: Formalize vibe-testing as personalized prompts + user-aware judging
   |
   v
Evidence: Personalization changes which model wins (coding benchmarks)
   |
   v
Conclusion: Formalized vibe-testing bridges benchmarks and experience

The Increment

One sentence: Before this paper, vibe-testing was folklore; after, it’s a formalized evaluation paradigm with explicit personalization axes.

Core Mechanism

The method has two stages. First, they studied how vibe-testing actually happens in the wild: surveyed 100+ users about their evaluation practices and collected 50+ model comparison reports from blogs/social media. From this empirical data, they identified the pattern: users personalize both what they test (prompt selection) and how they judge (evaluation criteria).

Second, they built a proof-of-concept pipeline that operationalizes this pattern. For prompt personalization, they generate test cases tailored to a user’s context—for coding, this means matching their typical libraries, coding style, and problem domains. For judgment personalization, they define user-aware criteria (e.g., “prefers concise code” vs “prefers verbose explanations”) and use an LLM judge that applies these preferences when comparing outputs.

Input: User context (coding style, preferences, domain)
   |
   v
[Prompt Generator] ---> Personalized test prompts
   |                     (match user's workflow)
   v
[Model A]  [Model B] ---> Generate outputs
   |           |
   v           v
[User-Aware Judge] ---> Comparison using personalized criteria
   |
   v
Output: Which model fits this user better

Think of it like restaurant recommendations. Generic reviews (benchmarks) tell you the average rating across all diners. But you don’t eat “on average”—you have preferences: spice tolerance, dietary restrictions, ambiance taste. Vibe-testing is like asking friends with similar tastes to try restaurants and report back. This paper formalizes that process: the prompt generator is like customizing the menu to your dietary profile, and the user-aware judge is like having a friend who knows you hate loud music evaluate the ambiance. The system doesn’t ask “is this restaurant good?” but “is this restaurant good for you?”

Key Concepts

• Vibe-testing: Informal, experience-based evaluation where users test models on tasks relevant to their own workflow rather than standardized benchmarks. It’s called “vibe” because it’s subjective and vibes-based—a developer might test if a model “gets” their coding style, or a writer checks if outputs “feel right” for their genre. The key insight: this isn’t sloppy evaluation, it’s a different evaluation paradigm optimizing for personal fit rather than aggregate performance.

• Personalization axes: The two dimensions along which vibe-testing varies per user. Prompt personalization means generating test cases that match the user’s actual use context (their libraries, their problem types, their data). Judgment personalization means applying evaluation criteria that reflect the user’s preferences (do they value brevity or thoroughness? correctness or creativity?). Most benchmarks fix both axes; vibe-testing varies both.

• User-aware judging: Using an LLM as a judge, but conditioning it on explicit user preferences. Instead of asking “which output is better?” you ask “which output is better for a user who prefers X, Y, Z?” This shifts evaluation from objective quality (which doesn’t exist for many tasks) to subjective fit. The judge becomes a proxy for the user’s own judgment, calibrated to their stated preferences.

Framework Shift

Before (standardized benchmarks):        After (formalized vibe-testing):

[Generic Test Suite]                     [User Context]
        |                                        |
        v                                        v
   All Models                            [Prompt Generator]
        |                                        |
        v                                        v
  [Fixed Judge]                          [Personalized Prompts]
        |                                        |
        v                                        v
  Aggregate Score                          All Models
                                                 |
                                                 v
                                          [User-Aware Judge]
                                                 |
                                                 v
                                          Personal Ranking

From one-size-fits-all measurement to context-dependent fit assessment, the core shift is treating evaluation as a personalization problem rather than a standardization problem.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The disconnect between benchmark scores and user satisfaction is well-documented in practitioner communities. The timing is right—as models converge in capability, personalized evaluation becomes the differentiator. It sits at the intersection of evaluation methodology and human-AI interaction, which is increasingly important.

Method maturity: The formalization is sound but the implementation is proof-of-concept level. Using an LLM judge for user-aware evaluation is clever but introduces circularity—you’re using a model to evaluate models. The prompt generation relies on heuristics (matching libraries, style) that could be more principled. A simpler approach might be letting users directly specify test cases and criteria, but that doesn’t scale. The trade-off is reasonable.

Experimental integrity: The experiments are on coding benchmarks only, which limits generalizability. The baselines are fair (standard benchmarks, generic prompts), but the paper doesn’t deeply probe when personalization helps vs hurts. There’s a risk of overfitting to user preferences—if I prefer verbose code, does the system just pick the most verbose model? The paper acknowledges this but doesn’t fully address it. The numbers are believable but not definitive.

Writing quality: The empirical grounding (survey + in-the-wild reports) is strong and well-presented. The formalization section is clear. But the experimental section feels rushed—more ablations would help (e.g., personalized prompts alone vs personalized judging alone). The related work undersells how much prior work exists on personalized evaluation in other domains (recommender systems, adaptive testing). Rewriting Section 4 with deeper analysis would elevate the paper.

Verdict: weak accept — Addresses a real problem with a reasonable formalization, but the execution is preliminary and the evaluation is narrow. It’s a good starting point for a research direction, not a definitive solution.

Takeaways

If you’re building evaluation systems, steal the two-axis personalization framework: customize what you test and how you judge. This applies beyond LLMs—any system where “average performance” misses the point (recommendation engines, content moderation, UI design). The specific technique of using an LLM judge with explicit preference conditioning is immediately usable for subjective evaluation tasks.

For practitioners: the paper validates that your informal testing isn’t sloppy—it’s a legitimate evaluation paradigm. The trick is making it systematic enough to be reproducible. Document your test cases and judgment criteria. If you’re comparing models, don’t just ask “which is better?” but “better for what kind of user?”

The deeper insight: evaluation is not a measurement problem, it’s a matching problem. Benchmarks measure capability; vibe-testing measures fit. Both are necessary, neither is sufficient.