CREATE: A Benchmark for Testing Associative Creativity in Language Models

Paper: 2603.09970 Authors: Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li, Greg Durrett Categories: cs.CL

The Gap

Existing LLM benchmarks test knowledge retrieval, reasoning chains, and instruction following—but they miss a fundamental aspect of human intelligence: associative creativity. Prior work on creativity either focuses on divergent thinking tasks (generating many ideas) without measuring connection quality, or on constrained creative writing that’s hard to evaluate objectively. The Remote Associates Test (RAT) exists but only tests recognition of single connections, not generation of multiple diverse pathways. No benchmark systematically evaluates whether models can generate many distinct, meaningful connections between concepts—the core of hypothesis generation, scientific discovery, and creative problem-solving.

Problem: LLMs lack creativity benchmarks
   |
   v
Assumption: Creativity = associative reasoning
            (connecting distant concepts meaningfully)
   |
   v
Method: CREATE benchmark
        - Generate concept connection paths
        - Score: specificity + diversity + quantity
   |
   v
Evidence: Frontier models show variance
          Thinking models don't always win
   |
   v
Conclusion: Associative creativity is measurable
            and remains challenging for LLMs

The Increment

One sentence: Before this paper, we had no way to objectively measure whether LLMs can generate multiple creative connections between concepts; after it, we have a benchmark showing that even GPT-4 and Claude struggle to produce diverse, high-quality associative paths.

Core Mechanism

CREATE asks models to generate multiple “paths” connecting two concepts using their parametric knowledge. Each path is a sequence of intermediate concepts (e.g., “coffee → caffeine → alertness → productivity → work”). The benchmark scores paths on two dimensions: specificity (how distinctive and tight the connection is) and diversity (how different paths are from each other). Models receive higher scores for producing more paths that are both specific and diverse.

The evaluation uses a combination of automated metrics and human judgment. Specificity is measured by having annotators rate connection strength and distinctiveness. Diversity is computed using embedding-based similarity between paths—penalizing redundant connections. The final “creative utility” score combines these factors, rewarding models that explore different regions of the conceptual space rather than generating variations on the same theme.

Input: Concept A ----------> Concept B
         |                        ^
         v                        |
    Model generates paths:
    
    Path 1: A -> X1 -> Y1 -> B   (specific, novel)
    Path 2: A -> X2 -> Y2 -> B   (specific, diverse from Path 1)
    Path 3: A -> X1 -> Y3 -> B   (specific, but similar to Path 1)
    
         |
         v
    Scoring:
    - Specificity: How tight/distinctive each connection?
    - Diversity: How different are paths from each other?
    - Quantity: How many valid paths generated?
         |
         v
    Creative Utility = f(specificity, diversity, quantity)

Think of CREATE like asking a chef to create multiple recipes connecting two ingredients—say, chocolate and salmon. A creative chef doesn’t just make five variations of chocolate-salmon mousse (low diversity). Instead, they might propose: (1) chocolate mole sauce with grilled salmon, (2) salmon roe with dark chocolate ganache, (3) smoked salmon with cocoa-rubbed spices, (4) salmon tartare with cacao nibs, (5) chocolate-beet puree with poached salmon. Each path uses different intermediate “concepts” (mole spices, roe texture, smoking technique, raw preparation, earthy vegetables) to bridge the gap. The benchmark rewards chefs who explore different culinary territories (diversity) while maintaining coherent flavor logic (specificity), not those who generate many similar ideas.

Key Concepts

• Associative Reasoning: The cognitive ability to find meaningful connections between seemingly unrelated concepts. Unlike logical reasoning (A→B→C where each step follows strict rules), associative reasoning operates in a vast, loosely-structured knowledge space where many paths exist but only some are insightful. Example: Connecting “ocean” to “democracy” through “waves → collective movement → voting patterns” is associative—it’s not logically necessary, but it reveals a meaningful parallel. The challenge for LLMs is that this requires both broad knowledge access and the ability to evaluate which connections are distinctive versus trivial.

• Specificity vs. Diversity Trade-off: Specificity measures how tight and distinctive a single connection is (avoiding generic bridges like “both are things”). Diversity measures how different multiple connections are from each other (avoiding redundant exploration). The tension: models can easily generate many paths by making small variations (high quantity, low diversity) or generate one perfect path (high specificity, low quantity). Creative utility requires balancing both—like a scientist proposing multiple distinct hypotheses rather than variations of one idea. This is why the benchmark uses a multiplicative scoring function that penalizes either extreme.

• Parametric Knowledge Paths: Unlike retrieval-augmented systems that can look up connections, CREATE tests whether models can generate paths using only their training-encoded knowledge. A path like “coffee → Ethiopia → highlands → altitude → thin air → breathing → meditation” requires the model to traverse its internal knowledge graph without external search. This is harder than it sounds because the model must: (1) recall relevant facts at each step, (2) evaluate which facts lead toward the target concept, (3) avoid getting stuck in local loops, and (4) track which conceptual territories it has already explored to maintain diversity.

Framework Shift

Before (mainstream approach):        After (this paper):

Creativity = Open-ended generation   Creativity = Structured exploration
                                     
User: "Write something creative"    User: "Connect A to B (many ways)"
  |                                    |
  v                                    v
Model: [generates text]              Model: [generates paths]
  |                                    |
  v                                    v
Eval: Subjective human rating        Eval: Specificity + Diversity metrics
      (hard to scale)                      (objective, scalable)
      
Focus: Output quality                Focus: Search space coverage
       (single artifact)                    (multiple distinct solutions)

One sentence: From evaluating creative outputs subjectively to measuring creative search objectively, the core shift is treating creativity as navigating a knowledge space rather than generating novel text.

Expert Assessment

Problem choice: This is a real gap. Creativity benchmarks have been notoriously difficult because they conflate generation quality with creative thinking. By focusing specifically on associative reasoning—a well-defined cognitive process—the authors sidestep the “what is creativity?” debate while capturing something genuinely important. The task sits at an interesting intersection: simple enough to evaluate objectively, complex enough to challenge frontier models. Smart positioning.

Method maturity: The approach is elegant but not groundbreaking—it’s essentially structured divergent thinking with automated diversity metrics. The specificity scoring still requires human annotation (a bottleneck), and the diversity metric using embeddings is reasonable but crude (semantically different paths might have similar embeddings). The multiplicative scoring function feels somewhat arbitrary—why not other aggregation methods? That said, the simplicity is a feature: the benchmark is immediately usable and interpretable.

Experimental integrity: The baselines are fair and comprehensive (GPT-4, Claude, Gemini, plus thinking models like o1). The finding that thinking models don’t consistently outperform is interesting and well-documented. However, I’m skeptical about one thing: the paper doesn’t deeply explore whether models are actually doing associative reasoning or just pattern-matching common conceptual bridges from training data. A stronger analysis would test on concept pairs unlikely to co-occur in training corpora. The human evaluation is solid but limited in scale (understandable given cost).

Writing quality: The paper is clear and well-structured, but the related work section is thin—it doesn’t engage deeply with cognitive science literature on creativity or with prior work on knowledge graph reasoning. The results section could be tightened; there’s repetition between main results and ablations. If I were revising, I’d expand the analysis of failure modes: show us examples where models get stuck, where they generate trivial paths, where diversity collapses. The paper tells us models struggle but doesn’t give enough insight into why.

Verdict: weak accept — Solid contribution with a useful benchmark, but the evaluation methodology has room for improvement and the insights into model behavior are somewhat surface-level.

Takeaways

For benchmark designers: The “specificity + diversity + quantity” framework is generalizable. You can apply this to any task requiring multiple distinct solutions—code generation (multiple implementations), experimental design (multiple hypotheses), problem decomposition (multiple approaches). The key insight: don’t just count outputs, measure coverage of the solution space.

For prompt engineers: The paper shows that standard creative prompting techniques (temperature sampling, “think creatively” instructions) give marginal improvements. This suggests that associative creativity might require architectural changes, not just better prompts. If you need diverse outputs, explicitly track and penalize similarity to previous generations in your sampling strategy.

For model developers: The finding that thinking models don’t dominate is revealing. It suggests that chain-of-thought reasoning helps with logical tasks but doesn’t automatically improve exploration of conceptual spaces. This points to a specific capability gap: models need better mechanisms for maintaining diversity during generation, perhaps through explicit memory of explored regions or diversity-aware beam search.

Steal this: The evaluation framework itself. If you’re building any system that generates multiple solutions, adopt the specificity-diversity-quantity triad. It’s simple, interpretable, and captures what matters: not just generating many things, but generating many different good things.