General365: When LLMs Excel at Math but Stumble on Everyday Reasoning

Paper: 2604.11778 Authors: Junlin Liu, Shengnan An, Shuang Zhou, Dan Ma, Shixiong Luo, Ying Xie, Yuan Zhang, Wenling Yuan, Yifan Zhou, Xiaoyu Li Categories: cs.CL, cs.AI

The Gap

Existing LLM benchmarks like GSM8K (math) and GPQA (physics) show near-perfect performance from top models—GPT-4 hits 95%+ on these specialized tasks. But here’s the puzzle: these same models struggle with everyday reasoning that requires no expert knowledge. Prior work assumed strong domain-specific reasoning would transfer to general contexts. It doesn’t.

The problem: we’ve been measuring reasoning ability through the lens of specialized domains, where success might come from memorized patterns rather than genuine reasoning flexibility. No benchmark explicitly decouples reasoning difficulty from domain expertise.

Problem: LLMs ace specialized benchmarks but fail real-world tasks
                    |
                    v
Assumption: Domain reasoning =/= general reasoning
            (expertise masks reasoning brittleness)
                    |
                    v
Method: General365 - K-12 knowledge only, high reasoning complexity
        (365 seed + 1095 variants across 8 categories)
                    |
                    v
Evidence: Top model 62.8% vs 95%+ on domain benchmarks
          (26 models tested, consistent gap)
                    |
                    v
Conclusion: Current LLM reasoning is domain-dependent,
            not genuinely general-purpose

The Increment

One sentence: Before this paper, we thought LLMs had strong reasoning because they solved hard math problems; after, we know they’re pattern-matching specialists who struggle when expertise can’t compensate for reasoning gaps.

Core Mechanism

General365 constructs problems where reasoning complexity is high but required knowledge is deliberately constrained to K-12 level. The benchmark has three layers: 365 manually crafted seed problems spanning 8 categories (logic puzzles, spatial reasoning, constraint satisfaction, etc.), each seed spawns 2 variants with modified parameters, and problems are designed with nested logical branches, complex constraints, and semantic interference.

The evaluation pipeline is straightforward: problems are presented in natural language, models generate free-form answers, and responses are judged by both exact match and LLM-as-judge (GPT-4) for semantic correctness. The key innovation isn’t in evaluation mechanics but in problem design—each task requires multi-step reasoning chains where you can’t shortcut through memorized domain patterns.

Think of General365 as a driving test for reasoning. Domain-specific benchmarks are like testing someone on a closed track where they’ve practiced the exact route 1000 times—they might drive perfectly but you don’t know if they can handle real roads. General365 is the actual driving test: familiar traffic rules (K-12 knowledge), but novel situations requiring genuine decision-making. The test has three components: the route (seed problems), weather variations (parameter variants), and unexpected obstacles (nested constraints, semantic interference). You pass only if you can reason through unfamiliar combinations of familiar elements. A model that memorized “turn left at the oak tree” fails when the tree is replaced with a mailbox—same reasoning required, different surface features.

Key Concepts

• Reasoning vs Knowledge Decoupling: Most benchmarks conflate these—solving a calculus problem requires both calculus knowledge AND reasoning ability. You can’t tell which one failed when the model gets it wrong. General365 fixes this by restricting knowledge to what a 12-year-old knows (basic arithmetic, simple geometry, common sense), then cranking up reasoning complexity. It’s like testing someone’s chess strategy using only pawns and kings—limited pieces, unlimited strategic depth. If they fail, it’s not because they don’t know how the pieces move.

• Semantic Interference: This is when surface-level language patterns mislead reasoning. Example: “All birds can fly. Penguins are birds. Can penguins fly?” The linguistic pattern suggests “yes” but reasoning says “no” (the premise is flawed). General365 deliberately includes problems where naive pattern-matching produces wrong answers. It’s the reasoning equivalent of optical illusions—your first instinct is wrong, you need deliberate analysis to get it right.

• Nested Logical Branches: Real-world reasoning rarely follows a single chain—you evaluate multiple conditions simultaneously, backtrack when constraints conflict, and maintain state across reasoning steps. A simple example: “If Alice is taller than Bob, and Bob is taller than Carol unless Carol wears heels, and today is Tuesday when Carol always wears heels, who is tallest?” You can’t solve this linearly; you need to branch, evaluate conditionals, and synthesize. Domain benchmarks often allow linear reasoning paths; General365 forces branching.

Framework Shift

Before (domain benchmarks):          After (General365):

Expert Knowledge                     K-12 Knowledge
      +                                    +
Simple Reasoning                     Complex Reasoning
      ||                                   ||
      vv                                   vv
  [Problem]                            [Problem]
      ||                                   ||
      vv                                   vv
  High Score                           Low Score
      ||                                   ||
      vv                                   vv
"Model reasons well"                 "Model pattern-matches"

Difficulty source: Knowledge         Difficulty source: Reasoning
Success metric: Domain accuracy      Success metric: Transfer ability

One sentence: From testing reasoning through expert domains (where knowledge and reasoning are entangled) to isolating reasoning by constraining knowledge (revealing that current LLMs reason weakly but memorize strongly).

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has been celebrating 95%+ scores on GSM8K while ignoring that these models fail at planning a dinner party with dietary constraints. The timing is perfect—we’re deploying LLMs for general-purpose tasks but measuring them on specialist exams. The gap sits at a critical juncture: between research benchmarks and real-world deployment.

Method maturity: Elegant in its simplicity. The core insight—decouple knowledge from reasoning—is obvious in hindsight but nobody did it systematically. The execution is solid: 365 seeds with 2 variants each provides decent coverage without being unwieldy. The 8 categories span different reasoning types (spatial, logical, constraint-based). One weakness: “K-12 knowledge” is fuzzy—what counts as common sense varies by culture and education system. They don’t rigorously define this boundary, which could lead to disputes about whether a problem is “fair.”

Experimental integrity: Testing 26 models is thorough. The 62.8% top score (vs 95%+ on domain benchmarks) is the headline finding and it’s stark enough to be believable. However, I’m skeptical of the LLM-as-judge evaluation—using GPT-4 to judge GPT-4’s answers creates circularity. They do include exact match as a sanity check, but the paper would be stronger with human evaluation on a subset. The variant problems (changing parameters) are clever for testing robustness, but 2 variants per seed feels thin—you could get lucky with parameter choices.

Writing quality: The paper is clear but conservative. The introduction spends too much time motivating why general reasoning matters (we get it). The problem categorization section is dry—they list 8 categories with brief descriptions but don’t explain the taxonomy’s design principles. The results section is where they should have dug deeper: why do models fail? What error patterns emerge? They give aggregate numbers but sparse failure analysis. Rewriting Section 4 (Results) with detailed error taxonomy and case studies would elevate this from “interesting benchmark” to “diagnostic tool.”

Verdict: Weak accept — The benchmark addresses a real gap and the results are eye-opening, but the paper feels like it stops at “look, models fail” without deeply investigating why or how to fix it.

Takeaways

For benchmark designers: The decoupling strategy is generalizable—pick any capability you want to measure, then systematically remove confounding factors. Want to test creativity? Constrain the domain to familiar concepts. Want to test robustness? Keep the task fixed, vary surface features.

For model developers: If your model scores 95% on domain benchmarks but 60% on General365, you’ve built a specialist, not a generalist. The gap suggests current training overindexes on pattern memorization. Practical fix: augment training data with reasoning chains that require branching logic and constraint satisfaction, not just domain knowledge application.

For practitioners: Don’t trust domain benchmark scores as proxies for general capability. If you’re deploying an LLM for open-ended tasks (customer support, planning, analysis), test it on General365-style problems first. The 62.8% ceiling means even top models will fail 1 in 3 reasoning tasks that a human would find straightforward.

Steal this: The variant generation strategy—take a seed problem, change parameters while preserving reasoning structure. This is cheap to implement and immediately reveals whether your model learned the reasoning pattern or memorized the specific instance.