Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact

Paper: 2603.00883 Authors: Michael Hardy, Yunsung Kim Categories: cs.LG, cs.AI, cs.CY, stat.AP

The Gap

The AI community has been chasing benchmark scores like they’re the finish line. MMLU, HumanEval, GSM8K — models keep getting better at these standardized tests. The implicit assumption: if a model aces benchmarks, it’ll perform well on real-world tasks in that domain. But here’s the problem nobody was rigorously measuring: benchmark performance is a proxy metric, and we’ve been treating it like ground truth.

Prior work evaluated LLMs on educational tasks using either synthetic datasets or correlation with other AI benchmarks. What was missing? Direct measurement of alignment between model behavior and actual downstream impact — specifically, whether LLM judgments about teaching quality correlate with what expert educators value and, critically, whether they predict student learning outcomes. The field had validation theater but no validation substance.

The Increment

Before: We assumed models good at benchmarks would be good at real tasks. After: We have quantitative proof that benchmark excellence can be orthogonal or even negatively correlated with real-world impact.

The authors designed a measurement framework that treats alignment as a three-way relationship. Think of it like tuning a musical instrument: you need a reference pitch (expert human judgments on teaching quality), the instrument you’re tuning (LLM outputs), and the actual sound quality you care about (student learning outcomes). Most prior work only checked if the LLM matched the reference pitch. This paper checks if matching that pitch actually produces better music.

They evaluated leading foundation models on out-of-distribution educational tasks — rating teaching episodes, predicting learning gains. The core finding is structural: LLMs correlate more strongly with each other than with human experts, revealing a shared bias space inherited from pretraining. When you ensemble models (either by majority vote or weighted by benchmark performance), you’re not averaging out noise — you’re amplifying the shared misalignment. It’s like asking a committee of people who all learned from the same textbook: consensus doesn’t equal correctness when the textbook itself is biased.

The measurement apparatus itself is the contribution. They decompose misalignment error into model-specific and shared components, finding that ~50% of the error variance is common across all foundation models. This suggests the problem isn’t in fine-tuning or prompting strategies — it’s baked into the pretraining corpus and objectives.

Key Concepts

Out-of-Distribution (OOD) Tasks in Context

When we say “out-of-distribution,” we usually mean data the model hasn’t seen. But there’s a subtler meaning here: tasks where the objective function differs from pretraining. LLMs are trained to predict next tokens on internet text. That objective correlates with many useful capabilities, but it’s not the same as “judge teaching quality in ways that predict student learning.”

Concrete example: Imagine a teaching video where the instructor makes a common pedagogical error — say, introducing a complex concept without scaffolding. An LLM might rate this highly if the explanation is articulate and uses sophisticated vocabulary (features correlated with “good text” in pretraining data). But expert educators know this approach confuses students. The learning outcome data proves the experts right. The LLM’s notion of “quality” is misaligned not because it’s stupid, but because it’s optimizing a different objective.

Shared Bias Space

The paper measures that different foundation models make correlated errors. This isn’t random — it’s structural. All these models were pretrained on overlapping internet corpora with similar objectives (next-token prediction, sometimes with RLHF on similar feedback datasets).

Think of it like this: if you train ten different students using the same textbook that has a subtle error on page 47, all ten students will likely make the same mistake on problems related to that concept. The error isn’t in the students’ individual reasoning — it’s in the shared knowledge source. The authors quantify this: 50% of misalignment variance is shared. That’s huge. It means you can’t fix this by switching models or ensembling — you need to change the training diet.

Negative Alignment with Learning Outcomes

This is the most alarming finding. It’s not just that LLMs fail to predict learning outcomes — in some cases, their judgments are negatively correlated. When the model says “this is good teaching,” students actually learn less.

Why? The hypothesis: LLMs latch onto surface features that correlate with “impressive text” but anti-correlate with effective pedagogy. Verbose explanations, technical jargon, complex sentence structures — these might score high in the model’s latent representation of quality, but they’re pedagogical red flags. Expert teachers know that clarity, simplicity, and student-centered pacing matter more. The model has knowledge (it can generate articulate text about teaching) but lacks wisdom (understanding what actually helps students learn).

Expert Assessment

Problem significance: This is critical. LLMs are already being deployed in educational contexts — tutoring systems, teacher training feedback, curriculum design. If these systems are optimizing for metrics that anti-correlate with learning, we’re not just wasting resources, we’re actively harming students. The affected community is massive: every student and educator in systems adopting AI tools.

Method maturity: This is a measurement study, not a solution. The authors provide a diagnostic framework but don’t offer a fix. That’s appropriate for the scope, but it means this is early-stage science. The methods for measuring alignment are sound, but translating these insights into better training procedures is left as future work. One limitation they don’t emphasize enough: their educational tasks are specific (K-12 teaching episodes). Generalization to other domains is plausible but unproven.

Experimental rigor: The baselines are fair — they compare multiple leading foundation models and use real expert annotations plus learning outcome data. The datasets are small by ML standards (inherent to getting ground-truth learning data), which limits statistical power. A red flag: the paper doesn’t deeply explore whether the misalignment is task-specific or a general phenomenon. Are LLMs bad at all “impact prediction” tasks, or specifically educational ones? More domain diversity would strengthen the claims.

Verdict: Weak accept — important negative result with solid measurement methodology, but limited scope and no proposed solutions.

Takeaways

Proxy metrics can be worse than useless: If your proxy anti-correlates with your true objective, optimizing it actively harms performance. This applies beyond education — any domain where you’re using AI benchmarks as a stand-in for real-world impact should be measuring alignment, not assuming it.

Ensemble methods amplify shared biases: When models share a bias (from common pretraining), ensembling doesn’t help — it makes things worse. This is counterintuitive because ensembles usually improve robustness. The lesson: diversity in ensembles requires diversity in training, not just architecture.

Measure downstream impact, not intermediate proxies: The paper’s core methodological contribution is the three-way alignment check (model vs. experts vs. outcomes). This framework is portable. If you’re deploying AI in healthcare, don’t just check if it agrees with doctors — check if it predicts patient outcomes. If you’re using AI for hiring, don’t just validate against recruiter judgments — track actual job performance.

Pretraining creates a shared reality distortion field: 50% shared error variance means the problem is upstream. Fine-tuning and prompting are band-aids. If you want models aligned with domain-specific objectives that differ from “predict internet text,” you need domain-specific pretraining or fundamentally different training objectives.