GeoCodeBench: Why AI Still Can't Write PhD-Level 3D Vision Code

Paper: 2603.30038 Authors: Wenyi Li, Renkai Luo, Yue Yu, Huan-ang Gao, Mingju Gao, Li Yuan, Chaoyou Fu, Hao Zhao Categories: cs.CV

The Gap

AI coding assistants have conquered LeetCode and can scaffold web apps, but they crumble when asked to implement the geometric transformations and novel algorithms that drive 3D computer vision research. Existing benchmarks (HumanEval, MBPP) test generic programming; domain-specific ones (DS-1000) cover data science but skip the spatial reasoning and mathematical rigor of geometric vision. No one has systematically measured whether models can translate cutting-edge 3D vision papers into working code—the kind researchers write daily but can’t delegate to current AI.

Problem: AI codes well on generic tasks, fails on 3D geometric vision
   |
   v
Hypothesis: Lack of benchmarks for PhD-level 3D coding hides this gap
   |
   v
Method: Curate fill-in functions from recent papers + edge-case tests
   |
   v
Evidence: Best model (GPT-5) achieves only 36.6% pass rate
   |
   v
Conclusion: Current AI far from trustworthy scientific 3D coding

The Increment

One sentence: Before this paper, we assumed AI coding progress would naturally extend to scientific domains; after, we have quantitative proof that 3D geometric vision coding remains a frontier where even frontier models struggle.

Core Mechanism

GeoCodeBench extracts 164 function-level implementation tasks from official repositories of papers published at top venues (CVPR, ICCV, NeurIPS, etc.). A tool first proposes candidate functions, then human experts screen for core 3D geometric components—things like rotation matrix composition, ray-surface intersection, or novel view synthesis logic. Each selected function becomes a fill-in-the-blank problem: the model receives the paper text (with controllable context length), function signature, and docstring, then must produce the implementation.

The benchmark auto-generates diverse unit tests for each target, including edge cases (zero vectors, degenerate triangles, boundary conditions). Scoring is fully automatic and reproducible: code either passes all tests or doesn’t. Tasks organize into a two-level hierarchy—General 3D (geometric transforms, physics/optics formulas) and Research (novel algorithms, complex geometric routing)—letting you diagnose where models fail.

Paper repo --> Tool proposes functions --> Human screens for 3D core
                                                    |
                                                    v
                                         Selected function becomes task
                                                    |
                    +-------------------------------+
                    |
                    v
    [Paper text] + [Function signature] --> Model --> [Implementation]
                                                              |
                                                              v
                                                    Auto-generated tests
                                                              |
                                                              v
                                                        Pass / Fail

Think of GeoCodeBench as a flight simulator for research coding. Generic coding benchmarks are like learning to drive in a parking lot—controlled, flat, forgiving. This benchmark is the mountain pass in a snowstorm: you need spatial reasoning (where am I in 3D?), mathematical precision (one sign error crashes everything), and algorithmic creativity (the paper’s pseudocode is a sketch, not a blueprint). The simulator doesn’t just check if your car moves; it throws edge cases at you—what if the road vanishes? What if gravity reverses? Only implementations that handle degenerate geometry, numerical instability, and algorithmic corner cases pass. The two-level hierarchy is your license classification: General 3D is the standard license (can you handle basic transforms?), Research is the commercial license (can you implement something no one’s coded before?).

Key Concepts

• Fill-in-the-function task: Instead of “write a program from scratch,” you get a function signature, docstring, and surrounding context—your job is to complete the body. This mirrors real research workflows: you’re not building a codebase from zero, you’re implementing the one tricky geometric subroutine that makes or breaks the paper. It isolates coding ability from software engineering, focusing on algorithmic and mathematical correctness.

• Edge-case unit tests: Standard tests check happy paths (does rotation work for 45°?). Edge-case tests probe boundaries: What if the rotation angle is zero? What if the input vector has length zero? What if two triangles are coplanar? In 3D geometry, edge cases aren’t rare—they’re where bugs hide. Auto-generating these tests ensures models can’t just memorize common patterns; they must handle the degenerate cases that break naive implementations.

• Context ablation (Method-only vs Full-paper): The paper runs experiments feeding models different amounts of paper text—just the Method section vs the entire paper. Surprisingly, Method-only statistically outperforms full-paper inputs. Why? Long papers contain tangential discussions, related work, and experimental details that dilute the signal. Models struggle with long-context scientific comprehension: more text ≠ better understanding. This reveals a failure mode: current models can’t efficiently extract the algorithmic core from verbose academic writing.

Framework Shift

Before (generic coding benchmarks):     After (GeoCodeBench):

  Problem statement                       Research paper (10+ pages)
         |                                         |
         v                                         v
  Write full program                      Fill in core function
         |                                         |
         v                                         v
  Test on examples                        Test on edge cases
         |                                         |
         v                                         v
  Pass/Fail (syntax/logic)                Pass/Fail (geometric correctness)

  Focus: General programming              Focus: 3D spatial reasoning
         + algorithmic thinking                   + mathematical rigor
                                                  + novel algorithm impl.

From “can you code?” to “can you translate cutting-edge geometric vision research into correct, robust implementations?”—the core shift is from testing programming fluency to testing scientific coding under constraints of spatial reasoning and mathematical precision.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The AI coding hype cycle has focused on productivity tools (autocomplete, bug fixes) and generic benchmarks. But researchers in 3D vision, robotics, and graphics can’t use these tools for their core work—implementing novel geometric algorithms. The paper sits at the intersection of two trends: AI-assisted coding reaching saturation on easy benchmarks, and scientific domains needing specialized evaluation. Timely and well-motivated.

Method maturity: The curation process (tool proposes, human screens) is pragmatic but introduces subjectivity—what counts as “core 3D geometric”? The paper doesn’t deeply justify selection criteria. However, the two-level hierarchy (General vs Research) is clever: it lets you diagnose whether failures stem from basic 3D incompetence or research-specific challenges. The context ablation (Method-only vs Full-paper) is the paper’s sharpest insight—it’s a simple experiment that reveals a fundamental limitation in long-context scientific reasoning.

Experimental integrity: Baselines are fair—eight models spanning open/closed, small/large. The 36.6% pass rate for GPT-5 is sobering and credible (not cherry-picked). One concern: the paper doesn’t report inter-annotator agreement for human screening, so we can’t assess curation reliability. Also, “PhD-level” is marketing—some tasks might be undergrad-level geometric transforms. But the core claim (models struggle on 3D coding) holds up.

Writing quality: The abstract oversells (“research of our community would change substantially”)—benchmarks don’t change research, better models do. The Method section is dense; a worked example of one task (paper excerpt → function signature → test cases) would clarify. The context ablation result is buried in experiments; it deserves prominence—it’s the paper’s most surprising finding.

Verdict: Weak accept—solid contribution with a sharp insight (context ablation), but curation subjectivity and overselling weaken it. The benchmark will be useful, but the paper’s lasting impact is showing that “more paper text” hurts performance, which challenges assumptions about long-context scientific AI.

Takeaways

For benchmark designers: Context ablation should be standard practice. Don’t assume more input helps—test it. GeoCodeBench’s Method-only result suggests that for scientific papers, extracting the algorithmic core (Method section) beats dumping the full paper into context.

For model developers: 3D geometric coding requires a different skill mix than web dev or data science. Spatial reasoning and mathematical precision don’t emerge from scaling alone. If you want models that help researchers, you need training data and evaluation that emphasize geometric edge cases, not just happy-path examples.

For researchers: Don’t expect current AI to implement your novel 3D algorithms reliably. Use it for boilerplate, but verify geometric correctness yourself. The 36.6% pass rate means you’ll spend more time debugging AI output than writing from scratch—for now.

Transferable technique: The “fill-in-the-function” format + auto-generated edge-case tests is a template for domain-specific coding benchmarks. If you work in computational physics, bioinformatics, or numerical optimization, you can adapt this approach: curate core functions from recent papers, generate tests that probe domain-specific failure modes, measure how models perform. The insight that Method-only context beats full-paper context likely transfers to other scientific domains where papers are verbose and models struggle with long-context reasoning.