The Leaderboard Illusion: Why 93% of Top AV Perception Models Aren't Production-Ready

Paper: 2603.02194 Authors: Mateus Karvat, Bram Adams, Sidney Givigi Categories: cs.CV, cs.LG, cs.RO, cs.SE

The Gap

The AV perception field has a dirty secret: we’ve been optimizing for the wrong metric. Researchers chase leaderboard rankings on KITTI and NuScenes by squeezing out every 0.1% mAP improvement, but nobody’s checking if the code can actually survive contact with production. The field treats perception as a pure ML problem—train model, publish paper, move on. Meanwhile, automotive safety standards like ISO 26262 demand code that won’t kill people, and there’s zero overlap between “state-of-the-art accuracy” and “won’t crash when deployed.”

Prior work evaluated perception models exclusively on detection metrics (precision, recall, mAP). Code quality? Maintainability? Security vulnerabilities? Not even on the radar. This creates a chasm: research code optimized for benchmark performance versus production systems that need to pass safety certification, handle edge cases, and be maintained by teams who didn’t write the original research prototype.

Research Reality                Production Requirements
     |                                  |
     v                                  v
Leaderboard      <---GAP--->      Safety-Critical
Performance                        Deployable Code
     |                                  |
     v                                  v
mAP, Recall      <---???--->      Zero Critical Bugs
F1 Score                          Security Hardened
                                  Maintainable
     |                                  |
     +----------------------------------+
                      |
                      v
              This Paper's Path:
              Measure the gap
              Quantify the mess
              Provide guidelines

The Increment

One sentence: Before this paper, we assumed top leaderboard models were deployment-ready; after, we know 93% fail basic production criteria and exactly which five security issues cause 80% of problems.

Core Mechanism

The authors built a three-layer diagnostic pipeline to X-ray 178 repositories from KITTI and NuScenes leaderboards. First layer: Pylint scans for code errors—syntax issues, undefined variables, import failures, the stuff that breaks at runtime. Second layer: Bandit hunts security vulnerabilities—hardcoded credentials, unsafe deserialization, SQL injection risks. Third layer: Radon measures maintainability through cyclomatic complexity and code metrics.

They didn’t just run tools and dump numbers. They defined “production-ready” as zero critical errors AND zero high-severity security issues—a surprisingly low bar that 93% of repos still failed. Then they correlated code quality with development practices: does the repo use CI/CD? How many contributors? How active is maintenance? This revealed that CI/CD adoption strongly predicts better maintainability scores.

The security analysis went deeper: they found issue concentration—five vulnerability types account for 80% of all security problems. This Pareto distribution meant they could write targeted guidelines that address the bulk of issues without boiling the ocean.

Input: 178 Repos from Leaderboards
         |
         v
    +----+----+----+
    |    |    |    |
    v    v    v    v
 Pylint Bandit Radon Metadata
 (errors) (sec) (maint) (CI/CD)
    |    |    |    |
    +----+----+----+
         |
         v
  Aggregate Metrics
         |
    +----+----+
    |         |
    v         v
Production  Correlation
Readiness   Analysis
 (7.3%)    (CI/CD effect)
    |         |
    +----+----+
         |
         v
   Guidelines
  (Top 5 issues)

Think of this like a health screening for code. Most medical checkups don’t test for every possible disease—they run a standard panel that catches 80% of common issues. The authors designed their “standard panel” for AV perception code: three tools that together reveal whether code is healthy enough for production. Pylint is the blood test (basic functionality), Bandit is the cancer screening (hidden dangers), and Radon is the fitness assessment (can this code survive long-term maintenance?).

The key insight: they didn’t try to fix every repo. Instead, they diagnosed the population to find patterns. Just like epidemiology identifies that five risk factors cause most heart disease, they identified five security issues causing most vulnerabilities. Now you can vaccinate against the common threats instead of treating each patient individually.

Key Concepts

• Production Readiness (as defined here): Not “enterprise-grade architecture” or “follows all best practices”—just the bare minimum: code that runs without critical errors and doesn’t have obvious security holes. Think of it as the difference between “this car passes safety inspection” versus “this car is luxurious.” The paper sets the bar at safety inspection level, and 93% of research code still fails. A critical error might be calling a function that doesn’t exist, or a high-severity vulnerability might be hardcoding an API key in the source. These aren’t nitpicks—they’re showstoppers for deployment.

• Static Analysis: Reading code without running it, like a grammar checker for programming. Pylint catches errors by parsing the code and checking if variables are defined before use, if imports exist, if function calls match signatures. Bandit looks for patterns known to be dangerous—like using eval() on user input or storing passwords in plain text. The beauty: you get safety guarantees without needing test data or execution environments. The limitation: can’t catch logic bugs or runtime-only issues.

• Cyclomatic Complexity: Count how many different paths execution can take through your code. A function with no if-statements or loops has complexity 1 (one path). Add an if-statement, now there are 2 paths. Add a loop with a break condition, more paths. High complexity means more places for bugs to hide and harder maintenance. Radon measures this automatically. A complexity over 10 is a yellow flag; over 20 is a red flag. Most research code doesn’t track this at all.

Framework Shift

Before (mainstream approach):        After (this paper):

Research Cycle:                      Research + Deployment Cycle:
                                     
  Train Model                          Train Model
      |                                    |
      v                                    v
  Benchmark Test                       Benchmark Test
      |                                    |
      v                                    v
  Leaderboard                          Code Quality Scan
      |                                    |
      v                                +---+---+
  Publish Paper                       |       |
      |                               v       v
      v                            Pass?    Fail?
  [Done]                              |       |
                                      v       v
                                  Deploy  Fix+Retry
                                  
Evaluation:                          Evaluation:
mAP, F1, Recall                      mAP + Errors + Security + Maintainability

From “accuracy is everything” to “accuracy is table stakes, now let’s talk about whether this code can survive production,” the core shift is adding a quality gate between research and deployment.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has genuinely ignored code quality while chasing benchmark numbers. It sits at a critical inflection point—AV systems are moving from research to production, and the mismatch between research practices and safety standards is becoming a bottleneck. The timing is perfect: enough deployed systems exist to make this urgent, but not so many that the lesson has been learned the hard way.

Method maturity: This is straightforward empirical work—run existing tools, aggregate results, find patterns. No clever algorithmic insights, but that’s appropriate here. The value is in asking the question and doing the legwork, not inventing new analysis techniques. A simpler approach would be sampling fewer repos, but 178 is a reasonable population size for statistical validity. The CI/CD correlation analysis is a nice touch that goes beyond just reporting problems.

Experimental integrity: The baselines are fair—they’re using standard, well-established static analysis tools (Pylint, Bandit, Radon). The 7.3% production-readiness number is stark but defensible given their criteria. One concern: the definition of “production-ready” is binary (zero critical errors), which might be too harsh. A repo with one critical error that’s trivial to fix gets lumped with repos that are fundamentally broken. The paper would be stronger with a severity-weighted score. The security concentration finding (top 5 issues = 80%) is compelling and passes the smell test.

Writing quality: The paper front-loads the shocking statistic (7.3%) effectively, but the guidelines section feels rushed. They identify the top five security issues but don’t deeply explain why these specific patterns are so common in perception code. A case study walking through one repo’s journey from “research prototype” to “production-ready” would make the abstract findings concrete. The related work section is thin—more engagement with software engineering research on ML systems would strengthen positioning.

Verdict: weak accept — Addresses a real problem with solid empirical work, but the analysis could go deeper and the guidelines need more development to be truly actionable.

Takeaways

If you’re building any ML system that might see production, steal this three-tool diagnostic: Pylint for errors, Bandit for security, Radon for maintainability. Run them in CI/CD from day one, not as an afterthought. The paper’s finding that CI/CD correlates with better code quality isn’t surprising, but it’s evidence you can show to managers who think “we’ll clean it up later.”

The security concentration insight transfers directly: in any codebase, a small number of issue types cause most problems. Don’t try to fix everything—profile your vulnerabilities, find the top five, write guidelines for those, and you’ve addressed 80% of the risk. This Pareto principle applies beyond AV perception to any domain.

For researchers: if you’re releasing code with a paper, run these three tools before publishing. It takes an hour and prevents your repo from becoming a cautionary tale in the next empirical study. The gap between “runs on my machine” and “someone else can use this” is smaller than you think, but only if you measure it.

The broader lesson: leaderboard metrics are necessary but not sufficient. Whatever your field’s equivalent of mAP is, it’s probably hiding quality problems that matter for real-world use. This paper’s framework—define minimal production criteria, measure gap, identify concentrated issues, provide targeted guidelines—works for any research-to-production transition.