AEGIS: When AI-Generated Academic Images Fool Even the Best Detectors

Paper: 2604.28177 Authors: Bo Zhang, Tzu-Yen Ma, Zichen Tang, Junpeng Ding, Zirui Wang, Yizhuo Zhao, Peilin Gao, Zijie Xi, Zixin Ding, Haiyang Sun Categories: cs.CV, cs.CY

The Gap

Existing deepfake detection benchmarks focus on faces and natural images. They test binary classification (real vs fake) on consumer content. But academic fraud operates differently: researchers splice gel bands, duplicate microscopy regions, enhance contrast in Western blots. These manipulations are subtle, domain-specific, and often partial — not wholesale generation.

Current benchmarks like FaceForensics++ and DFDC miss this entirely. They don’t cover scientific imagery. They don’t test localization (where is the forgery?). They don’t evaluate reasoning (why is this suspicious?). And they don’t model the actual forgery strategies academics use: copy-paste, enhancement, inpainting, full generation.

The gap: no benchmark tests whether detectors can handle the forensic complexity of academic image fraud across diverse scientific domains with realistic manipulation patterns.

Problem: Academic fraud uses domain-specific, partial manipulations
         Current benchmarks test wholesale fakes on consumer images
                                |
                                v
Assumption: Forensic difficulty comes from domain complexity + 
            forgery diversity + multi-task evaluation
                                |
                                v
Method: Build AEGIS with 7 academic domains, 4 forgery types,
        25 generators, test detection + reasoning + localization
                                |
                                v
Evidence: GPT-5.1 at 48.80% overall, expert models IoU 30.09%,
          11/25 generators fool detectors >50% of the time
                                |
                                v
Conclusion: Academic forensics lags behind generative advances;
            no single model excels across all forensic dimensions

The Increment

One sentence: Before AEGIS, we had no way to measure whether AI detectors could handle academic image fraud; after AEGIS, we know they mostly can’t.

Core Mechanism

AEGIS constructs a three-layer evaluation architecture. First layer: domain coverage. It samples images from seven academic fields (biology, chemistry, physics, medicine, materials science, earth science, astronomy) with 39 fine-grained subtypes (gel electrophoresis, mass spectra, circuit diagrams, CT scans, etc.). This captures the intrinsic visual complexity of scientific imagery.

Second layer: forgery simulation. For each real image, AEGIS applies four manipulation strategies matching actual academic fraud patterns: copy-paste (duplicating regions), enhancement (adjusting brightness/contrast), inpainting (filling regions with AI), and full generation (creating entire fake images). It uses 25 different generative models spanning diffusion models, GANs, and multimodal generators to create forgeries.

Third layer: multi-dimensional evaluation. Instead of just asking “real or fake?”, AEGIS tests three forensic capabilities: binary detection (is it manipulated?), textual reasoning (describe the suspicious artifacts), and spatial localization (draw a mask around the forgery). Each task probes a different aspect of forensic understanding. The benchmark evaluates 25 MLLMs, 9 expert detectors, and 1 unified model across all three dimensions.

Real Academic Images (7 domains, 39 subtypes)
         |
         v
    +---+---+---+---+
    |   |   |   |   |
    v   v   v   v   v
  Copy Enhance Inpaint Generate  <-- 4 forgery strategies
  Paste                              25 models each
    |   |   |   |   |
    +---+---+---+---+
         |
         v
   Forged Dataset
         |
    +----+----+----+
    |    |    |    |
    v    v    v    v
 Detect Reason Localize  <-- 3 evaluation tasks
  (binary) (text) (mask)
    |    |    |    |
    +----+----+----+
         |
         v
  Performance Metrics
  (accuracy, IoU, etc.)

Think of AEGIS as a medical licensing exam for forensic AI. A real medical exam doesn’t just show you photos of skin rashes and ask “cancer or not?” It tests anatomy knowledge, diagnostic reasoning, and surgical planning across diverse cases. Similarly, AEGIS doesn’t just flash images and ask for verdicts. It tests whether models understand scientific imagery (domain knowledge), can articulate what looks wrong (reasoning), and can pinpoint the problem (localization). The exam is hard because it covers seven “medical specialties” (academic domains), uses realistic “patient presentations” (actual forgery strategies), and requires three types of competence (detection, reasoning, localization). Just as a doctor who only memorized rash photos would fail, a detector trained only on face swaps fails here.

Key Concepts

• Forensic Localization vs Detection: Detection answers “is this image manipulated?” — a binary yes/no. Localization answers “where exactly is the manipulation?” — drawing a pixel-level mask around the forged region. Detection is easier: you can spot something’s off without knowing where. Localization is harder: you need to understand the image structure well enough to say “this gel band is duplicated from that one” or “this microscopy patch has different noise statistics.” In AEGIS, expert models hit 79.54% detection accuracy but only 30.09% IoU (intersection over union) on localization. That’s like a doctor saying “you’re sick” but unable to point to the affected organ. The gap reveals that models recognize global inconsistencies but lack fine-grained spatial understanding.

• Domain-Specific Complexity: A face has consistent structure: two eyes, one nose, skin texture. A gel electrophoresis image has variable band patterns, lane boundaries, background gradients. A mass spectrum has peaks at arbitrary positions with varying intensities. Each scientific domain has its own visual grammar. AEGIS’s 39 subtypes capture this diversity. The result: even GPT-5.1 struggles, hitting 48.80% overall. Why? Because models trained on natural images haven’t learned the visual rules of scientific imagery. They don’t know that gel bands should align vertically, or that mass spec peaks should have smooth baselines. Domain-specific complexity means you can’t just fine-tune a face detector and call it done.

• Forgery Strategy Modeling: Academic fraud isn’t random. Researchers don’t generate entire fake papers. They copy-paste a gel band to fake a replicate. They enhance contrast to make a weak signal look strong. They inpaint a region to remove an inconvenient artifact. AEGIS models these four strategies because they match real-world fraud patterns. The payoff: 11 out of 25 generative models produce forgeries that fool detectors more than 50% of the time. This isn’t a bug — it’s a feature. It shows that as generative models improve, forensic detection falls behind. The benchmark’s value is in exposing this arms race, not in declaring a winner.

Framework Shift

Before (mainstream approach):        After (this paper):

Natural Images                       Academic Images
     |                                    |
     v                                    v
Binary Task                          Multi-Task Evaluation
(real/fake)                          (detect + reason + localize)
     |                                    |
     v                                    v
Single Metric                        Complementary Strengths
(accuracy)                           (MLLMs reason, experts detect)
     |                                    |
     v                                    v
Face/Object Focus                    Domain-Specific Complexity
(consumer content)                   (7 fields, 39 subtypes)

From monolithic evaluation to diagnostic profiling, the core shift is treating forensics as a multi-dimensional capability rather than a single accuracy number.

Expert Assessment

Problem choice: Real and urgent. Academic image fraud is a documented crisis (retraction rates climbing, high-profile cases in Nature/Science). Existing benchmarks genuinely don’t cover this. The gap isn’t manufactured — it’s a blind spot in the forensics community.

Method maturity: Solid engineering, not groundbreaking insight. The three-layer architecture (domains + forgeries + tasks) is straightforward. The value is in execution: curating 39 subtypes, implementing 4 forgery strategies across 25 models, evaluating 35 detectors. This is infrastructure work, not algorithmic innovation. That’s fine — benchmarks should be infrastructure.

Experimental integrity: Baselines are fair. They test leading MLLMs (GPT-5.1, Claude, Gemini) and expert detectors (NPR, PSCC-Net, TruFor). The 30.09% IoU for localization is believable — spatial grounding is hard. One concern: no ablation on forgery difficulty. Are all four strategies equally hard, or does copy-paste dominate the failure cases? The paper doesn’t break this down. Also, the 48.80% GPT-5.1 accuracy is striking, but we don’t see error analysis. What types of images trip it up?

Writing quality: The abstract oversells (“holistic benchmark”) when it’s really “academic image benchmark.” Section 3 (dataset construction) is dense with tables but light on design rationale. Why these 39 subtypes and not others? How were forgery parameters chosen? Section 4 (experiments) dumps numbers without narrative. A rewrite of Section 5 (analysis) that clusters failure modes and proposes forensic strategies would elevate the paper from “here’s a benchmark” to “here’s what we learned.”

Verdict: weak accept — Fills a real gap with solid execution, but lacks analytical depth beyond “detectors struggle.”

Takeaways

Steal the multi-task evaluation framework. If you’re building any kind of AI auditing system (not just forensics), test multiple capabilities (detection, reasoning, localization) rather than collapsing everything into accuracy. AEGIS shows that models have complementary strengths: MLLMs excel at textual reasoning (84.74%), expert detectors at binary classification (79.54%). A production system should ensemble them, not pick one.

Steal the forgery strategy taxonomy. The four types (copy-paste, enhancement, inpainting, generation) transfer to other fraud domains. Financial document fraud? Same strategies. Medical record tampering? Same strategies. Use this as a checklist when red-teaming your own systems.

Steal the domain-specific complexity insight. If your detector works on consumer images but fails on specialized content (medical, satellite, industrial), don’t just add more data. Audit whether your model understands the domain’s visual grammar. AEGIS’s 39 subtypes are a template for how to structure domain coverage.

Don’t steal the benchmark as a training set. AEGIS is for evaluation, not fine-tuning. If you train on it, you’re teaching models to pass the test, not to do forensics.