OpAI-Bench: Progressive Human-AI Co-Editing Reveals Non-Monotonic Detection Patterns

Paper: 2606.06481 Authors: Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tianjun Yao, Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Hao Li Categories: cs.CL, cs.AI, cs.LG

The Gap

Existing AI-text detection benchmarks treat documents as binary: fully human or fully AI. They test detectors on static final outputs. But real-world writing increasingly involves iterative human-AI collaboration—a writer drafts, asks AI to polish a paragraph, revises AI suggestions, asks AI to expand another section. This progressive co-editing process is invisible in current benchmarks.

The problem: we don’t know when AI authorship becomes detectable during revision workflows, or whether detectability increases monotonically with AI contribution. Prior work (GPTZero, DetectGPT, watermarking methods) assumes more AI content = easier detection. No one has traced authorship signals through the messy middle of the editing process.

Problem: Binary benchmarks miss revision dynamics
   |
   v
Assumption: AI detectability varies non-monotonically 
            through editing operations
   |
   v
Method: OpAI-Bench constructs 9 revision versions per doc
        with controlled AI coverage + operation types
   |
   v
Evidence: Mixed-authorship versions (30-70% AI) are 
          harder to detect than endpoints
   |
   v
Conclusion: Detection depends on operation type and 
            cumulative history, not just AI proportion

The Increment

One sentence: Before this paper, we evaluated detectors on all-or-nothing outputs; after, we can trace how detectability emerges and sometimes disappears through realistic editing sequences.

Core Mechanism

OpAI-Bench starts with human-written documents and generates nine progressively AI-edited versions for each sample. The progression is controlled by two parameters: AI coverage (how much of the document has been touched) and edit operation type (paraphrasing, expansion, summarization, polishing, or insertion).

Each version preserves complete authorship provenance at four granularities: document-level (overall AI percentage), sentence-level (which sentences were AI-edited), token-level (which tokens came from AI), and span-level (continuous AI-edited regions). The benchmark includes 17 detectors operating at different granularities: 8 document-level (GPTZero, DetectGPT, etc.), 7 sentence-level, and 2 fine-grained token/span-level.

The construction process uses version control semantics. Each revision step applies one operation to a subset of sentences, tracking the edit history like Git tracks code changes. This lets researchers ask: “At what point in the editing timeline did the detector start failing?” The benchmark covers four domains (academic writing, news, creative writing, technical documentation) with 1,000 samples each.

Human doc -> [v1: 10% AI] -> [v2: 20% AI] -> ... -> [v9: 90% AI]
               |                |                        |
            Paraphrase      Expansion                Polishing
               |                |                        |
         Token-level      Sentence-level          Document-level
         provenance        provenance              provenance
               |                |                        |
               +----------------+------------------------+
                                |
                          Test 17 detectors
                          across granularities

Think of OpAI-Bench as a crime scene investigation kit for text authorship. Existing benchmarks only show you the body (final document) and ask “human or AI?” OpAI-Bench gives you the security camera footage—nine snapshots from the moment the document was alive (fully human) through various stages of intervention to near-death (heavily AI-edited). Each snapshot comes with forensic tags marking exactly which parts were touched, when, and how.

The operation types are different weapons: paraphrasing leaves superficial marks but preserves structure (like changing clothes), expansion adds bulk in obvious places (like prosthetics), summarization removes material (like surgery), polishing smooths over rough edges (like makeup), and insertion adds entirely new content (like grafting). Each weapon leaves different traces. The detectors are forensic specialists—some look at overall blood spatter patterns (document-level), some examine individual wounds (sentence-level), and some use microscopes (token-level).

The non-monotonic detection pattern is the key finding: the victim isn’t always easiest to identify when most dead. Sometimes a body with 50% intervention looks more “natural” than one with 30% or 80% because the intervention operations cancel each other’s fingerprints in complex ways.

Key Concepts

• AI Coverage vs. Detectability: AI coverage is the percentage of a document’s sentences that have been edited by AI at least once. Intuition says 80% AI coverage should be easier to detect than 20%. OpAI-Bench shows this is false. A document with 50% coverage using mixed operations (some paraphrasing, some expansion) can have weaker AI signals than a 30% coverage document where all edits are concentrated insertions. Why? Because detection models look for statistical anomalies. Concentrated, uniform AI edits create clear anomalies. Scattered, diverse AI edits blend into the human baseline, especially when different operations introduce contradictory signals (paraphrasing makes text more formulaic, but polishing can restore naturalness). Concrete example: A research paper where AI expanded the introduction (adding wordy transitions) and summarized the conclusion (removing redundant phrases) shows weaker detection signals than a paper where AI only expanded the introduction, even if the first has higher overall AI coverage.

• Edit Operation Fingerprints: Not all AI edits leave the same traces. Paraphrasing changes word choice but preserves semantic structure, creating detectable patterns in word frequency distributions. Expansion adds transitional phrases and elaborations that often use high-perplexity tokens (unusual word sequences). Summarization compresses information, creating unusually information-dense sentences. Polishing adjusts grammar and flow, leaving minimal statistical traces. Insertion adds entirely new content with different stylistic properties. Detectors trained on full AI-generated text perform poorly on polishing (too subtle) and surprisingly well on insertion (discontinuity with surrounding human text). Think of it like document tampering: changing a few words (paraphrasing) is easier to catch with linguistic analysis than adjusting punctuation (polishing), but inserting a whole new paragraph creates stylistic discontinuity that’s obvious in context.

• Granularity Mismatch: Document-level detectors output one score for the entire text. Sentence-level detectors score each sentence independently. Token-level detectors mark individual words. When a document has mixed authorship (human intro, AI middle, human conclusion), document-level detectors average away the signal and often fail. Sentence-level detectors catch the AI middle section but miss how it integrates with human sections. Token-level detectors find exact boundaries but struggle with coherent AI-edited paragraphs where every token looks plausible. OpAI-Bench shows that granularity choice dramatically affects detection success—sentence-level detectors achieve 15-20% higher accuracy on mixed-authorship documents than document-level detectors. This matters because real-world use cases need to know “which part is AI?” not just “is there AI somewhere?”

Framework Shift

Before (binary endpoints):              After (revision provenance):

   Human -> [black box] -> AI           Human -> v1 -> v2 -> ... -> v9
     |                      |              |      |     |            |
  Test detector          Test           Track  Track Track       Track
     |                   detector        30%    50%   70%         90%
   Pass/Fail            Pass/Fail        edits  edits edits       edits
                                           |      |     |            |
                                        Compare detection curves
                                        across operation types
                                              
   One evaluation point              Nine evaluation points with
   per sample                        full authorship history

From binary classification to temporal tracing, the core shift is treating AI detection as a process property rather than a document property.

Expert Assessment

Problem choice: This is a real gap with practical consequences. Academic integrity tools, content platforms, and legal systems need to handle mixed authorship, not just classify endpoints. The problem sits at the intersection of detection research (technically mature but evaluation-limited) and real-world deployment (hungry for nuanced tools). The timing is right—AI writing assistants are ubiquitous, but detection research lags behind actual usage patterns.

Method maturity: The approach is straightforward—no algorithmic innovation, just careful benchmark construction. This is appropriate. The value is in the experimental design (controlled operations, multi-granularity labels, diverse domains) rather than novel techniques. One potential shortcoming: the AI coverage levels (10%, 20%, …, 90%) are evenly spaced, but real editing workflows might cluster at extremes (light touch-ups vs. heavy rewrites). A log-scale or user-study-informed distribution would be more ecologically valid.

Experimental integrity: The baselines are comprehensive (17 detectors, including recent methods like DNA-GPT and Binoculars). The non-monotonic detection pattern is convincing because it appears consistently across domains and detectors. However, the paper doesn’t discuss whether the AI edits were generated by a single model (likely GPT-4 or Claude) or diverse models. If one model, the findings might not generalize to detection of other AI assistants. The ablation on operation types is solid, but there’s no analysis of whether operation sequences matter (does paraphrase-then-expand differ from expand-then-paraphrase?).

Writing quality: The abstract and introduction are clear. The related work section is thin—it cites detection papers but doesn’t engage with human-AI co-creation research from HCI or the writing studies literature. The results section dumps numbers without enough interpretation. Section 4.3 on “cumulative editing effects” is the most interesting finding but gets only two paragraphs. If the authors rewrote this section to include case studies showing specific revision sequences that fool detectors, the paper would jump from “useful benchmark” to “must-read resource.”

Verdict: weak accept — Solid benchmark filling a real gap, but missed opportunities to deepen the analysis and connect to broader human-AI collaboration research.

Takeaways

For detector developers: Test on progressive editing sequences, not just endpoints. Your model might ace 0% and 100% AI content but fail catastrophically at 50% mixed authorship. Sentence-level granularity is more robust than document-level for real-world mixed-authorship scenarios.

For AI writing tool designers: Light polishing operations (grammar fixes, flow improvements) are nearly undetectable, while insertions and expansions leave strong traces. If you want to minimize detection risk, focus on refinement rather than generation.

For benchmark creators in other domains: The “version control for ML evaluation” idea transfers. Instead of testing models on static test sets, construct trajectories showing how performance degrades or improves as inputs are progressively modified. This applies to adversarial robustness, data augmentation, model editing, and any domain where inputs evolve through operations rather than appearing fully formed.

For researchers studying human-AI collaboration: The non-monotonic detectability curve suggests that “AI percentage” is a poor proxy for “AI-ness.” The structure of collaboration (which operations, in what sequence) matters more than the volume. This has implications for how we measure and regulate AI assistance in education, journalism, and creative work.