LLMs Impact on Peer Review: A Fine-Grained Analysis

Paper: 2604.19578 Authors: Wenqing Wu, Chengzhi Zhang, et al. (Nanjing University, Anhui University) Categories: cs.CL, cs.AI

Background

With the rapid advancement of LLMs, the academic community has faced unprecedented disruptions, particularly in academic communication. The primary function of peer review is improving manuscript quality.

Key Questions

1. Are LLMs altering the core evaluative functions of peer review?
2. How do LLMs affect linguistic form, evaluative focus, and recommendation signals?
3. What is the impact on informativeness of paper decisions?

Methodology

Analyzed changes in peer review reports for academic articles following LLM emergence:

• Linguistic features: Length, word/sentence complexity
• Evaluation aspects: Automatically annotated review sentences
• LLM-assisted detection: MLE method to identify LLM-generated reviews

Key Findings

Following LLM emergence, peer review texts have become:

• Longer and more fluent
• Increased emphasis on summaries and surface-level clarity
• More standardized linguistic patterns

Particularly for reviewers with lower confidence scores.

Decline in Deep Evaluation

Attention to deeper evaluative dimensions has declined:

• Originality
• Replicability
• Nuanced critical reasoning

Takeaways

• LLM assistance affects not just form but substantive evaluation focus
• Need to preserve human expertise in assessing novelty and impact
• Quality over quantity: longer reviews aren’t necessarily better reviews