Knowing When Not to Answer: Abstention-Aware Scientific Reasoning

Paper: 2602.14189 Authors: Samir Abdaljalil, Erchin Serpedin, Hasan Kurban Categories: cs.CL, cs.AI

Abstract

This paper introduces an abstention-aware framework for scientific claim verification that recognizes a fundamental limitation in current LLM evaluations: the assumption that models must always provide definitive answers. The authors argue that in scientific contexts, incorrect conclusions can be more damaging than admitting uncertainty. Their framework decomposes scientific claims into minimal verifiable conditions, audits each against available evidence using natural language inference, and makes selective decisions to support, refute, or abstain. Evaluated across SciFact and PubMedQA benchmarks with six diverse language models, the results demonstrate that while raw accuracy varies modestly across architectures, confidence-based abstention significantly reduces risk at moderate coverage levels, highlighting that the key challenge in scientific reasoning is determining when evidence suffices to justify an answer.

Key Contributions

• Introduction of an abstention-aware verification framework that decomposes scientific claims into minimal conditions and selectively decides whether to answer or abstain
• Comprehensive evaluation across two complementary benchmarks (SciFact and PubMedQA) covering both closed-book and open-domain evidence settings
• Experimental analysis with six diverse language models spanning encoder-decoder, open-weight chat models, and proprietary APIs
• Demonstration that confidence-based abstention substantially reduces error risk even when absolute accuracy improvements are limited
• Evidence that model selection matters less than determining when available evidence is sufficient for reliable answers

Methodology and Framework Design

The proposed framework operates through a three-stage pipeline designed to handle scientific claim verification with explicit abstention capabilities. First, the system decomposes complex scientific claims into minimal, atomic conditions that can be independently verified. This decomposition is crucial because scientific claims often contain multiple assertions that require separate evidence evaluation.

Second, each decomposed condition undergoes an evidence audit using natural language inference (NLI). The NLI component assesses whether available evidence entails, contradicts, or is neutral with respect to each condition. This granular approach allows the system to identify which specific aspects of a claim are supported by evidence and which lack sufficient backing.

Third, the framework employs a selective decision mechanism that aggregates the condition-level assessments and determines whether to support the overall claim, refute it, or abstain from judgment. The abstention decision is guided by confidence thresholds, allowing the system to recognize when evidence is insufficient or contradictory. This design philosophy prioritizes reliability over coverage, acknowledging that in scientific contexts, withholding judgment is often more responsible than forcing a potentially incorrect conclusion.

Experimental Results and Analysis

The experiments reveal several critical insights about abstention-aware scientific reasoning. Across both SciFact and PubMedQA benchmarks, raw accuracy differences between models are relatively modest, typically varying by only 5-10 percentage points. This suggests that architectural choices alone do not dramatically impact baseline performance on scientific verification tasks.

However, the introduction of confidence-based abstention mechanisms reveals a different picture. When models are allowed to abstain on low-confidence predictions, error rates drop substantially at moderate coverage levels (70-85% coverage). For instance, at 80% coverage, models consistently achieve 15-25% lower error rates compared to forced-answer scenarios. This improvement is particularly pronounced in open-domain settings where evidence retrieval introduces additional uncertainty.

The results also demonstrate that abstention behavior varies significantly across model families. Encoder-decoder models tend to exhibit more conservative abstention patterns, while chat-optimized models show higher confidence even on ambiguous cases. Proprietary API models generally achieve better calibration between confidence scores and actual accuracy, suggesting that alignment training may improve epistemic awareness.

Interestingly, the framework’s effectiveness is relatively model-agnostic. Even smaller open-weight models benefit substantially from abstention mechanisms, achieving risk-coverage trade-offs comparable to larger proprietary models when allowed to abstain strategically. This finding has important practical implications for deploying scientific reasoning systems in resource-constrained environments.

Implications for Scientific AI Systems

This work challenges the prevailing evaluation paradigm in scientific NLP, which typically measures success solely through accuracy on forced-answer tasks. The authors demonstrate that this approach misses a critical dimension of reliability: knowing when not to answer. In scientific domains where incorrect information can propagate through literature and influence research directions, the cost of false positives often exceeds the cost of abstention.

The abstention-aware framework provides a more realistic model of how AI systems should interact with scientific knowledge. Rather than treating every claim as equally answerable, the system acknowledges the inherent limitations of available evidence and model capabilities. This epistemic humility is essential for building trustworthy scientific AI assistants.

The findings also have implications for model development priorities. Rather than focusing exclusively on improving raw accuracy through larger models or more training data, developers should invest in better confidence calibration and uncertainty quantification. The results suggest that even modest improvements in confidence estimation can yield substantial gains in practical reliability when combined with selective abstention.

Furthermore, the work highlights the importance of benchmark design. Future scientific reasoning benchmarks should incorporate abstention as a first-class evaluation dimension, measuring not just whether models can answer correctly, but whether they can recognize when they should not answer at all.

Takeaways

1. In scientific reasoning tasks, the ability to abstain from answering is as important as the ability to answer correctly, as unsupported conclusions can be more harmful than admitting uncertainty.

2. Confidence-based abstention reduces error risk by 15-25% at moderate coverage levels (70-85%), even when absolute accuracy improvements are limited.

3. Raw accuracy varies only modestly across different model architectures (5-10 percentage points), suggesting that model selection alone is not the primary determinant of scientific reasoning performance.

4. The key challenge in scientific claim verification is determining when available evidence is sufficient to justify an answer, rather than simply selecting the most accurate model.

5. Abstention-aware evaluation provides a practical, model-agnostic framework for assessing scientific reliability that better reflects real-world deployment requirements than traditional forced-answer metrics.