ThermEval: A Structured Benchmark for Evaluating Vision-Language Models on Thermal Imagery

Paper: 2602.14989 Authors: Ayush Shrivastava, Kirtan Gangani, Laksh Jain, Mayank Goel, Nipun Batra Categories: cs.CV, cs.AI, cs.LG

Abstract

Vision-language models (VLMs) have demonstrated impressive capabilities on RGB imagery, but their performance collapses when applied to thermal images. This paper introduces ThermEval, a structured benchmark designed to evaluate VLMs on thermal imagery understanding. The benchmark comprises approximately 55,000 thermal visual question answering pairs that assess foundational primitives required for thermal vision-language comprehension. The authors evaluate 25 state-of-the-art VLMs and find systematic failures in temperature-grounded reasoning, sensitivity to colormap transformations, and over-reliance on language priors. These findings highlight a critical gap in current multimodal AI systems and establish ThermEval as an essential benchmark for advancing thermal vision-language modeling.

Key Contributions

• ThermEval-B Benchmark: A structured evaluation framework with ~55,000 thermal VQA pairs covering temperature reasoning, spatial understanding, and semantic comprehension tasks
• ThermEval-D Dataset: The first dataset providing dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments
• Comprehensive VLM Evaluation: Systematic assessment of 25 open-source and closed-source vision-language models, revealing fundamental limitations in thermal understanding
• Analysis of Failure Modes: Identification of three critical failure patterns: temperature-grounded reasoning failures, colormap transformation sensitivity, and defaulting to language priors

The Thermal Vision Gap

Thermal imaging encodes fundamentally different information than RGB imagery. While RGB cameras capture reflected visible light representing color and texture, thermal cameras measure emitted infrared radiation corresponding to physical temperature. This distinction creates unique challenges for vision-language models trained predominantly on RGB data.

Thermal sensing is critical in scenarios where visible light is insufficient or unavailable: nighttime surveillance, search and rescue operations in low-visibility conditions, autonomous vehicle perception in adverse weather, medical screening for fever detection, and industrial inspection for heat anomalies. Despite these applications, existing VLM benchmarks focus exclusively on RGB imagery, leaving thermal understanding largely unexplored.

The authors argue that thermal comprehension requires distinct perceptual and reasoning capabilities. Models must interpret temperature distributions, understand thermal properties of materials, reason about heat transfer, and ground language descriptions in physical temperature values rather than visual appearance. Current RGB-centric evaluation frameworks cannot assess these capabilities.

Benchmark Design and Methodology

ThermEval-B is structured around three core evaluation dimensions:

Temperature Reasoning: Tasks requiring quantitative temperature understanding, including absolute temperature estimation, relative temperature comparison, and temperature range identification. These tasks assess whether models can extract and reason about numerical temperature values from thermal imagery.

Spatial Understanding: Evaluation of spatial reasoning capabilities specific to thermal images, including hotspot localization, thermal gradient detection, and spatial temperature distribution analysis. These tasks test whether models can identify and describe temperature patterns across image regions.

Semantic Comprehension: Assessment of semantic understanding in thermal contexts, including object recognition in thermal imagery, activity inference from thermal signatures, and scene understanding based on temperature distributions.

The benchmark integrates existing public thermal datasets with ThermEval-D, a newly collected dataset providing unprecedented annotation density. ThermEval-D includes per-pixel temperature measurements calibrated to physical units (Celsius/Fahrenheit), semantic segmentation masks for body parts and objects, and diverse capture conditions spanning indoor/outdoor environments, varying ambient temperatures, and multiple subjects.

Experimental Results and Findings

The evaluation of 25 VLMs reveals systematic failures across all model categories:

Temperature-Grounded Reasoning Failures: Models consistently struggle with quantitative temperature tasks. When asked to estimate absolute temperatures or compare temperature values, models either refuse to answer, provide wildly inaccurate estimates, or default to generic responses. Even when temperature information is explicitly encoded in colormaps, models fail to extract numerical values accurately.

Colormap Transformation Sensitivity: Performance degrades significantly when thermal images are presented with different colormap schemes (e.g., grayscale vs. jet vs. iron colormaps). This sensitivity indicates that models rely on superficial color patterns rather than understanding the underlying temperature information. The same thermal scene presented with different colormaps yields inconsistent and often contradictory responses.

Language Prior Dominance: Models frequently default to language priors or generate fixed responses regardless of image content. For example, when asked about temperature in various thermal images, models often respond with stereotypical phrases like “the person appears warm” without grounding in actual temperature data. This behavior suggests models are pattern-matching language rather than performing genuine visual reasoning.

Limited Improvement from Interventions: Both prompting strategies (few-shot examples, chain-of-thought reasoning, explicit temperature instructions) and supervised fine-tuning yield only marginal improvements. Even after fine-tuning on thermal data, models show minimal gains in temperature reasoning accuracy, suggesting fundamental architectural limitations rather than simple data distribution issues.

Quantitatively, the best-performing models achieve less than 40% accuracy on temperature reasoning tasks, compared to over 80% accuracy on analogous RGB tasks. The performance gap is even more pronounced for tasks requiring precise temperature estimation, where accuracy drops below 20%.

Implications for Multimodal AI

These findings have significant implications for the development of multimodal AI systems:

Domain Generalization Limitations: The failure of VLMs on thermal imagery demonstrates that current models do not learn generalizable visual reasoning capabilities. Instead, they appear to learn RGB-specific pattern recognition that does not transfer to other imaging modalities encoding different physical properties.

Need for Physics-Grounded Understanding: Effective thermal understanding requires models to reason about physical properties (temperature, heat transfer, thermal conductivity) rather than purely visual features. This suggests a need for architectures that can incorporate physical knowledge and constraints.

Evaluation Beyond RGB: The thermal domain reveals evaluation blind spots in current multimodal AI research. Benchmarks focused exclusively on RGB imagery may overestimate model capabilities and miss critical failure modes that emerge in other sensing modalities.

Application-Critical Gaps: For safety-critical applications like autonomous driving, medical screening, and search and rescue, the inability to process thermal imagery represents a significant limitation. These domains cannot rely on RGB sensing alone, making thermal understanding essential for real-world deployment.

Takeaways

1. Current vision-language models fail systematically at thermal image understanding, achieving less than 40% accuracy on temperature reasoning tasks despite strong RGB performance
2. ThermEval-B provides a structured benchmark with 55,000 thermal VQA pairs, establishing the first comprehensive evaluation framework for thermal vision-language understanding
3. Models exhibit three critical failure modes: inability to perform temperature-grounded reasoning, sensitivity to colormap transformations, and over-reliance on language priors
4. Neither advanced prompting strategies nor supervised fine-tuning significantly improve thermal understanding, suggesting fundamental architectural limitations
5. The thermal domain reveals that VLMs learn RGB-specific pattern recognition rather than generalizable visual reasoning, highlighting the need for physics-grounded multimodal architectures