OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

Paper: 2602.17665 Authors: Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir, Hiyam Debary, Mustansar Fiaz, Muhammad Zaigham Zaheer, Paolo Fraccaro, Fahad Shahbaz Khan, Muhammad Haris Khan, Xiao Xiang Zhu Categories: cs.CV

Abstract

OpenEarthAgent addresses a critical gap in multimodal AI by extending reasoning capabilities to the remote sensing domain. While recent advances enable agents to interpret imagery and perform structured tasks, geospatial analysis demands reasoning over spatial scales, geographic structures, and multispectral indices while maintaining coherent multi-step logic. This work introduces a unified framework for developing tool-augmented geospatial agents trained on satellite imagery, natural-language queries, and detailed reasoning traces through supervised fine-tuning over structured reasoning trajectories.

The framework includes a comprehensive corpus of 14,538 training instances and 1,169 evaluation instances, containing over 100K reasoning steps in training and 7K+ steps in evaluation. The dataset spans urban planning, environmental monitoring, disaster response, and infrastructure assessment, incorporating GIS-based operations alongside spectral index analyses including NDVI (Normalized Difference Vegetation Index), NBR (Normalized Burn Ratio), and NDBI (Normalized Difference Built-up Index). The resulting agent demonstrates structured reasoning, stable spatial understanding, and interpretable behavior through tool-driven interactions, showing consistent improvements over baselines and competitive performance against recent open and closed-source models.

Key Contributions

• A unified framework for training tool-augmented geospatial agents that integrate satellite imagery analysis with natural language reasoning and GIS operations
• A large-scale corpus with 14,538 training and 1,169 evaluation instances containing 100K+ structured reasoning trajectories across diverse geospatial domains
• Supervised fine-tuning methodology that aligns models with verified multi-step tool interactions, enabling coherent reasoning over spatial scales and multispectral indices
• Demonstrated improvements over strong baselines with competitive performance relative to state-of-the-art open and closed-source models in geospatial reasoning tasks

Framework Architecture and Training Pipeline

OpenEarthAgent’s architecture centers on aligning multimodal foundation models with geospatial reasoning through structured tool interactions. The training pipeline employs supervised fine-tuning (SFT) over reasoning trajectories that explicitly encode the sequence of analytical steps required for complex geospatial queries.

Each training instance consists of a satellite image, a natural language query, and a detailed reasoning trace that includes intermediate tool calls, spatial operations, and index computations. The model learns to decompose complex queries into executable sub-tasks, select appropriate tools (GIS operations, spectral index calculators, spatial analyzers), and synthesize results into coherent responses.

The framework supports multiple tool categories: (1) spectral analysis tools for computing NDVI, NBR, NDBI, and other indices; (2) GIS operations for spatial queries, buffer analysis, and overlay operations; (3) change detection modules for temporal analysis; and (4) object detection and segmentation tools for feature extraction. By grounding reasoning in explicit tool interactions, the agent maintains interpretability while handling diverse analytical contexts.

The training corpus spans four primary domains: urban development (land use classification, infrastructure mapping), environmental monitoring (vegetation health, water quality), disaster response (fire damage assessment, flood extent mapping), and infrastructure analysis (road networks, building footprints). This diversity ensures the agent generalizes across geospatial applications while maintaining domain-specific reasoning capabilities.

Experimental Results and Performance Analysis

OpenEarthAgent demonstrates consistent improvements over baseline models across multiple evaluation metrics. The evaluation protocol assesses both task completion accuracy and reasoning quality, measuring the agent’s ability to select appropriate tools, execute correct analytical sequences, and generate accurate interpretations.

Compared to a strong baseline model without tool augmentation, OpenEarthAgent achieves significant gains in multi-step reasoning tasks requiring spatial analysis and index computation. The structured reasoning traces enable the model to maintain coherence across complex analytical workflows, reducing error propagation in multi-step operations.

Performance analysis reveals several key strengths: (1) robust spatial understanding across different scales, from local feature detection to regional pattern analysis; (2) accurate spectral index computation and interpretation, particularly for NDVI-based vegetation assessment and NBR-based burn severity mapping; (3) effective tool selection and sequencing, with the model learning to chain operations appropriately for complex queries; and (4) interpretable reasoning, with explicit tool calls providing transparency into the analytical process.

Comparative evaluation against recent open-source models (specialized geospatial models and general-purpose multimodal agents) and closed-source systems shows competitive performance. While closed-source models may excel in certain linguistic aspects, OpenEarthAgent’s tool-augmented approach provides superior accuracy in domain-specific geospatial operations and maintains better interpretability through structured reasoning traces.

Implications for Geospatial AI and Remote Sensing

OpenEarthAgent represents a significant step toward practical AI systems for remote sensing applications. The framework’s emphasis on structured reasoning and tool integration addresses key challenges in deploying AI for geospatial analysis: the need for domain-specific operations, interpretability requirements for critical applications, and the complexity of multi-step analytical workflows.

The tool-augmented approach offers several advantages over end-to-end learned models. By explicitly incorporating GIS operations and spectral indices, the agent leverages established remote sensing methodologies while adding natural language interfaces and automated reasoning. This hybrid approach combines the reliability of traditional geospatial tools with the flexibility of modern AI systems.

The large-scale reasoning corpus also provides a valuable resource for future research. The 100K+ structured reasoning steps offer insights into how complex geospatial queries decompose into executable operations, potentially informing the design of more efficient analytical pipelines and improved training methodologies.

Future directions include expanding the tool library to incorporate additional remote sensing techniques (SAR analysis, hyperspectral processing, 3D reconstruction), extending temporal reasoning capabilities for long-term monitoring applications, and developing more sophisticated spatial reasoning mechanisms for handling complex geographic relationships and multi-scale analysis.

Takeaways

1. OpenEarthAgent introduces a unified framework for training tool-augmented geospatial agents through supervised fine-tuning on structured reasoning trajectories, bridging the gap between multimodal AI and remote sensing applications.

2. The framework includes a comprehensive corpus of 14,538 training instances with 100K+ reasoning steps spanning urban, environmental, disaster, and infrastructure domains, incorporating GIS operations and spectral indices like NDVI, NBR, and NDBI.

3. The tool-augmented approach enables interpretable, multi-step reasoning over satellite imagery while maintaining domain-specific accuracy through explicit integration of established geospatial analytical methods.

4. Experimental results demonstrate consistent improvements over baselines and competitive performance against state-of-the-art models, with particular strengths in spatial understanding, spectral analysis, and coherent multi-step reasoning.

5. The framework’s emphasis on structured reasoning and tool integration provides a practical pathway for deploying AI in critical remote sensing applications where interpretability and domain expertise are essential.