Aletheia Tackles FirstProof Autonomously: AI Agent Solves 6 of 10 Mathematical Research Problems

Paper: 2602.21201 Authors: Tony Feng, Junehyuk Jung, Sang-hyun Kim, Carlo Pagano, Sergei Gukov, Chiang-Chiang Tsai, David Woodruff, Adel Javanmard, Aryan Mokhtari, Dawsen Hwang Categories: cs.AI, cs.CL, cs.LG

Abstract

This paper presents the performance evaluation of Aletheia, a mathematics research agent powered by Gemini 3 Deep Think, on the inaugural FirstProof challenge. The challenge consisted of 10 mathematical research problems that required autonomous problem-solving capabilities. Aletheia successfully solved 6 problems (specifically problems 2, 5, 7, 8, 9, and 10) within the competition timeframe, as assessed by a majority of expert evaluators. The authors note that Problem 8 was the only case where expert consensus was not unanimous. The paper emphasizes transparency by providing detailed explanations of their interpretation of the FirstProof challenge, experimental methodology, and evaluation procedures. All raw prompts and outputs have been made publicly available on GitHub for reproducibility and further analysis.

Key Contributions

• Demonstrated autonomous mathematical problem-solving at research level, achieving a 60% success rate on the FirstProof challenge
• Leveraged Gemini 3 Deep Think’s advanced reasoning capabilities to tackle open-ended mathematical research problems
• Provided full transparency with publicly available prompts, outputs, and evaluation methodology
• Established a benchmark for AI agents in mathematical research, with detailed documentation of both successes and limitations
• Contributed to the understanding of AI capabilities in formal mathematical reasoning and proof generation

System Architecture and Methodology

Aletheia represents a significant advancement in AI-driven mathematical research agents. Built on top of Gemini 3 Deep Think, the system is designed to autonomously approach mathematical problems that typically require human-level research capabilities. The architecture integrates several key components:

The core reasoning engine utilizes Gemini 3 Deep Think’s extended thinking capabilities, allowing the system to engage in prolonged deliberation and exploration of mathematical concepts. This is crucial for research-level problems that cannot be solved through pattern matching or simple heuristics alone.

The agent operates autonomously within the FirstProof challenge framework, meaning it must interpret problem statements, formulate approaches, generate proofs or solutions, and verify its work without human intervention during the solving process. This end-to-end autonomy distinguishes Aletheia from systems that require human guidance or interactive refinement.

The experimental setup adhered strictly to the FirstProof challenge constraints, including time limits and resource allocations. The authors emphasize their interpretation of the challenge rules and how they configured Aletheia to operate within these boundaries, ensuring fair comparison with other potential approaches.

Results and Performance Analysis

Aletheia’s performance on the FirstProof challenge reveals both the capabilities and current limitations of AI in mathematical research:

Successful Problems (2, 5, 7, 8, 9, 10): The agent demonstrated strong performance across 60% of the challenge problems. These successes span different mathematical domains and problem types, suggesting that Aletheia’s capabilities are not narrowly specialized but rather exhibit some degree of generalization across mathematical reasoning tasks.

Problem 8 - Contested Assessment: Interestingly, Problem 8 was the only case where expert evaluators did not reach unanimous consensus on whether Aletheia’s solution was correct. This highlights an important aspect of research-level mathematics: even human experts may disagree on the validity or completeness of certain proofs or solutions. This ambiguity is characteristic of frontier mathematical research and represents a challenge for both AI systems and their evaluation.

Failed Problems (1, 3, 4, 6): The four problems that Aletheia did not solve represent important areas for future improvement. While the paper does not provide detailed analysis of these failures, they likely indicate specific types of mathematical reasoning or problem structures that remain challenging for current AI systems.

The 60% success rate is particularly impressive given that these are research-level problems designed to challenge the boundaries of AI mathematical capabilities. This performance suggests that AI agents are approaching a threshold where they can meaningfully contribute to mathematical research, though significant gaps remain.

Implications for AI and Mathematical Research

The results presented in this paper have several important implications for the future of AI in mathematics and scientific research more broadly:

Autonomous Research Capabilities: Aletheia’s performance demonstrates that AI systems can now tackle open-ended research problems with minimal human intervention. This represents a qualitative shift from previous systems that primarily assisted human mathematicians or solved well-defined problems with clear solution paths.

Transparency and Reproducibility: The authors’ commitment to full transparency, including public release of all prompts and outputs, sets an important precedent for AI research. This level of openness enables the community to scrutinize the system’s reasoning, identify failure modes, and build upon this work. The availability of raw data at the GitHub repository allows for independent verification and analysis.

Evaluation Challenges: The disagreement among experts on Problem 8 highlights the difficulty of evaluating AI systems on research-level tasks. Unlike benchmark problems with known solutions, research problems may have multiple valid approaches or solutions that are difficult to assess definitively. This suggests that future AI evaluation frameworks for mathematical research will need to incorporate mechanisms for handling ambiguity and expert disagreement.

Hybrid Human-AI Research: While Aletheia operates autonomously, the most promising path forward likely involves collaboration between AI systems and human mathematicians. AI agents like Aletheia could explore large solution spaces, generate candidate proofs, or identify promising research directions, while humans provide high-level guidance, intuition, and final verification.

Takeaways

1. Aletheia achieved a 60% success rate (6/10 problems) on the FirstProof challenge, demonstrating significant progress in autonomous mathematical reasoning at research level.

2. The system is powered by Gemini 3 Deep Think, leveraging advanced reasoning capabilities to tackle open-ended mathematical problems without human intervention during solving.

3. Problem 8 revealed evaluation challenges in AI mathematical research, as expert assessors did not reach unanimous consensus on the solution’s validity.

4. Full transparency is provided through public release of all prompts and outputs on GitHub, enabling community scrutiny and reproducibility.

5. The results suggest AI agents are approaching a threshold where they can meaningfully contribute to mathematical research, though significant challenges remain in the 40% of unsolved problems.

6. This work establishes an important benchmark for future AI systems in mathematical research and highlights the need for robust evaluation frameworks that can handle ambiguity in research-level problems.