AFIB: A Multi-Dimensional Benchmark for Financial Intelligence in Large Language Models

Paper: 2603.08704 Authors: Akshay Gulati, Kanha Singhania, Tushar Banga, Parth Arora, Anshul Verma, Vaibhav Kumar Singh, Agyapal Digra, Jayant Singh Bisht, Danish Sharma, Varun Singla Categories: cs.AI

The Gap

LLMs are being deployed for financial analysis—equity research, investment decisions, portfolio management—but we’re flying blind. Existing benchmarks test general reasoning (MMLU, BBH) or narrow financial knowledge (FinQA focuses on numerical reasoning over tables), but none systematically evaluate whether these models can actually do what analysts do: synthesize multi-source data, maintain consistency across related queries, distinguish facts from hallucinations, and work with current market information. The field has capability without measurement.

Problem: LLMs used in finance without domain-specific evaluation
    |
    v
Assumption: Financial intelligence is multi-dimensional
            (not just accuracy, but recency, consistency, completeness)
    |
    v
Method: AFIB benchmark with 5 evaluation dimensions
        + 95 structured questions from real equity research
    |
    v
Evidence: SuperInvesting (8.96/10 accuracy, 56.65/70 completeness)
          outperforms GPT, Gemini, Claude, Perplexity
    |
    v
Conclusion: Specialized systems with structured data access
            beat general-purpose LLMs for financial tasks

The Increment

One sentence: Before this paper, we had no systematic way to compare LLMs on real financial analysis workflows; after, we have a multi-dimensional benchmark showing that domain-specialized systems outperform general models.

Core Mechanism

AFIB evaluates five dimensions across 95 questions derived from actual equity research tasks. The dimensions are: (1) factual accuracy—can the model get basic financial facts right? (2) analytical completeness—does it cover all relevant aspects of a question? (3) data recency—is it working with current information? (4) model consistency—does it give the same answer to semantically identical questions? (5) failure patterns—what types of errors does it make?

Each question gets scored by domain experts using rubrics. Factual accuracy uses a 0-10 scale. Completeness breaks down into sub-components (for a company analysis question: business model, financials, competitive position, risks, etc.) with points allocated to each. Recency checks whether data is from the last quarter. Consistency tests involve asking the same question in different phrasings and measuring response similarity.

The benchmark tests five systems: GPT-4, Gemini, Claude, Perplexity, and SuperInvesting (a specialized financial AI). Questions span company analysis, sector comparisons, financial metric calculations, and market trend interpretation—the actual work of equity analysts.

Question Set (95 items)
    |
    +---> Dimension 1: Factual Accuracy (0-10 scale)
    |         |
    |         +---> Expert scoring against ground truth
    |
    +---> Dimension 2: Completeness (0-70 points)
    |         |
    |         +---> Rubric with sub-components
    |
    +---> Dimension 3: Data Recency (binary)
    |         |
    |         +---> Check if data < 1 quarter old
    |
    +---> Dimension 4: Consistency (similarity score)
    |         |
    |         +---> Rephrase question, measure response variance
    |
    +---> Dimension 5: Failure Patterns (categorization)
              |
              +---> Hallucination / Outdated / Incomplete / Refusal

Think of AFIB like a driving test for financial analysts. A written exam (factual accuracy) isn’t enough—you need to test parallel parking (completeness), highway merging (recency), consistency across different road conditions (model consistency), and what happens when things go wrong (failure patterns). A general-purpose LLM is like someone who passed a generic driving test; SuperInvesting is like someone trained specifically for Formula 1 racing. The benchmark reveals that financial analysis isn’t just “answer questions correctly”—it’s a multi-skill performance where specialized training matters.

Key Concepts

• Analytical Completeness: When an analyst evaluates a company, they don’t just state facts—they build a complete picture. If you ask “Should I invest in Tesla?”, a complete answer covers: business model (how they make money), financial health (margins, cash flow, debt), competitive position (vs BYD, legacy automakers), growth drivers (new models, energy storage), and risks (Musk dependency, regulatory changes). Completeness means hitting all the dimensions a professional would consider. A model scoring 30/70 might nail the financials but miss competitive dynamics entirely—dangerous for actual investment decisions.

• Hallucination in Financial Context: In finance, hallucination isn’t just making up facts—it’s stating specific numbers, dates, or events that are verifiably false. If a model says “Apple’s Q3 2024 revenue was $95B" when it was actually$85B, that’s a hallucination. If it claims “Tesla announced a $25K model in March 2024” when no such announcement happened, that’s a hallucination. The benchmark tracks these explicitly because in finance, a single wrong number can cascade into catastrophically bad decisions. It’s not about being approximately right—precision matters.

• Data Recency: Financial markets move fast. A company analysis using 2023 data in mid-2024 is worthless—earnings have been reported, guidance has changed, new products launched. Recency measures whether the model accesses current information. Perplexity scores high here because it searches the live web. Base LLMs with training cutoffs fail. This dimension captures a fundamental limitation: no matter how smart your reasoning, if you’re working with stale data, your conclusions are wrong.

Framework Shift

Before (general LLM evaluation):        After (AFIB):

Single metric (accuracy)                Five dimensions
      |                                       |
      v                                       +---> Accuracy
[Question] --> [Answer] --> [Score]           +---> Completeness
                                              +---> Recency
Generic questions                             +---> Consistency
                                              +---> Failure modes
                                              
                                        Domain-specific questions
                                        (real equity research tasks)

From “can it answer correctly?” to “can it do the job?”, the core shift is evaluating capability as a multi-dimensional performance profile rather than a single accuracy number.

Expert Assessment

Problem choice: Real gap. Finance is a high-stakes LLM deployment area, and existing benchmarks genuinely don’t capture what matters. The field needed this. However, the paper has a massive conflict of interest—the authors are affiliated with SuperInvesting, which conveniently wins their benchmark. This doesn’t invalidate the benchmark design, but it raises questions about evaluation rigor.

Method maturity: The five-dimensional framework is solid and well-motivated. But the execution is basic—95 questions is small, expert scoring introduces subjectivity, and there’s no inter-rater reliability reported. The consistency testing (rephrasing questions) is clever but underexplored. A more mature version would have 500+ questions, multiple expert raters with agreement metrics, and deeper analysis of failure modes.

Experimental integrity: Red flag: SuperInvesting wins on every dimension. That’s suspicious. Either it’s genuinely better (possible—specialized systems should outperform general models), or the benchmark is tuned to its strengths. The paper doesn’t provide enough detail on question selection or scoring procedures to judge. Baselines are reasonable (GPT-4, Claude, Gemini are the right comparisons), but I’d want to see the raw data and scoring rubrics to trust these results.

Writing quality: The paper reads like a product announcement disguised as research. The related work section is thin—no serious engagement with financial NLP literature or existing benchmarks. The results section is mostly tables without deep analysis of why systems differ. If they rewrote Section 4 (Results) to include error analysis, case studies of specific failures, and honest discussion of SuperInvesting’s limitations, this would be a much stronger contribution.

Verdict: weak accept — The benchmark design addresses a real need and the multi-dimensional framework is valuable, but the execution is preliminary and the conflict of interest undermines trust in the results.

Takeaways

The five-dimensional evaluation framework is immediately useful for anyone building domain-specific LLM applications. Don’t just measure accuracy—measure completeness (did it cover all relevant aspects?), consistency (does it give the same answer to equivalent questions?), recency (is it using current data?), and failure modes (how does it break?). This framework transfers directly to legal, medical, or technical domains.

The finding that specialized systems outperform general LLMs is important but not surprising—the real insight is quantifying how much specialization matters across different dimensions. Perplexity wins on recency but loses on analytical depth; GPT-4 is consistent but often outdated. This suggests a design principle: for high-stakes domains, you need both current data access and domain-specific reasoning, not just one or the other.

The consistency testing approach (rephrasing questions and measuring response variance) is a simple but powerful technique for evaluating reliability. Steal this for any application where users might ask the same thing in different ways—if your model gives contradictory answers, that’s a trust problem.