Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

TL;DR

Fin-RATE is a comprehensive benchmark for evaluating large language models on complex financial analysis tasks involving SEC filings, highlighting significant performance challenges and diagnostic insights across multiple reasoning dimensions.

Contribution

Introduces Fin-RATE, a novel benchmark that assesses LLMs on multi-faceted financial analysis tasks, addressing gaps in existing benchmarks by evaluating reasoning, retrieval, and cross-entity tracking.

Findings

LLMs' accuracy drops by up to 18.60% in complex tasks

Performance degradation linked to increased hallucinations and mismatches

Existing benchmarks do not quantify reasoning and factual errors effectively

Abstract

With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures. However, existing benchmarks often focus on isolated details, failing to reflect the complexity of professional analysis that requires synthesizing information across multiple documents, reporting periods, and corporate entities. Furthermore, these benchmarks do not disentangle whether errors arise from retrieval failures, generation inaccuracies, domain-specific reasoning mistakes, or misinterpretation of the query or context, making it difficult to precisely diagnose performance bottlenecks. To bridge these gaps, we introduce Fin-RATE, a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst workflows through three pathways: detail-oriented reasoning within individual disclosures, cross-entity comparison under shared topics, and longitudinal tracking of the same firm across reporting periods. We benchmark 17 leading LLMs, spanning open-source, closed-source, and finance-specialized models, under both ground-truth context and retrieval-augmented settings. Results show substantial performance degradation, with accuracy dropping by 18.60\% and 14.35\% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis. This degradation is driven by increased comparison hallucinations, temporal and entity mismatches, and is further reflected in declines in reasoning quality and factual consistency--limitations that existing benchmarks have yet to formally categorize or quantify.