FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles

TL;DR

FinRule-Bench is a new benchmark designed to evaluate large language models' ability to perform rule-based reasoning and verification on real-world financial statements, highlighting their strengths and limitations in high-stakes financial analysis.

Contribution

The paper introduces FinRule-Bench, a comprehensive benchmark with tasks for verifying, identifying, and diagnosing rule violations in financial statements, using real data and explicit principles.

Findings

Models excel at simple rule verification.

Performance drops significantly on complex rule discrimination.

Multi-violation detection remains challenging for LLMs.

Abstract

Large language models (LLMs) are increasingly applied to financial analysis, yet their ability to audit structured financial statements under explicit accounting principles remains poorly explored. Existing benchmarks primarily evaluate question answering, numerical reasoning, or anomaly detection on synthetically corrupted data, making it unclear whether models can reliably verify or localize rule compliance on correct financial statements. We introduce FinRule-Bench, a benchmark for evaluating diagnostic completeness in rule-based financial reasoning over real-world financial tables. FinRule-Bench pairs ground-truth financial statements with explicit, human-curated accounting principles and spans four canonical statement types: Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark defines three auditing tasks that require progressively stronger reasoning capabilities: (i) rule verification, which tests compliance with a single principle; (ii) rule identification, which requires selecting the violated principle from a provided rule set; and (iii) joint rule diagnosis, which requires detecting and localizing multiple simultaneous violations at the record level. We evaluate LLMs under zero-shot and few-shot prompting, and introduce a causal-counterfactual reasoning protocol that enforces consistency between decisions, explanations, and counterfactual judgments. Across tasks and statement types, we find that while models perform well on isolated rule verification, performance degrades sharply for rule discrimination and multi-violation diagnosis. FinRule-Bench provides a principled and reproducible testbed for studying rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis.