TaxCalcBench: Evaluating Frontier Models on the Tax Calculation Task

TL;DR

TaxCalcBench is a new benchmark designed to evaluate AI models' ability to accurately calculate US personal income taxes, revealing current models' significant limitations in this complex task.

Contribution

We introduce TaxCalcBench, a benchmark for assessing models' proficiency in tax calculation, highlighting the gaps in current AI capabilities for this application.

Findings

State-of-the-art models solve less than a third of tax cases

Models frequently misuse tax tables and make calculation errors

Models often incorrectly determine taxpayer eligibility

Abstract

Can AI file your taxes? Not yet. Calculating US personal income taxes is a task that requires building an understanding of vast amounts of English text and using that knowledge to carefully compute results. We propose TaxCalcBench, a benchmark for determining models' abilities to calculate personal income tax returns given all of the necessary information. Our experiment shows that state-of-the-art models succeed in calculating less than a third of federal income tax returns even on this simplified sample set. Our analysis concludes that models consistently misuse tax tables, make errors in tax calculation, and incorrectly determine eligibility. Our findings point to the need for additional infrastructure to apply LLMs to the personal income tax calculation task.