OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
TL;DR
This paper investigates GPT-4's ability to perform tax calculations, analyzing its errors and limitations in understanding complex tax law examples demonstrated by OpenAI.
Contribution
It provides an in-depth analysis of GPT-4's shortcomings in applying tax law, highlighting specific failure points and the challenges of reliable tax computation by AI.
Findings
GPT-4 often produces incorrect tax calculations.
The model struggles with complex tax law reasoning.
OpenAI's demonstration contained an example sourced from external content.
Abstract
The authors explain where OpenAI got the tax law example in its livestream demonstration of GPT-4, why GPT-4 got the wrong answer, and how it fails to reliably calculate taxes.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFinancial Literacy, Pension, Retirement Analysis
MethodsAttention Is All You Need · Softmax · Dense Connections · Position-Wise Feed-Forward Layer · Absolute Position Encodings · Residual Connection · Adam · Linear Layer · Multi-Head Attention · Dropout