Large Language Models Acing Chartered Accountancy

TL;DR

This paper evaluates the financial, legal, and quantitative reasoning abilities of large language models using a new Chartered Accountancy benchmark based on Indian CA exams, revealing performance gaps and potential improvements.

Contribution

Introduces CA-Ben, a novel benchmark for assessing LLMs on domain-specific CA knowledge, and evaluates six prominent models on this challenging financial reasoning task.

Findings

Claude 3.5 Sonnet and GPT-4o outperform others

Models struggle with numerical computations and legal interpretations

Performance varies across different reasoning tasks

Abstract

Advanced intelligent systems, particularly Large Language Models (LLMs), are significantly reshaping financial practices through advancements in Natural Language Processing (NLP). However, the extent to which these models effectively capture and apply domain-specific financial knowledge remains uncertain. Addressing a critical gap in the expansive Indian financial context, this paper introduces CA-Ben, a Chartered Accountancy benchmark specifically designed to evaluate the financial, legal, and quantitative reasoning capabilities of LLMs. CA-Ben comprises structured question-answer datasets derived from the rigorous examinations conducted by the Institute of Chartered Accountants of India (ICAI), spanning foundational, intermediate, and advanced CA curriculum stages. Six prominent LLMs i.e. GPT 4o, LLAMA 3.3 70B, LLAMA 3.1 405B, MISTRAL Large, Claude 3.5 Sonnet, and Microsoft Phi 4 were evaluated using standardized protocols. Results indicate variations in performance, with Claude 3.5 Sonnet and GPT-4o outperforming others, especially in conceptual and legal reasoning. Notable challenges emerged in numerical computations and legal interpretations. The findings emphasize the strengths and limitations of current LLMs, suggesting future improvements through hybrid reasoning and retrieval-augmented generation methods, particularly for quantitative analysis and accurate legal interpretation.