Accounting Reasoning in Large Language Models: Concepts, Evaluation, and Empirical Analysis

TL;DR

This paper introduces a framework for evaluating accounting reasoning in large language models, assesses several models including GPT-4, and finds that current LLMs still need improvement for real-world accounting tasks.

Contribution

It proposes a set of evaluation criteria for accounting reasoning in LLMs and benchmarks multiple models, highlighting the current limitations and potential for future improvements.

Findings

GPT-4 shows the strongest accounting reasoning performance among tested models.

Prompt engineering can improve model performance variably.

Current LLMs are insufficient for real-world accounting applications.

Abstract

Large language models (LLMs) are increasingly reshaping learning paradigms, cognitive processes, and research methodologies across diverse domains. As their adoption expands, effectively integrating LLMs into professional fields and clarifying their role in domain-specific applications has become a key challenge for enterprise digital transformation and broader societal development. In the accounting domain, successful integration requires a systematic understanding of LLMs' domain-specific reasoning capabilities. In this study, we introduce the concept of accounting reasoning and propose a set of evaluation criteria grounded in an analysis of the training data characteristics of representative GLM-series models. These criteria establish a foundation for studying accounting-oriented reasoning paradigms and provide benchmarks for assessing and improving model performance. Building on this framework, we evaluate several representative LLMs, including GLM-6B, GLM-130B, GLM-4, and GPT-4, across a range of accounting reasoning tasks. Our experimental results show that prompt engineering strategies can yield varying degrees of performance improvement across models, with GPT-4 demonstrating the strongest overall accounting reasoning capability. Nevertheless, the results indicate that current LLMs remain insufficient for real-world accounting applications. In particular, further optimization is required for deployment in enterprise-level accounting scenarios to fully realize the potential value of LLMs in this domain.