Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: An Experimental Study

TL;DR

This study evaluates large language models' ability to assist in Austrian VAT legal decision-making, highlighting their potential to support tax professionals while noting current limitations in automation and contextual understanding.

Contribution

It systematically assesses LLM performance in legal VAT tasks using fine-tuning and RAG, providing insights into their capabilities and challenges in legal decision support.

Findings

LLMs can support tax professionals with routine VAT tasks.

Properly configured LLMs can generate legally grounded justifications.

Current prototypes are not yet suitable for full automation due to contextual limitations.

Abstract

This paper provides an experimental evaluation of the capability of large language models (LLMs) to assist in legal decision-making within the framework of Austrian and European Union value-added tax (VAT) law. In tax consulting practice, clients often describe cases in natural language, making LLMs a prime candidate for supporting automated decision-making and reducing the workload of tax professionals. Given the requirement for legally grounded and well-justified analyses, the propensity of LLMs to hallucinate presents a considerable challenge. The experiments focus on two common methods for enhancing LLM performance: fine-tuning and retrieval-augmented generation (RAG). In this study, these methods are applied on both textbook cases and real-world cases from a tax consulting firm to systematically determine the best configurations of LLM-based systems and assess the legal-reasoning capabilities of LLMs. The findings highlight the potential of using LLMs to support tax consultants by automating routine tasks and providing initial analyses, although current prototypes are not ready for full automation due to the sensitivity of the legal domain. The findings indicate that LLMs, when properly configured, can effectively support tax professionals in VAT tasks and provide legally grounded justifications for decisions. However, limitations remain regarding the handling of implicit client knowledge and context-specific documentation, underscoring the need for future integration of structured background information.