BR-TaxQA-R: A Dataset for Question Answering with References for Brazilian Personal Income Tax Law, including case law

TL;DR

This paper introduces BR-TaxQA-R, a new dataset for question answering on Brazilian tax law, and evaluates a retrieval-augmented generation system that outperforms commercial tools in relevance but highlights the importance of human review.

Contribution

The paper presents a novel dataset for legal question answering in Brazilian tax law and benchmarks a RAG system, comparing it with commercial AI models.

Findings

Our RAG pipeline outperforms commercial tools in Response Relevancy.

Commercial models score higher in Factual Correctness and fluency.

Human expert evaluation is crucial for legal validity in AI-generated answers.

Abstract

This paper presents BR-TaxQA-R, a novel dataset designed to support question answering with references in the context of Brazilian personal income tax law. The dataset contains 715 questions from the 2024 official Q\&A document published by Brazil's Internal Revenue Service, enriched with statutory norms and administrative rulings from the Conselho Administrativo de Recursos Fiscais (CARF). We implement a Retrieval-Augmented Generation (RAG) pipeline using OpenAI embeddings for searching and GPT-4o-mini for answer generation. We compare different text segmentation strategies and benchmark our system against commercial tools such as ChatGPT and Perplexity.ai using RAGAS-based metrics. Results show that our custom RAG pipeline outperforms commercial systems in Response Relevancy, indicating stronger alignment with user queries, while commercial models achieve higher scores in Factual Correctness and fluency. These findings highlight a trade-off between legally grounded generation and linguistic fluency. Crucially, we argue that human expert evaluation remains essential to ensure the legal validity of AI-generated answers in high-stakes domains such as taxation. BR-TaxQA-R is publicly available at https://huggingface.co/datasets/unicamp-dl/BR-TaxQA-R.