Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law

TL;DR

This paper investigates the reliability of large language models in legal reasoning, revealing contamination issues and demonstrating that neuro-symbolic systems provide more robust and generalizable legal AI solutions.

Contribution

It introduces a contamination detection protocol, compares monolithic LLMs with hybrid neuro-symbolic systems, and provides a new test suite for evaluating legal reasoning generalization.

Findings

Performance can be inflated by data contamination.

Neuro-symbolic frameworks outperform pure LLMs in robustness.

Legal reasoning is inherently compositional.

Abstract

Recent advances in large language models (LLMs) have significantly enhanced automated legal reasoning. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability. We show that performance can be inflated by contamination. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations. Our findings indicate that legal reasoning is inherently compositional and that neuro-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations.