NitiBench: A Comprehensive Study of LLM Framework Capabilities for Thai Legal Question Answering

TL;DR

This paper introduces NitiBench, a benchmark for Thai legal question answering, evaluating LLM approaches and proposing new metrics, revealing current limitations and guiding future research in Thai legal NLP.

Contribution

The paper presents NitiBench, the first comprehensive benchmark for Thai legal QA, and evaluates LLM-based methods, proposing tailored metrics and analysis of retrieval and reasoning challenges.

Findings

Section-based chunking improves retrieval performance.

Current retrievers struggle with complex legal queries.

Long-context LLMs underperform RAG systems in Thai legal QA.

Abstract

The application of large language models (LLMs) in the legal domain holds significant potential for information retrieval and question answering, yet Thai legal QA systems face challenges due to a lack of standardized evaluation benchmarks and the complexity of Thai legal structures. This paper introduces NitiBench, a benchmark comprising two datasets: the NitiBench-CCL, covering general Thai financial law, and the NitiBench-Tax, which includes real-world tax law cases requiring advanced legal reasoning. We evaluate retrieval-augmented generation (RAG) and long-context LLM-based approaches to address three key research questions: the impact of domain-specific components like section-based chunking and cross-referencing, the comparative performance of different retrievers and LLMs, and the viability of long-context LLMs as an alternative to RAG. Our results show that section-based chunking significantly improves retrieval and end-to-end performance, current retrievers struggle with complex queries, and long-context LLMs still underperform RAG-based systems in Thai legal QA. To support fair evaluation, we propose tailored multi-label retrieval metrics and the use of an LLM-as-judge for coverage and contradiction detection method. These findings highlight the limitations of current Thai legal NLP solutions and provide a foundation for future research in the field. We also open-sourced our codes and dataset to available publicly.