PolicyBot - Reliable Question Answering over Policy Documents

TL;DR

PolicyBot is a transparent, reproducible question-answering system over complex policy documents, combining advanced retrieval and generation techniques to improve accessibility and trustworthiness for citizens seeking legal information.

Contribution

The paper introduces PolicyBot, a novel RAG system tailored for policy documents that emphasizes transparency, multilingual support, and open-source implementation for trustworthy governance-related QA.

Findings

Effective domain-specific semantic chunking improves retrieval accuracy.

Citation tracing reduces hallucinations and enhances trust.

Open-source pipeline facilitates adaptation to other domains.

Abstract

All citizens of a country are affected by the laws and policies introduced by their government. These laws and policies serve essential functions for citizens. Such as granting them certain rights or imposing specific obligations. However, these documents are often lengthy, complex, and difficult to navigate, making it challenging for citizens to locate and understand relevant information. This work presents PolicyBot, a retrieval-augmented generation (RAG) system designed to answer user queries over policy documents with a focus on transparency and reproducibility. The system combines domain-specific semantic chunking, multilingual dense embeddings, multi-stage retrieval with reranking, and source-aware generation to provide responses grounded in the original documents. We implemented citation tracing to reduce hallucinations and improve user trust, and evaluated alternative retrieval and generation configurations to identify effective design choices. The end-to-end pipeline is built entirely with open-source tools, enabling easy adaptation to other domains requiring document-grounded question answering. This work highlights design considerations, practical challenges, and lessons learned in deploying trustworthy RAG systems for governance-related contexts.