Ancient Wisdom, Modern Tools: Exploring Retrieval-Augmented LLMs for Ancient Indian Philosophy

TL;DR

This paper investigates the use of retrieval-augmented generation models to improve factual accuracy and comprehensiveness in answering questions about Advaita Vedanta, demonstrating significant improvements over standard LLMs through a new dataset and hybrid retrieval methods.

Contribution

It introduces VedantaNY-10M, a new dataset on Indian philosophy, and benchmarks RAG models with hybrid retrieval, showing enhanced performance in specialized knowledge domains.

Findings

RAG models outperform standard LLMs in factual accuracy

Hybrid retrieval improves response quality further

Human evaluations favor RAG-generated answers

Abstract

LLMs have revolutionized the landscape of information retrieval and knowledge dissemination. However, their application in specialized areas is often hindered by factual inaccuracies and hallucinations, especially in long-tail knowledge distributions. We explore the potential of retrieval-augmented generation (RAG) models for long-form question answering (LFQA) in a specialized knowledge domain. We present VedantaNY-10M, a dataset curated from extensive public discourses on the ancient Indian philosophy of Advaita Vedanta. We develop and benchmark a RAG model against a standard, non-RAG LLM, focusing on transcription, retrieval, and generation performance. Human evaluations by computational linguists and domain experts show that the RAG model significantly outperforms the standard model in producing factual and comprehensive responses having fewer hallucinations. In addition, a keyword-based hybrid retriever that emphasizes unique low-frequency terms further improves results. Our study provides insights into effectively integrating modern large language models with ancient knowledge systems. Project page with dataset and code: https://sites.google.com/view/vedantany-10m