Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool use

TL;DR

This paper demonstrates that simple keyword search tools within agentic frameworks can achieve over 90% of RAG system performance, offering a cost-effective alternative without relying on vector databases.

Contribution

It provides a systematic comparison showing that keyword search can replace vector databases in RAG, simplifying implementation and reducing costs.

Findings

Keyword search achieves over 90% of RAG performance.

Agentic keyword search is simpler and more cost-effective.

Effective for scenarios with frequent knowledge base updates.

Abstract

While Retrieval-Augmented Generation (RAG) has proven effective for generating accurate, context-based responses based on existing knowledge bases, it presents several challenges including retrieval quality dependencies, integration complexity and cost. Recent advances in agentic-RAG and tool-augmented LLM architectures have introduced alternative approaches to information retrieval and processing. We question how much additional value vector databases and semantic search bring to RAG over simple, agentic keyword search in documents for question-answering. In this study, we conducted a systematic comparison between RAG-based systems and tool-augmented LLM agents, specifically evaluating their retrieval mechanisms and response quality when the agent only has access to basic keyword search tools. Our empirical analysis demonstrates that tool-based keyword search implementations within an agentic framework can attain over $90\%$ of the performance metrics compared to traditional RAG systems without using a standing vector database. Our approach is simple to implement, cost effective, and is particularly useful in scenarios requiring frequent updates to knowledge bases.