Exploring Information Retrieval Landscapes: An Investigation of a Novel Evaluation Techniques and Comparative Document Splitting Methods

TL;DR

This paper investigates different document splitting methods and introduces a novel evaluation technique for Retrieval-Augmented Generation systems, emphasizing the importance of document characteristics and proposing metrics for improved retrieval accuracy.

Contribution

It presents a new evaluation method using an open-source model and compares document splitting techniques, highlighting the Recursive Character Splitter's superiority in maintaining context.

Findings

Recursive Character Splitter outperforms Token-based Splitter

New evaluation dataset generated with an open-source model

Weighted metrics improve assessment of retrieval relevance

Abstract

The performance of Retrieval-Augmented Generation (RAG) systems in information retrieval is significantly influenced by the characteristics of the documents being processed. In this study, the structured nature of textbooks, the conciseness of articles, and the narrative complexity of novels are shown to require distinct retrieval strategies. A comparative evaluation of multiple document-splitting methods reveals that the Recursive Character Splitter outperforms the Token-based Splitter in preserving contextual integrity. A novel evaluation technique is introduced, utilizing an open-source model to generate a comprehensive dataset of question-and-answer pairs, simulating realistic retrieval scenarios to enhance testing efficiency and metric reliability. The evaluation employs weighted scoring metrics, including SequenceMatcher, BLEU, METEOR, and BERT Score, to assess the system's accuracy and relevance. This approach establishes a refined standard for evaluating the precision of RAG systems, with future research focusing on optimizing chunk and overlap sizes to improve retrieval accuracy and efficiency.