Reducing Redundancy in Retrieval-Augmented Generation through Chunk Filtering

TL;DR

This paper proposes chunk filtering strategies to reduce redundancy in retrieval-augmented generation, decreasing storage costs while maintaining retrieval quality.

Contribution

It introduces lightweight filtering methods, including entity-based filtering, to effectively reduce index size without sacrificing retrieval performance.

Findings

Entity-based filtering reduces index size by 25-36%.

Filtering maintains high retrieval quality close to baseline.

Redundancy in chunking can be mitigated through simple filtering techniques.

Abstract

Standard Retrieval-Augmented Generation (RAG) chunking methods often create excessive redundancy, increasing storage costs and slowing retrieval. This study explores chunk filtering strategies, such as semantic, topic-based, and named-entity-based methods in order to reduce the indexed corpus while preserving retrieval quality. Experiments are conducted on multiple corpora. Retrieval performance is evaluated using a token-based framework based on precision, recall, and intersection-over-union metrics. Results indicate that entity-based filtering can reduce vector index size by approximately 25% to 36% while maintaining high retrieval quality close to the baseline. These findings suggest that redundancy introduced during chunking can be effectively reduced through lightweight filtering, improving the efficiency of retrieval-oriented components in RAG pipelines.