Evaluating Chunking Strategies For Retrieval-Augmented Generation in Oil and Gas Enterprise Documents

TL;DR

This study compares four document chunking strategies for retrieval-augmented generation in oil and gas documents, finding structure-aware methods improve retrieval effectiveness and efficiency, but struggle with visually encoded diagrams, highlighting the need for multimodal approaches.

Contribution

It provides an empirical evaluation of chunking strategies in a specialized domain, emphasizing the importance of structure-aware methods and identifying limitations with visual documents.

Findings

Structure-aware chunking improves retrieval metrics.

Structure-aware methods have lower computational costs.

All methods perform poorly on visually encoded diagrams.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a framework to address the constraints of Large Language Models (LLMs). Yet, its effectiveness fundamentally hinges on document chunking - an often-overlooked determinant of its quality. This paper presents an empirical study quantifying performance differences across four chunking strategies: fixed-size sliding window, recursive, breakpoint-based semantic, and structure-aware. We evaluated these methods using a proprietary corpus of oil and gas enterprise documents, including text-heavy manuals, table-heavy specifications, and piping and instrumentation diagrams (P and IDs). Our findings show that structure-aware chunking yields higher overall retrieval effectiveness, particularly in top-K metrics, and incurs significantly lower computational costs than semantic or baseline strategies. Crucially, all four methods demonstrated limited effectiveness on P and IDs, underscoring a core limitation of purely text-based RAG within visually and spatially encoded documents. We conclude that while explicit structure preservation is essential for specialised domains, future work must integrate multimodal models to overcome current limitations.