Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report

TL;DR

This paper reports on developing Retrieval Augmented Generation (RAG) systems from PDFs, detailing the pipeline, challenges, solutions, and practical insights for integrating retrieval with LLMs to improve accuracy and transparency.

Contribution

It provides an end-to-end practical framework for building RAG systems from PDFs, comparing approaches with OpenAI and open-source models, and sharing technical solutions.

Findings

Effective pipeline for PDF-based RAG system development

Insights into technical challenges and solutions

Practical guidance for deploying RAG with LLMs

Abstract

This paper presents an experience report on the development of Retrieval Augmented Generation (RAG) systems using PDF documents as the primary data source. The RAG architecture combines generative capabilities of Large Language Models (LLMs) with the precision of information retrieval. This approach has the potential to redefine how we interact with and augment both structured and unstructured knowledge in generative models to enhance transparency, accuracy, and contextuality of responses. The paper details the end-to-end pipeline, from data collection, preprocessing, to retrieval indexing and response generation, highlighting technical challenges and practical solutions. We aim to offer insights to researchers and practitioners developing similar systems using two distinct approaches: OpenAI's Assistant API with GPT Series and Llama's open-source models. The practical implications of this research lie in enhancing the reliability of generative AI systems in various sectors where domain-specific knowledge and real-time information retrieval is important. The Python code used in this work is also available at: https://github.com/GPT-Laboratory/RAG-LLM-Development-Guidebook-from-PDFs.