RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture

TL;DR

This paper compares Retrieval-Augmented Generation and fine-tuning for LLMs, presenting a pipeline, tradeoffs, and a case study in agriculture demonstrating improved accuracy and geographic knowledge integration.

Contribution

It introduces a comprehensive pipeline for RAG and fine-tuning, evaluates their tradeoffs across multiple LLMs, and applies this to agriculture for location-specific insights.

Findings

Fine-tuning improves accuracy by over 6 percentage points.

Combining RAG with fine-tuning yields an additional 5 percentage point accuracy gain.

Fine-tuned models better leverage cross-geography information, increasing answer similarity from 47% to 72%.

Abstract

There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of Large Language Models (LLMs): Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the tradeoffs of both for multiple popular LLMs, including Llama2-13B, GPT-3.5, and GPT-4. Our pipeline consists of multiple stages, including extracting information from PDFs, generating questions and answers, using them for fine-tuning, and leveraging GPT-4 for evaluating the results. We propose metrics to assess the performance of different stages of the RAG and fine-Tuning pipeline. We conduct an in-depth study on an agricultural dataset. Agriculture as an industry has not seen much penetration of AI, and we study a potentially disruptive application - what if we could provide location-specific insights to a farmer? Our results show the effectiveness of our dataset generation pipeline in capturing geographic-specific knowledge, and the quantitative and qualitative benefits of RAG and fine-tuning. We see an accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further. In one particular experiment, we also demonstrate that the fine-tuned model leverages information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. Overall, the results point to how systems built using LLMs can be adapted to respond and incorporate knowledge across a dimension that is critical for a specific industry, paving the way for further applications of LLMs in other industrial domains.