On the Effectiveness of Proposed Techniques to Reduce Energy Consumption in RAG Systems: A Controlled Experiment

TL;DR

This study empirically evaluates five techniques to reduce energy consumption in retrieval-augmented generation (RAG) systems, identifying methods that significantly cut energy use while maintaining accuracy, thus guiding sustainable RAG development.

Contribution

First comprehensive empirical analysis of energy-efficient techniques in RAG systems, highlighting effective strategies like threshold tuning and embedding size reduction.

Findings

Energy reduction up to 60% with certain techniques.

Optimal retrieval threshold and embedding size reduce energy and latency without accuracy loss.

Some techniques cause unacceptable accuracy decreases, indicating trade-offs.

Abstract

The rising energy demands of machine learning (ML), e.g., implemented in popular variants like retrieval-augmented generation (RAG) systems, have raised significant concerns about their environmental sustainability. While previous research has proposed green tactics for ML-enabled systems, their empirical evaluation within RAG systems remains largely unexplored. This study presents a controlled experiment investigating five practical techniques aimed at reducing energy consumption in RAG systems. Using a production-like RAG system developed at our collaboration partner, the Software Improvement Group, we evaluated the impact of these techniques on energy consumption, latency, and accuracy. Through a total of 9 configurations spanning over 200 hours of trials using the CRAG dataset, we reveal that techniques such as increasing similarity retrieval thresholds, reducing embedding sizes, applying vector indexing, and using a BM25S reranker can significantly reduce energy usage, up to 60% in some cases. However, several techniques also led to unacceptable accuracy decreases, e.g., by up to 30% for the indexing strategies. Notably, finding an optimal retrieval threshold and reducing embedding size substantially reduced energy consumption and latency with no loss in accuracy, making these two techniques truly energy-efficient. We present the first comprehensive, empirical study on energy-efficient design techniques for RAG systems, providing guidance for developers and researchers aiming to build sustainable RAG applications.