Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models

TL;DR

Open-RAG enhances open-source LLMs' reasoning by transforming them into sparse MoE models, improving accuracy in complex tasks through dynamic expert selection and hybrid retrieval strategies.

Contribution

The paper introduces Open-RAG, a novel framework that significantly improves reasoning in open-source LLMs by transforming them into sparse MoE models with dynamic expert navigation and hybrid retrieval.

Findings

Open-RAG outperforms state-of-the-art models like ChatGPT in knowledge-intensive tasks.

The hybrid retrieval method balances performance and inference speed effectively.

Transforming LLMs into sparse MoE models enhances reasoning capabilities.

Abstract

Retrieval-Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs), but existing methods often suffer from limited reasoning capabilities in effectively using the retrieved evidence, particularly when using open-source LLMs. To mitigate this gap, we introduce a novel framework, Open-RAG, designed to enhance reasoning capabilities in RAG with open-source LLMs. Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG uniquely trains the model to navigate challenging distractors that appear relevant but are misleading. As a result, Open-RAG leverages latent learning, dynamically selecting relevant experts and integrating external knowledge effectively for more accurate and contextually relevant responses. In addition, we propose a hybrid adaptive retrieval method to determine retrieval necessity and balance the trade-off between performance gain and inference speed. Experimental results show that the Llama2-7B-based Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT, Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source our code and models at https://openragmoe.github.io/