Enhancing Scientific Literature Chatbots with Retrieval-Augmented Generation: A Performance Evaluation of Vector and Graph-Based Systems

TL;DR

This paper evaluates how combining vector and graph-based retrieval systems in scientific literature chatbots enhances access to scientific knowledge, focusing on accuracy and relevance in different retrieval scenarios.

Contribution

It introduces a hybrid retrieval-augmented generation framework for scientific chatbots and systematically evaluates its performance across multiple retrieval scenarios.

Findings

Hybrid RAG systems improve retrieval accuracy and relevance.

Graph-based retrieval complements vector methods for better results.

The approach supports evidence-based decision making in scientific research.

Abstract

This paper investigates the enhancement of scientific literature chatbots through retrieval-augmented generation (RAG), with a focus on evaluating vector- and graph-based retrieval systems. The proposed chatbot leverages both structured (graph) and unstructured (vector) databases to access scientific articles and gray literature, enabling efficient triage of sources according to research objectives. To systematically assess performance, we examine two use-case scenarios: retrieval from a single uploaded document and retrieval from a large-scale corpus. Benchmark test sets were generated using a GPT model, with selected outputs annotated for evaluation. The comparative analysis emphasizes retrieval accuracy and response relevance, providing insight into the strengths and limitations of each approach. The findings demonstrate the potential of hybrid RAG systems to improve accessibility to scientific knowledge and to support evidence-based decision making.