StreamingRAG: Real-time Contextual Retrieval and Generation Framework

TL;DR

StreamingRAG is a real-time retrieval and generation framework that constructs evolving knowledge graphs from multi-modal data streams, significantly improving speed, accuracy, and resource efficiency for applications like healthcare and remote sensing.

Contribution

It introduces a novel streaming RAG framework that builds temporal knowledge graphs in real-time, enabling faster and more accurate multi-modal data analysis.

Findings

5-6x faster throughput in real-time analysis

Improved contextual accuracy with temporal knowledge graphs

Reduced resource consumption by 2-3x using lightweight models

Abstract

Extracting real-time insights from multi-modal data streams from various domains such as healthcare, intelligent transportation, and satellite remote sensing remains a challenge. High computational demands and limited knowledge scope restrict the applicability of Multi-Modal Large Language Models (MM-LLMs) on these data streams. Traditional Retrieval-Augmented Generation (RAG) systems address knowledge limitations of these models, but suffer from slow preprocessing, making them unsuitable for real-time analysis. We propose StreamingRAG, a novel RAG framework designed for streaming data. StreamingRAG constructs evolving knowledge graphs capturing scene-object-entity relationships in real-time. The knowledge graph achieves temporal-aware scene representations using MM-LLMs and enables timely responses for specific events or user queries. StreamingRAG addresses limitations in existing methods, achieving significant improvements in real-time analysis (5-6x faster throughput), contextual accuracy (through a temporal knowledge graph), and reduced resource consumption (using lightweight models by 2-3x).