LLM-Driven Online Aggregation for Unstructured Text Analytics

TL;DR

This paper introduces OLLA, an online aggregation framework leveraging Large Language Models to enable real-time, incremental text analytics with improved speed and accuracy, addressing the latency issues of traditional batch processing.

Contribution

The paper presents a novel LLM-driven online aggregation method with semantic stratified sampling, significantly accelerating unstructured text analysis compared to existing batch systems.

Findings

OLLA achieves less than 4% of full-data processing time to reach 1% accuracy.

OLLA speeds up text analytics by 1.6x to 38x across various domains.

The approach provides progressive results, improving real-time responsiveness in unstructured data analysis.

Abstract

Large Language Models (LLMs) exhibit strong capabilities in text processing, and recent research has augmented SQL and DataFrame with LLM-powered semantic operators for data analysis. However, LLM-based data processing is hindered by slower token generation speeds compared to relational queries. To enhance real-time responsiveness, we propose OLLA, an LLM-driven online aggregation framework that accelerates semantic processing within relational queries. In contrast to batch-processing systems that yield results only after the entire dataset is processed, our approach incrementally transforms text into a structured data stream and applies online aggregation to provide progressive output. To enhance our online aggregation process, we introduce a semantic stratified sampling approach that improves data selection and expedites convergence to the ground truth. Evaluations show that OLLA reaches 1% accuracy error bound compared with labeled ground truth using less than 4% of the full-data time. It achieves speedups ranging from 1.6$\times$ to 38$\times$ across diverse domains, measured by comparing the time to reach a 5% error bound with that of full-data time. We release our code at https://github.com/olla-project/llm-online-agg.git.