QUEST: Query Optimization in Unstructured Document Analysis

TL;DR

QUEST introduces novel optimization strategies for unstructured document analysis using LLMs, significantly reducing costs and improving accuracy by employing index-based retrieval, evidence augmentation, and document-specific query plans.

Contribution

The paper presents a new query optimization framework, QUEST, tailored for LLM-based unstructured document analysis, with strategies to minimize extraction costs and enhance retrieval accuracy.

Findings

Achieves 30%-6x cost savings over baselines.

Improves F1 score by 10%-27%.

Demonstrates effectiveness on three real-world datasets.

Abstract

Most recently, researchers have started building large language models (LLMs) powered data systems that allow users to analyze unstructured text documents like working with a database because LLMs are very effective in extracting attributes from documents. In such systems, LLM-based extraction operations constitute the performance bottleneck of query execution due to the high monetary cost and slow LLM inference. Existing systems typically borrow the query optimization principles popular in relational databases to produce query execution plans, which unfortunately are ineffective in minimizing LLM cost. To fill this gap, we propose QUEST, which features a bunch of novel optimization strategies for unstructured document analysis. First, we introduce an index-based strategy to minimize the cost of each extraction operation. With this index, QUEST quickly retrieves the text segments relevant to the target attributes and only feeds them to LLMs. Furthermore, we design an evidence-augmented retrieval strategy to reduce the possibility of missing relevant segments. Moreover, we develop an instance-optimized query execution strategy: because the attribute extraction cost could vary significantly document by document, QUEST produces different plans for different documents. For each document, QUEST produces a plan to minimize the frequency of attribute extraction. The innovations include LLM cost-aware operator ordering strategies and an optimized join execution approach that transforms joins into filters. Extensive experiments on 3 real-world datasets demonstrate the superiority of QUEST, achieving 30%-6x cost savings while improving the F1 score by 10% -27% compared with state-of-the-art baselines.