TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL

TL;DR

TableCache introduces a method to precompute and efficiently manage KV caches for Text-to-SQL tasks, significantly reducing latency by leveraging primary foreign key relationships and a specialized cache lookup structure.

Contribution

It presents a novel offline precomputation of table representations with primary foreign key preservation and a Table Trie for efficient cache retrieval, improving inference speed.

Findings

Achieves up to 3.62x speedup in TTFT

Maintains negligible performance degradation

Enhances cache hit rates with reranking strategy

Abstract

In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets-offering an opportunity for KV cache sharing across queries-current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management system with a query reranking strategy to improve cache hit rates and a computation loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62x speedup in Time to First Token (TTFT) with negligible performance degradation.