TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction

TL;DR

This paper introduces a token compression scheme for retrieval-augmented LLMs, significantly reducing input size and costs while maintaining high accuracy through summarization and semantic compression methods.

Contribution

It proposes a novel token compression framework combining summarization and semantic filtering to lower retrieval context size for LLMs.

Findings

Summarization compression reduces token size by 65% with 0.3% accuracy gain.

Semantic compression achieves 20% token reduction with 1.6% accuracy loss.

The methods effectively balance cost reduction and model performance.

Abstract

Since ChatGPT released its API for public use, the number of applications built on top of commercial large language models (LLMs) increase exponentially. One popular usage of such models is leveraging its in-context learning ability and generating responses given user queries leveraging knowledge obtained by retrieval augmentation. One problem of deploying commercial retrieval-augmented LLMs is the cost due to the additionally retrieved context that largely increases the input token size of the LLMs. To mitigate this, we propose a token compression scheme that includes two methods: summarization compression and semantic compression. The first method applies a T5-based model that is fine-tuned by datasets generated using self-instruct containing samples with varying lengths and reduce token size by doing summarization. The second method further compresses the token size by removing words with lower impact on the semantic. In order to adequately evaluate the effectiveness of the proposed methods, we propose and utilize a dataset called Food-Recommendation DB (FRDB) focusing on food recommendation for women around pregnancy period or infants. Our summarization compression can reduce 65% of the retrieval token size with further 0.3% improvement on the accuracy; semantic compression provides a more flexible way to trade-off the token size with performance, for which we can reduce the token size by 20% with only 1.6% of accuracy drop.