Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression

TL;DR

This paper proposes Instruction-Aware Contextual Compression to filter irrelevant information in retrieval-augmented LLMs, significantly reducing costs and latency while maintaining comparable performance.

Contribution

It introduces a novel method for filtering context in retrieval-augmented LLMs, improving efficiency without sacrificing accuracy.

Findings

50% reduction in context-related costs

2.2-fold increase in inference speed

0.047 Rouge-1 score drop

Abstract

Large Language Models (LLMs) have garnered widespread attention due to their remarkable performance across various tasks. However, to mitigate the issue of hallucinations, LLMs often incorporate retrieval-augmented pipeline to provide them with rich external knowledge and context. Nevertheless, challenges stem from inaccurate and coarse-grained context retrieved from the retriever. Supplying irrelevant context to the LLMs can result in poorer responses, increased inference latency, and higher costs. This paper introduces a method called Instruction-Aware Contextual Compression, which filters out less informative content, thereby accelerating and enhancing the use of LLMs. The experimental results demonstrate that Instruction-Aware Contextual Compression notably reduces memory consumption and minimizes generation latency while maintaining performance levels comparable to those achieved with the use of the full context. Specifically, we achieved a 50% reduction in context-related costs, resulting in a 5% reduction in inference memory usage and a 2.2-fold increase in inference speed, with only a minor drop of 0.047 in Rouge-1. These findings suggest that our method strikes an effective balance between efficiency and performance.