Speeding Up Question Answering Task of Language Models via Inverted Index

TL;DR

This paper introduces an inverted index mechanism to significantly enhance the efficiency of large language model-based question answering systems, reducing response time and improving answer quality for closed-domain questions.

Contribution

The study proposes a novel inverted indexing approach to accelerate LLM question answering, demonstrating substantial efficiency gains and improved answer quality.

Findings

Average response time reduced by 97.44%

BLEU score increased by 0.23

Efficiency improvements enable practical deployment of LLMs in real-world applications

Abstract

Natural language processing applications, such as conversational agents and their question-answering capabilities, are widely used in the real world. Despite the wide popularity of large language models (LLMs), few real-world conversational agents take advantage of LLMs. Extensive resources consumed by LLMs disable developers from integrating them into end-user applications. In this study, we leverage an inverted indexing mechanism combined with LLMs to improve the efficiency of question-answering models for closed-domain questions. Our experiments show that using the index improves the average response time by 97.44%. In addition, due to the reduced search scope, the average BLEU score improved by 0.23 while using the inverted index.