Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization

TL;DR

This paper explores multi-query prompting for large language models to reduce inference costs in meeting summarization, demonstrating that combining queries can maintain performance while saving resources.

Contribution

It introduces a multi-query prompting approach for LLMs in meeting summarization, analyzing its effectiveness and cost-efficiency across various models.

Findings

Multi-query prompting reduces inference costs significantly.

Closed-source LLMs achieve higher reliability in expected formats.

Open-source LLMs lag in response reliability, except some 7B models.

Abstract

This work focuses on the task of query-based meeting summarization in which the summary of a context (meeting transcript) is generated in response to a specific query. When using Large Language Models (LLMs) for this task, usually a new call to the LLM inference endpoint/API is triggered for each new query, even if the context stays the same. However, repeated calls to the LLM inference endpoints would significantly increase the costs of using them in production, making LLMs impractical for many real-world use cases. To address this problem, in this paper, we investigate whether combining the queries for the same input context in a single prompt to minimize repeated calls can be successfully used in meeting summarization. In this regard, we conduct extensive experiments by comparing the performance of various popular LLMs: GPT-4, Gemini, Claude-3, LLaMA-2, Mistral, Phi-3, and Qwen-2 in single-query and multi-query settings. We observe that 100% reliability in generating the response in the expected format is usually limited to certain closed-source LLMs, with most open-source LLMs lagging behind (except a few 7B parameters LLMs like Mistral and Phi-3). We conclude that multi-query prompting could be useful to significantly optimize the inference costs in meeting summarization.