A Survey on Efficient Inference for Large Language Models

TL;DR

This survey reviews techniques for improving the efficiency of large language model inference, addressing challenges like large model size and quadratic attention complexity, and categorizes existing methods across data, model, and system levels.

Contribution

It offers a comprehensive taxonomy of current efficient inference methods for LLMs, analyzes their primary challenges, and provides comparative experimental insights.

Findings

Quadratic attention complexity is a major bottleneck.

Data, model, and system-level optimizations are key strategies.

Comparative experiments highlight the effectiveness of different approaches.

Abstract

Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. Efforts within the field have been directed towards developing techniques aimed at enhancing the efficiency of LLM inference. This paper presents a comprehensive survey of the existing literature on efficient LLM inference. We start by analyzing the primary causes of the inefficient LLM inference, i.e., the large model size, the quadratic-complexity attention operation, and the auto-regressive decoding approach. Then, we introduce a comprehensive taxonomy that organizes the current literature into data-level, model-level, and system-level optimization. Moreover, the paper includes comparative experiments on representative methods within critical sub-fields to provide quantitative insights. Last but not least, we provide some knowledge summary and discuss future research directions.