Inference acceleration for large language models using "stairs" assisted greedy generation

TL;DR

This paper introduces a 'stairs' assisted greedy generation method that reduces inference time for large language models by 9.58% to 17.24% without sacrificing accuracy, leveraging smaller models and batch prediction.

Contribution

It proposes a novel 'stairs' assisted greedy generation technique that accelerates large language model inference using a combination of smaller models and validation steps.

Findings

Achieves 9.58% to 17.24% inference speedup

Maintains accuracy comparable to large models

Utilizes a hybrid generation approach with 'stairs' validation

Abstract

Large Language Models (LLMs) with billions of parameters are known for their impressive predicting capabilities but require lots of resources to run. With their massive rise in popularity, even a small reduction in required resources could have an impact on environment. On the other hand, smaller models require fewer resources but may sacrifice accuracy. In this work, we are proposing an implementation of ``stairs'' assisted greedy generation. It is a modified assisted generation methodology that makes use of a smaller model's fast generation, large model's batch prediction, and "stairs" validation in order to achieve a speed up in prediction generation. Results show between 9.58 and 17.24 percent inference time reduction compared to a stand-alone large LLM prediction in a text generation task without a loss in accuracy.