First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

TL;DR

This paper proposes a training-free, threshold-based dynamic activation method that leverages sequence information to improve inference efficiency in large language models by exploiting inherent sparsity, with theoretical analysis of key sparsity features.

Contribution

It introduces a novel training-free dynamic activation technique that enhances LLM inference speed and provides theoretical insights into model sparsity.

Findings

Accelerates generation speed by 18-25%

Maintains task performance with minimal compromise

Provides theoretical analysis of activation uncertainty and inertia

Abstract

Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25\% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness.