COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection

TL;DR

This paper introduces COUNTDOWN, a novel linear-based sparsity method for large language models that significantly reduces computation during inference with minimal performance loss.

Contribution

It proposes two new sparsity methods, M-COUNTDOWN and D-COUNTDOWN, based on linear combination insights, achieving high computational savings with minimal accuracy impact.

Findings

D-COUNTDOWN omits 90% of computations with 5.5% performance loss

M-COUNTDOWN outperforms existing methods by up to 29.4% in performance preservation

Specialized kernels enable practical acceleration of the proposed methods

Abstract

The growing size of large language models has created significant computational inefficiencies. To address this challenge, sparse activation methods selectively deactivates non-essential parameters during inference, reducing computational costs in FFNN layers. While existing methods focus on non-linear gating mechanisms, we hypothesize that the sparsity of the FFNN layer lies globally in the form of a linear combination over its internal down projection matrix. Based on this insight, we propose two methods: M-COUNTDOWN, leveraging indirect coefficients, and D-COUNTDOWN, utilizing direct coefficients of the linear combination. Experimental results demonstrate that D-COUNTDOWN can omit 90% of computations with performance loss as low as 5.5% ideally, while M-COUNTDOWN provides a predictor-free solution with up to 29.4% better performance preservation compared to existing methods. Our specialized kernel implementations effectively realize these theoretical gains into substantial real-world acceleration.