Sirius: Contextual Sparsity with Correction for Efficient LLMs

TL;DR

Sirius is a correction mechanism that improves the reasoning performance of Contextual Sparsity compressed large language models, achieving significant efficiency gains and maintaining high accuracy on complex tasks.

Contribution

This paper introduces Sirius, a novel correction method that recovers the reasoning accuracy of CS-based LLMs while preserving their inference efficiency.

Findings

Sirius significantly improves CS model performance on reasoning tasks.

Sirius reduces latency by approximately 20% for 8B models and 35% for 70B models.

Sirius maintains efficiency gains while enhancing model accuracy.

Abstract

With the blossom of large language models (LLMs), inference efficiency becomes increasingly important. Various approximation methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without quality degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, CS significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models often share general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces Sirius, an efficient correction mechanism, which significantly recovers CS models quality on reasoning tasks while maintaining its efficiency gain. Sirius is evaluated on 6 models with 8 difficult generation tasks in reasoning, math, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for Sirius and show that Sirius achieves roughly 20% reduction in latency for 8B model on-chip and 35% reduction for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.