PTSBench: A Comprehensive Post-Training Sparsity Benchmark Towards Algorithms and Models

TL;DR

This paper introduces PTSBench, a comprehensive benchmark for evaluating post-training sparsity algorithms and models, providing new insights, evaluations, and an open-source framework to advance research in model efficiency.

Contribution

It presents the first extensive benchmark for post-training sparsity, evaluating over 40 models and 10 techniques across multiple tasks, offering valuable insights and a reusable framework.

Findings

New observations on PTS algorithms' effectiveness.

In-depth evaluation of models' sparsification ability.

Open-source framework for future research.

Abstract

With the increased attention to model efficiency, post-training sparsity (PTS) has become more and more prevalent because of its effectiveness and efficiency. However, there remain questions on better practice of PTS algorithms and the sparsification ability of models, which hinders the further development of this area. Therefore, a benchmark to comprehensively investigate the issues above is urgently needed. In this paper, we propose the first comprehensive post-training sparsity benchmark called PTSBench towards algorithms and models. We benchmark 10+ PTS general-pluggable fine-grained techniques on 3 typical tasks using over 40 off-the-shelf model architectures. Through extensive experiments and analyses, we obtain valuable conclusions and provide several insights from both algorithms and model aspects. Our PTSBench can provide (1) new observations for a better understanding of the PTS algorithms, (2) in-depth and comprehensive evaluations for the sparsification ability of models, and (3) a well-structured and easy-integrate open-source framework. We hope this work will provide illuminating conclusions and advice for future studies of post-training sparsity methods and sparsification-friendly model design. The code for our PTSBench is released at \href{https://github.com/ModelTC/msbench}{https://github.com/ModelTC/msbench}.