Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping

TL;DR

This paper introduces Ternary SVD, a novel parameterized linear mapping method that enhances network compression by using ternary matrices, reducing computational costs and achieving state-of-the-art results across diverse models.

Contribution

The paper proposes Ternary SVD, a new form of SVD with ternary matrices, along with algorithms and theoretical analysis for training and transition, improving network compression performance.

Findings

Achieves state-of-the-art compression on ConvNext, Swim, BERT, and OPT.

Reduces computation to addition instructions, avoiding expensive multiplications.

Provides theoretical convergence analysis for transition algorithms.

Abstract

We present a simple yet novel parameterized form of linear mapping to achieves remarkable network compression performance: a pseudo SVD called Ternary SVD (TSVD). Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive multiplication instructions, TSVD only requires addition instructions when computing $U(\cdot)$ and $V(\cdot)$. We provide direct and training transition algorithms for TSVD like Post Training Quantization and Quantization Aware Training respectively. Additionally, we analyze the convergence of the direct transition algorithms in theory. In experiments, we demonstrate that TSVD can achieve state-of-the-art network compression performance in various types of networks and tasks, including current baseline models such as ConvNext, Swim, BERT, and large language model like OPT.