Ternary Weight Networks

TL;DR

This paper introduces Ternary Weight Networks (TWNs), which use weights constrained to +1, 0, and -1, achieving high model compression and efficiency while maintaining competitive accuracy across multiple datasets and tasks.

Contribution

The paper proposes a novel ternary weight quantization method with a fast approximation function, improving model compression and performance over binary networks.

Findings

TWNs achieve up to 16× model compression.

TWNs outperform binary-weight networks in accuracy.

TWNs perform close to full precision networks on MNIST and CIFAR-10.

Abstract

We present a memory and computation efficient ternary weight networks (TWNs) - with weights constrained to +1, 0 and -1. The Euclidian distance between full (float or double) precision weights and the ternary weights along with a scaling factor is minimized in training stage. Besides, a threshold-based ternary function is optimized to get an approximated solution which can be fast and easily computed. TWNs have shown better expressive abilities than binary precision counterparts. Meanwhile, TWNs achieve up to 16$\times$ model compression rate and need fewer multiplications compared with the float32 precision counterparts. Extensive experiments on MNIST, CIFAR-10, and ImageNet datasets show that the TWNs achieve much better result than the Binary-Weight-Networks (BWNs) and the classification performance on MNIST and CIFAR-10 is very close to the full precision networks. We also verify our method on object detection task and show that TWNs significantly outperforms BWN by more than 10\% mAP on PASCAL VOC dataset. The pytorch version of source code is available at: https://github.com/Thinklab-SJTU/twns.