Block-term Tensor Neural Networks

TL;DR

This paper introduces block-term tensor layers (BT-layers) that approximate weight matrices in neural networks using low-rank block-term tensors, significantly reducing parameters while maintaining or enhancing performance.

Contribution

The paper proposes a novel tensor-based layer structure for neural networks that enables high compression and improved representation power, adaptable to CNNs and RNNs.

Findings

BT-layers achieve high compression ratios in CNNs and RNNs.

BT-layers preserve or improve neural network performance.

Experiments demonstrate effective parameter reduction with maintained accuracy.

Abstract

Deep neural networks (DNNs) have achieved outstanding performance in a wide range of applications, e.g., image classification, natural language processing, etc. Despite the good performance, the huge number of parameters in DNNs brings challenges to efficient training of DNNs and also their deployment in low-end devices with limited computing resources. In this paper, we explore the correlations in the weight matrices, and approximate the weight matrices with the low-rank block-term tensors. We name the new corresponding structure as block-term tensor layers (BT-layers), which can be easily adapted to neural network models, such as CNNs and RNNs. In particular, the inputs and the outputs in BT-layers are reshaped into low-dimensional high-order tensors with a similar or improved representation power. Sufficient experiments have demonstrated that BT-layers in CNNs and RNNs can achieve a very large compression ratio on the number of parameters while preserving or improving the representation power of the original DNNs.