Implicit Feature Decoupling with Depthwise Quantization

TL;DR

This paper introduces Depthwise Quantization (DQ), a novel method that applies quantization along feature axes to enhance representation capacity and efficiency in deep neural networks, improving performance on image likelihood tasks.

Contribution

The paper proposes Depthwise Quantization, enabling implicit feature decoupling with minimal architectural changes, leading to higher capacity and better performance in hierarchical auto-encoders.

Findings

Outperforms previous state-of-the-art on CIFAR-10, ImageNet-32, and ImageNet-64.

Uses 69% fewer parameters and converges faster.

Enables training of larger models with improved efficiency.

Abstract

Quantization has been applied to multiple domains in Deep Neural Networks (DNNs). We propose Depthwise Quantization (DQ) where $\textit{quantization}$ is applied to a decomposed sub-tensor along the $\textit{feature axis}$ of weak statistical dependence. The feature decomposition leads to an exponential increase in $\textit{representation capacity}$ with a linear increase in memory and parameter cost. In addition, DQ can be directly applied to existing encoder-decoder frameworks without modification of the DNN architecture. We use DQ in the context of Hierarchical Auto-Encoder and train end-to-end on an image feature representation. We provide an analysis on cross-correlation between spatial and channel features and we propose a decomposition of the image feature representation along the channel axis. The improved performance of the depthwise operator is due to the increased representation capacity from implicit feature decoupling. We evaluate DQ on the likelihood estimation task, where it outperforms the previous state-of-the-art on CIFAR-10, ImageNet-32 and ImageNet-64. We progressively train with increasing image size a single hierarchical model that uses 69% less parameters and has a faster convergence than the previous works.