Coordinate Descent for Network Linearization

TL;DR

This paper introduces a coordinate descent method for directly reducing ReLU activations in neural networks, achieving state-of-the-art results by producing sparse solutions without the performance loss typical of approximation-based methods.

Contribution

It presents a novel discrete optimization approach using coordinate descent for network linearization, avoiding performance degradation from thresholding.

Findings

Achieves state-of-the-art ReLU reduction on benchmarks.

Produces inherently sparse network solutions.

Outperforms smooth approximation methods in accuracy and efficiency.

Abstract

ReLU activations are the main bottleneck in Private Inference that is based on ResNet networks. This is because they incur significant inference latency. Reducing ReLU count is a discrete optimization problem, and there are two common ways to approach it. Most current state-of-the-art methods are based on a smooth approximation that jointly optimizes network accuracy and ReLU budget at once. However, the last hard thresholding step of the optimization usually introduces a large performance loss. We take an alternative approach that works directly in the discrete domain by leveraging Coordinate Descent as our optimization framework. In contrast to previous methods, this yields a sparse solution by design. We demonstrate, through extensive experiments, that our method is State of the Art on common benchmarks.