# Channel selection using Gumbel Softmax

## TL;DR

This paper introduces a unified end-to-end framework leveraging Gumbel Softmax for adaptive channel selection, enabling neural networks to optimize the accuracy-speed tradeoff during training and inference.

## Contribution

It proposes a novel Gumbel reparameterization method for joint pruning and conditional computation in neural networks, improving efficiency without separate procedures.

## Key findings

- Achieved 45-52% reduction in computation on ImageNet with ResNet.
- Supports both pruning and conditional computation within a single framework.
- Demonstrated promising results in inference efficiency improvements.

## Abstract

Important applications such as mobile computing require reducing the computational costs of neural network inference. Ideally, applications would specify their preferred tradeoff between accuracy and speed, and the network would optimize this end-to-end, using classification error to remove parts of the network. Increasing speed can be done either during training - e.g., pruning filters - or during inference - e.g., conditionally executing a subset of the layers. We propose a single end-to-end framework that can improve inference efficiency in both settings. We use a combination of batch activation loss and classification loss, and Gumbel reparameterization to learn network structure. We train end-to-end, and the same technique supports pruning as well as conditional computation. We obtain promising experimental results for ImageNet classification with ResNet (45-52% less computation).