# Improving Discrete Latent Representations With Differentiable Approximation Bridges

## TL;DR

This paper introduces a differentiable approximation bridge (DAB) that enables training neural networks with non-differentiable functions, leading to improved performance across various tasks.

## Contribution

The novel DAB method allows non-differentiable functions in neural networks to be trained effectively, expanding the range of usable functions beyond traditional differentiable ones.

## Key findings

- 10% improvement in image reconstruction quality over Gumbel-Softmax
- 9% better variational lower bound compared to RELAX
- 77% accuracy increase in sequence sorting

## Abstract

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable approximation bridge (DAB) neural network which approximates the non-differentiable forward function and provides gradient updates during backpropagation. We present strong empirical results (performing over 600 experiments) in four different domains: unsupervised (image) representation learning, variational (image) density estimation, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that training with DAB aided discrete non-differentiable functions improves image reconstruction quality and posterior linear separability by 10% against the Gumbel-Softmax relaxed estimator [37, 26] as well as providing a 9% improvement in the test variational lower bound in comparison to the state of the art RELAX [16] discrete estimator. We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator [5] in an image classification setting. The DAB network is not used for inference and expands the class of functions that are usable in neural networks.