Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient

TL;DR

This paper demonstrates that Hamiltonian Monte Carlo with leapfrog integrator is inefficient for ReLU neural networks due to non-differentiability causing high local error rates, leading to increased rejection and computational cost.

Contribution

The paper provides a theoretical analysis showing the inefficiency of HMC on ReLU networks and empirically verifies the high rejection rates caused by non-differentiability.

Findings

HMC leapfrog has large local error rate on ReLU networks

High rejection rates make HMC inefficient for ReLU networks

Empirical results confirm theoretical analysis of inefficiency

Abstract

We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $\Omega(\epsilon)$ rather than the classical error rate of $O(\epsilon^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.