Poisson Hyperplane Processes with Rectified Linear Units

TL;DR

This paper establishes a probabilistic connection between Poisson hyperplane processes and two-layer ReLU neural networks, proposing a scalable Bayesian inference method that outperforms traditional models.

Contribution

It introduces a novel probabilistic representation of ReLU networks using PHP with a Gaussian prior and develops an efficient Bayesian inference algorithm.

Findings

The PHP-based model outperforms classic ReLU neural networks in experiments.

The proposed Bayesian inference method is scalable to large problems.

A new connection between hyperplane processes and neural network representations is established.

Abstract

Neural networks have shown state-of-the-art performances in various classification and regression tasks. Rectified linear units (ReLU) are often used as activation functions for the hidden layers in a neural network model. In this article, we establish the connection between the Poisson hyperplane processes (PHP) and two-layer ReLU neural networks. We show that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network. In addition, we show that a two-layer neural network constructed by PHP is scalable to large-scale problems via the decomposition propositions. Finally, we propose an annealed sequential Monte Carlo algorithm for Bayesian inference. Our numerical experiments demonstrate that our proposed method outperforms the classic two-layer ReLU neural network. The implementation of our proposed model is available at https://github.com/ShufeiGe/Pois_Relu.git.