Robust model training and generalisation with Studentising flows

TL;DR

This paper introduces Studentising flows, a robust normalising flow approach using fat-tailed Student's t-distributions to improve model robustness, generalisation, and likelihood without sacrificing consistency.

Contribution

It proposes replacing Gaussian base distributions in normalising flows with Student's t-distributions, enhancing robustness and generalisation capabilities.

Findings

Fat-tailed distributions improve robustness similar to gradient clipping.

Models with Student's t-distributions show reduced generalisation gap.

Experimental results confirm improved likelihood and robustness.

Abstract

Normalising flows are tractable probabilistic models that leverage the power of deep learning to describe a wide parametric family of distributions, all while remaining trainable using maximum likelihood. We discuss how these methods can be further improved based on insights from robust (in particular, resistant) statistics. Specifically, we propose to endow flow-based models with fat-tailed latent distributions such as multivariate Student's $t$, as a simple drop-in replacement for the Gaussian distribution used by conventional normalising flows. While robustness brings many advantages, this paper explores two of them: 1) We describe how using fatter-tailed base distributions can give benefits similar to gradient clipping, but without compromising the asymptotic consistency of the method. 2) We also discuss how robust ideas lead to models with reduced generalisation gap and improved held-out data likelihood. Experiments on several different datasets confirm the efficacy of the proposed approach in both regards.