Embracing the chaos: analysis and diagnosis of numerical instability in variational flows

TL;DR

This paper analyzes the effects of numerical instability in variational flows, revealing that despite significant errors, practical results often remain accurate, and provides a diagnostic method to validate such flows.

Contribution

It introduces a dynamical systems perspective to variational flows, uses shadowing theory for error analysis, and develops a diagnostic procedure for practical validation.

Findings

Numerical errors can significantly deviate flow maps from exact maps.

Despite errors, flow results are often sufficiently accurate for applications.

A diagnostic procedure effectively validates results from unstable flows.

Abstract

In this paper, we investigate the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. We first empirically demonstrate that common flows can exhibit a catastrophic accumulation of error: the numerical flow map deviates significantly from the exact map -- which affects sampling -- and the numerical inverse flow map does not accurately recover the initial input -- which affects density and ELBO computations. Surprisingly though, we find that results produced by flows are often accurate enough for applications despite the presence of serious numerical instability. In this work, we treat variational flows as dynamical systems, and leverage shadowing theory to elucidate this behavior via theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. Finally, we develop and empirically test a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.