Parametrization Cookbook: A set of Bijective Parametrizations for using Machine Learning methods in Statistical Inference

TL;DR

This paper introduces a collection of bijective, diffeomorphic parametrizations that transform constrained statistical inference problems into unconstrained ones, facilitating the use of modern computational methods while preserving statistical properties.

Contribution

It provides a set of bijective, diffeomorphic parametrizations and a Python package to simplify transforming constrained problems into unconstrained ones for statistical inference.

Findings

Enables use of automatic differentiation and GPU computing in constrained inference

Maintains identifiability and other statistical properties through bijective parametrizations

Provides a practical Python toolkit for implementing these transformations

Abstract

We present in this paper a way to transform a constrained statistical inference problem into an unconstrained one in order to be able to use modern computational methods, such as those based on automatic differentiation, GPU computing, stochastic gradients with mini-batch. Unlike the parametrizations classically used in Machine Learning, the parametrizations introduced here are all bijective and are even diffeomorphisms, thus allowing to keep the important properties from a statistical inference point of view, first of all identifiability. This cookbook presents a set of recipes to use to transform a constrained problem into a unconstrained one. For an easy use of parametrizations, this paper is at the same time a cookbook, and a Python package allowing the use of parametrizations with numpy, but also JAX and PyTorch, as well as a high level and expressive interface allowing to easily describe a parametrization to transform a difficult problem of statistical inference into an easier problem addressable with modern optimization tools.