Learning in Integer Latent Variable Models with Nested Automatic Differentiation

TL;DR

This paper introduces advanced nested automatic differentiation algorithms for exact inference and learning in complex integer latent variable models, achieving faster, more stable, and polynomial-time computations for nested derivatives.

Contribution

The paper presents novel, efficient AD algorithms for integer latent variable models, enabling exact gradient computation and improved learning performance.

Findings

Faster and more stable AD algorithms for nested derivatives

Exact gradient computation for complex models

Polynomial-time complexity in nesting levels

Abstract

We develop nested automatic differentiation (AD) algorithms for exact inference and learning in integer latent variable models. Recently, Winner, Sujono, and Sheldon showed how to reduce marginalization in a class of integer latent variable models to evaluating a probability generating function which contains many levels of nested high-order derivatives. We contribute faster and more stable AD algorithms for this challenging problem and a novel algorithm to compute exact gradients for learning. These contributions lead to significantly faster and more accurate learning algorithms, and are the first AD algorithms whose running time is polynomial in the number of levels of nesting.