Herding Dynamic Weights for Partially Observed Random Field Models

TL;DR

This paper introduces herding dynamics, a deterministic method for learning parameters in partially observable random field models by generating complex trajectories that approximate data averages without expensive computations.

Contribution

It proposes a novel herding-based algorithm that treats parameters as dynamical variables, enabling efficient learning in complex random field models.

Findings

Convergence of trajectory averages to data averages under certain conditions

Herding dynamics avoids expensive exponentiation operations

The method is fully deterministic and computationally efficient

Abstract

Learning the parameters of a (potentially partially observable) random field model is intractable in general. Instead of focussing on a single optimal parameter value we propose to treat parameters as dynamical quantities. We introduce an algorithm to generate complex dynamics for parameters and (both visible and hidden) state vectors. We show that under certain conditions averages computed over trajectories of the proposed dynamical system converge to averages computed over the data. Our "herding dynamics" does not require expensive operations such as exponentiation and is fully deterministic.