Full-Span Log-Linear Model and Fast Learning Algorithm

TL;DR

The paper introduces the full-span log-linear (FSLL) model, a high-order Boltzmann machine capable of representing complex distributions efficiently, with a fast learning algorithm suitable for large discrete systems up to 2^25 states.

Contribution

It presents the FSLL model as a novel high-order Boltzmann machine with an efficient dual-parameter computation and learning algorithm for large discrete distributions.

Findings

Successfully learned distributions with up to 2^20 states within one minute

Can represent arbitrary positive distributions without hyperparameter tuning

Computes dual parameters in O(|X| log |X|) time

Abstract

The full-span log-linear(FSLL) model introduced in this paper is considered an $n$-th order Boltzmann machine, where $n$ is the number of all variables in the target system. Let $X=(X_0,...,X_{n-1})$ be finite discrete random variables that can take $|X|=|X_0|...|X_{n-1}|$ different values. The FSLL model has $|X|-1$ parameters and can represent arbitrary positive distributions of $X$. The FSLL model is a "highest-order" Boltzmann machine; nevertheless, we can compute the dual parameters of the model distribution, which plays important roles in exponential families, in $O(|X|\log|X|)$ time. Furthermore, using properties of the dual parameters of the FSLL model, we can construct an efficient learning algorithm. The FSLL model is limited to small probabilistic models up to $|X|\approx2^{25}$; however, in this problem domain, the FSLL model flexibly fits various true distributions underlying the training data without any hyperparameter tuning. The experiments presented that the FSLL successfully learned six training datasets such that $|X|=2^{20}$ within one minute with a laptop PC.