Relative Loss Bounds for On-line Density Estimation with the Exponential Family of Distributions

TL;DR

This paper establishes bounds on the additional loss incurred by an online density estimation algorithm using exponential family distributions, compared to an offline optimal, for arbitrary data sequences.

Contribution

It introduces a divergence-based analysis to derive and bound the relative loss of online algorithms in exponential family density estimation.

Findings

Derived bounds on online vs. offline loss differences

Applicable to arbitrary data sequences

Uses a divergence based on relative entropy between exponential distributions

Abstract

We consider on-line density estimation with a parameterized density from the exponential family. The on-line algorithm receives one example at a time and maintains a parameter that is essentially an average of the past examples. After receiving an example the algorithm incurs a loss which is the negative log-likelihood of the example w.r.t. the past parameter of the algorithm. An off-line algorithm can choose the best parameter based on all the examples. We prove bounds on the additional total loss of the on-line algorithm over the total loss of the off-line algorithm. These relative loss bounds hold for an arbitrary sequence of examples. The goal is to design algorithms with the best possible relative loss bounds. We use a certain divergence to derive and analyze the algorithms. This divergence is a relative entropy between two exponential distributions.