Non-Uniform Stochastic Average Gradient Method for Training Conditional Random Fields

TL;DR

This paper introduces a non-uniform stochastic average gradient method tailored for training conditional random fields, enhancing efficiency and convergence through structure-aware implementation and sampling strategies.

Contribution

It presents a practical, memory-efficient SAG algorithm with non-uniform sampling for CRFs, along with convergence analysis and empirical validation.

Findings

Significantly outperforms existing training methods in objective reduction

Performs as well or better than optimally-tuned stochastic gradient methods in test error

Improves practical training speed and convergence stability

Abstract

We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substantially improves practical performance, and analyze the rate of convergence of the SAGA variant under non-uniform sampling. Our experimental results reveal that our method often significantly outperforms existing methods in terms of the training objective, and performs as well or better than optimally-tuned stochastic gradient methods in terms of test error.