Structured Prediction: From Gaussian Perturbations to Linear-Time Principled Algorithms

TL;DR

This paper introduces a PAC-Bayes framework analysis of a linear-time, sampling-based loss function for structured prediction, providing theoretical guarantees and explaining recent empirical successes.

Contribution

It offers a theoretical justification for using maximum loss over random outputs in structured prediction, showing it yields tighter bounds and is a principled learning method.

Findings

Tighter upper bounds on Gibbs decoder distortion with this loss.

The method is linear-time and easily parallelizable.

Theoretical insights explain empirical success of recent sampling-based approaches.

Abstract

Margin-based structured prediction commonly uses a maximum loss over all possible structured outputs \cite{Altun03,Collins04b,Taskar03}. In natural language processing, recent work \cite{Zhang14,Zhang15} has proposed the use of the maximum loss over random structured outputs sampled independently from some proposal distribution. This method is linear-time in the number of random structured outputs and trivially parallelizable. We study this family of loss functions in the PAC-Bayes framework under Gaussian perturbations \cite{McAllester07}. Under some technical conditions and up to statistical accuracy, we show that this family of loss functions produces a tighter upper bound of the Gibbs decoder distortion than commonly used methods. Thus, using the maximum loss over random structured outputs is a principled way of learning the parameter of structured prediction models. Besides explaining the experimental success of \cite{Zhang14,Zhang15}, our theoretical results show that more general techniques are possible.