Fine-grained Generalization Analysis of Structured Output Prediction

TL;DR

This paper improves theoretical understanding of structured output prediction by providing new generalization bounds with logarithmic label set dependency and stability-based bounds independent of label size, applicable to large-scale problems.

Contribution

It introduces novel high-probability and expectation-based generalization bounds for SOPPs with large label sets, extending to weakly dependent data.

Findings

Logarithmic dependency on label set size in generalization bounds.

Stability-based bounds independent of label set size.

Extension to weakly dependent data scenarios.

Abstract

In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains where SOPPs naturally occur include natural language processing, speech recognition, and computer vision. Typical SOPPs have an extremely large label set, which grows exponentially as a function of the size of the output. Existing generalization analysis implies generalization bounds with at least a square-root dependency on the cardinality $d$ of the label set, which can be vacuous in practice. In this paper, we significantly improve the state of the art by developing novel high-probability bounds with a logarithmic dependency on $d$. Moreover, we leverage the lens of algorithmic stability to develop generalization bounds in expectation without any dependency on $d$. Our results therefore build a solid theoretical foundation for learning in large-scale SOPPs. Furthermore, we extend our results to learning with weakly dependent data.