Adversarial Structured Prediction for Multivariate Measures

TL;DR

This paper introduces an adversarial training framework for structured prediction that directly optimizes multivariate performance measures like F-score and AER, addressing the limitations of surrogate loss methods.

Contribution

It proposes a novel adversarial approach that approximates training data while directly optimizing the true multivariate evaluation metrics for structured prediction tasks.

Findings

Effective for word alignment with AER

Improves named entity recognition performance

Addresses mismatch issues in surrogate loss methods

Abstract

Many predicted structured objects (e.g., sequences, matchings, trees) are evaluated using the F-score, alignment error rate (AER), or other multivariate performance measures. Since inductively optimizing these measures using training data is typically computationally difficult, empirical risk minimization of surrogate losses is employed, using, e.g., the hinge loss for (structured) support vector machines. These approximations often introduce a mismatch between the learner's objective and the desired application performance, leading to inconsistency. We take a different approach: adversarially approximate training data while optimizing the exact F-score or AER. Structured predictions under this formulation result from solving zero-sum games between a predictor seeking the best performance and an adversary seeking the worst while required to (approximately) match certain structured properties of the training data. We explore this approach for word alignment (AER evaluation) and named entity recognition (F-score evaluation) with linear-chain constraints.