Efficient Structured Surrogate Loss and Regularization in Structured Prediction

TL;DR

This paper introduces a new class of surrogate losses called bi-criteria surrogate loss for structured prediction, along with efficient inference methods, and proposes a normalization technique and shared Frobenius norm for hierarchical classification with structural imbalance.

Contribution

It develops a novel bi-criteria surrogate loss with efficient inference algorithms and introduces a shared Frobenius norm for better regularization in hierarchical classification.

Findings

Bi-criteria surrogate loss improves performance and efficiency.

Efficient inference algorithms reduce computational time.

Shared Frobenius norm effectively handles structural imbalance.

Abstract

In this dissertation, we focus on several important problems in structured prediction. In structured prediction, the label has a rich intrinsic substructure, and the loss varies with respect to the predicted label and the true label pair. Structured SVM is an extension of binary SVM to adapt to such structured tasks. In the first part of the dissertation, we study the surrogate losses and its efficient methods. To minimize the empirical risk, a surrogate loss which upper bounds the loss, is used as a proxy to minimize the actual loss. Since the objective function is written in terms of the surrogate loss, the choice of the surrogate loss is important, and the performance depends on it. Another issue regarding the surrogate loss is the efficiency of the argmax label inference for the surrogate loss. Efficient inference is necessary for the optimization since it is often the most time-consuming step. We present a new class of surrogate losses named bi-criteria surrogate loss, which is a generalization of the popular surrogate losses. We first investigate an efficient method for a slack rescaling formulation as a starting point utilizing decomposability of the model. Then, we extend the algorithm to the bi-criteria surrogate loss, which is very efficient and also shows performance improvements. In the second part of the dissertation, another important issue of regularization is studied. Specifically, we investigate a problem of regularization in hierarchical classification when a structural imbalance exists in the label structure. We present a method to normalize the structure, as well as a new norm, namely shared Frobenius norm. It is suitable for hierarchical classification that adapts to the data in addition to the label structure.