Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classification

TL;DR

This paper analyzes top-k error in large-scale image classification, compares loss functions, and explores transitioning from multiclass to multilabel learning, providing insights and efficient training algorithms.

Contribution

It offers a comprehensive analysis of top-k error, compares loss functions, and introduces efficient algorithms for top-k and multilabel training.

Findings

Softmax loss and multiclass SVM are competitive across all k.

Top-k loss functions improve specific k performance.

Multilabel classifiers can be effectively trained from single labels on Pascal VOC.

Abstract

Top-k error is currently a popular performance measure on large scale image classification benchmarks such as ImageNet and Places. Despite its wide acceptance, our understanding of this metric is limited as most of the previous research is focused on its special case, the top-1 error. In this work, we explore two directions that shed more light on the top-k error. First, we provide an in-depth analysis of established and recently proposed single-label multiclass methods along with a detailed account of efficient optimization algorithms for them. Our results indicate that the softmax loss and the smooth multiclass SVM are surprisingly competitive in top-k error uniformly across all k, which can be explained by our analysis of multiclass top-k calibration. Further improvements for a specific k are possible with a number of proposed top-k loss functions. Second, we use the top-k methods to explore the transition from multiclass to multilabel learning. In particular, we find that it is possible to obtain effective multilabel classifiers on Pascal VOC using a single label per image for training, while the gap between multiclass and multilabel methods on MS COCO is more significant. Finally, our contribution of efficient algorithms for training with the considered top-k and multilabel loss functions is of independent interest.