Hybrid Generative/Discriminative Learning for Automatic Image Annotation

TL;DR

This paper introduces a hybrid generative-discriminative model for automatic image annotation that effectively handles data ambiguity and large label vocabularies, improving annotation accuracy and scalability.

Contribution

It proposes an Exponential-Multinomial Mixture model combined with discriminative learning to address ambiguity and overfitting in image annotation tasks.

Findings

Achieves superior annotation performance

Handles large label vocabularies effectively

Improves scalability of image annotation

Abstract

Automatic image annotation (AIA) raises tremendous challenges to machine learning as it requires modeling of data that are both ambiguous in input and output, e.g., images containing multiple objects and labeled with multiple semantic tags. Even more challenging is that the number of candidate tags is usually huge (as large as the vocabulary size) yet each image is only related to a few of them. This paper presents a hybrid generative-discriminative classifier to simultaneously address the extreme data-ambiguity and overfitting-vulnerability issues in tasks such as AIA. Particularly: (1) an Exponential-Multinomial Mixture (EMM) model is established to capture both the input and output ambiguity and in the meanwhile to encourage prediction sparsity; and (2) the prediction ability of the EMM model is explicitly maximized through discriminative learning that integrates variational inference of graphical models and the pairwise formulation of ordinal regression. Experiments show that our approach achieves both superior annotation performance and better tag scalability.