Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation

TL;DR

This paper introduces SMILE, a novel training method for image captioning that encourages models to generate more detailed and descriptive captions by selectively optimizing for richness without penalizing conciseness.

Contribution

The paper proposes Semipermeable MaxImum Likelihood Estimation (SMILE), a new training approach that balances richness and conciseness to improve caption descriptiveness.

Findings

SMILE significantly improves caption richness on MSCOCO and Flickr30K datasets.

Models trained with SMILE produce longer, more detailed captions.

Extensive experiments validate the effectiveness of SMILE in enhancing caption quality.

Abstract

Image captioning aims to describe visual content in natural language. As 'a picture is worth a thousand words', there could be various correct descriptions for an image. However, with maximum likelihood estimation as the training objective, the captioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred to as conciseness optimization. In contrast, predictions that are more concise than labels lead to richness optimization. Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we introduce Semipermeable MaxImum Likelihood Estimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness of generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.