A Comprehensive Analysis of Real-World Image Captioning and Scene Identification

TL;DR

This paper evaluates the performance of various image captioning models on a new real-world dataset with diverse scene classes, highlighting challenges in real-world conditions often overlooked in controlled studies.

Contribution

It introduces a new dataset of over 800 images with detailed captions, and assesses multiple models' effectiveness in complex, real-world environments.

Findings

Models struggle with poor-quality images in real-world settings

The IC3 captioning approach provides more descriptive summaries

Performance varies significantly across different scene classes

Abstract

Image captioning is a computer vision task that involves generating natural language descriptions for images. This method has numerous applications in various domains, including image retrieval systems, medicine, and various industries. However, while there has been significant research in image captioning, most studies have focused on high quality images or controlled environments, without exploring the challenges of real-world image captioning. Real-world image captioning involves complex and dynamic environments with numerous points of attention, with images which are often very poor in quality, making it a challenging task, even for humans. This paper evaluates the performance of various models that are built on top of different encoding mechanisms, language decoders and training procedures using a newly created real-world dataset that consists of over 800+ images of over 65 different scene classes, built using MIT Indoor scenes dataset. This dataset is captioned using the IC3 approach that generates more descriptive captions by summarizing the details that are covered by standard image captioning models from unique view-points of the image.