SPOLRE: Semantic Preserving Object Layout Reconstruction for Image Captioning System Testing

TL;DR

SPOLRE is an automated tool that reconstructs object layouts in images to test image captioning systems, producing realistic test images and effectively identifying caption errors with high precision.

Contribution

It introduces a novel automated method for semantic-preserving object layout reconstruction, reducing manual effort and improving realism in IC system testing.

Findings

SPOLRE-generated images are more realistic according to survey respondents.

SPOLRE detects more caption errors with higher precision than existing methods.

It identified 6,236 unique issues in Microsoft Azure's IC system.

Abstract

Image captioning (IC) systems, such as Microsoft Azure Cognitive Service, translate image content into descriptive language but can generate inaccuracies leading to misinterpretations. Advanced testing techniques like MetaIC and ROME aim to address these issues but face significant challenges. These methods require intensive manual labor for detailed annotations and often produce unrealistic images, either by adding unrelated objects or failing to remove existing ones. Additionally, they generate limited test suites, with MetaIC restricted to inserting specific objects and ROME limited to a narrow range of variations. We introduce SPOLRE, a novel automated tool for semantic-preserving object layout reconstruction in IC system testing. SPOLRE leverages four transformation techniques to modify object layouts without altering the image's semantics. This automated approach eliminates the need for manual annotations and creates realistic, varied test suites. Our tests show that over 75% of survey respondents find SPOLRE-generated images more realistic than those from state-of-the-art methods. SPOLRE excels in identifying caption errors, detecting 31,544 incorrect captions across seven IC systems with an average precision of 91.62%, surpassing other methods which average 85.65% accuracy and identify 17,160 incorrect captions. Notably, SPOLRE identified 6,236 unique issues within Azure, demonstrating its effectiveness against one of the most advanced IC systems.