Automatic Annotation of Structured Facts in Images

TL;DR

This paper introduces an automatic method for extracting structured visual facts from images with captions, enabling large-scale fact annotation with high accuracy and efficiency.

Contribution

The authors propose a novel language-based approach that automatically collects and localizes hundreds of thousands of visual facts from images with captions, significantly advancing data collection for image understanding.

Findings

Collected over 380,000 visual fact annotations

Achieved 83% accuracy in fact annotation

Processed data in less than one day on standard CPU

Abstract

Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., <flower, red>), actions (e.g., <baby, smile>), interactions (e.g., <man, walking, dog>), and positional information (e.g., <vase, on, table>). The collected annotations are in the form of fact-image pairs (e.g.,<man, walking, dog> and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83% according to human judgment. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms.