Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking

TL;DR

This paper introduces Pixel-Level Visual Entity Linking (PL-VEL), a new task that uses pixel masks for fine-grained visual understanding, supported by a large-scale dataset and a semantic tokenization method that improves accuracy.

Contribution

The paper proposes PL-VEL, a novel pixel-level entity linking task, and constructs the MaskOVEN-Wiki dataset with over 5 million annotations, along with a semantic tokenization approach for enhanced performance.

Findings

The reverse annotation framework achieved 94.8% success rate.

Models trained on the dataset improved accuracy by 18 points.

Semantic tokenization improved accuracy by 5 points.

Abstract

Visual Entity Linking (VEL) is a crucial task for achieving fine-grained visual understanding, matching objects within images (visual mentions) to entities in a knowledge base. Previous VEL tasks rely on textual inputs, but writing queries for complex scenes can be challenging. Visual inputs like clicks or bounding boxes offer a more convenient alternative. Therefore, we propose a new task, Pixel-Level Visual Entity Linking (PL-VEL), which uses pixel masks from visual inputs to refer to objects, supplementing reference methods for VEL. To facilitate research on this task, we have constructed the MaskOVEN-Wiki dataset through an entirely automatic reverse region-entity annotation framework. This dataset contains over 5 million annotations aligning pixel-level regions with entity-level labels, which will advance visual understanding towards fine-grained. Moreover, as pixel masks correspond to semantic regions in an image, we enhance previous patch-interacted attention with region-interacted attention by a visual semantic tokenization approach. Manual evaluation results indicate that the reverse annotation framework achieved a 94.8% annotation success rate. Experimental results show that models trained on this dataset improved accuracy by 18 points compared to zero-shot models. Additionally, the semantic tokenization method achieved a 5-point accuracy improvement over the trained baseline.