Are Multimodal Large Language Models Good Annotators for Image Tagging?

TL;DR

This paper evaluates the potential of Multimodal Large Language Models (MLLMs) as automated image taggers, demonstrating significant cost savings and promising annotation quality, and introduces TagLLM, a framework to improve MLLM annotation accuracy for downstream tasks.

Contribution

The paper analyzes MLLM annotation capabilities, quantifies cost and quality gaps compared to humans, and proposes TagLLM, a novel framework to enhance MLLM-based image tagging accuracy.

Findings

MLLMs can reduce annotation costs to less than 0.1% of human effort.

MLLM annotations reach 50-80% of human quality.

TagLLM narrows the annotation gap, improving downstream task performance by 60-80%.

Abstract

Image tagging, a fundamental vision task, traditionally relies on human-annotated datasets to train multi-label classifiers, which incurs significant labor and costs. While Multimodal Large Language Models (MLLMs) offer promising potential to automate annotation, their capability to replace human annotators remains underexplored. This paper aims to analyze the gap between MLLM-generated and human annotations and to propose an effective solution that enables MLLM-based annotation to replace manual labeling. Our analysis of MLLM annotations reveals that, under a conservative estimate, MLLMs can reduce annotation cost to as low as one-thousandth of the human cost, mainly accounting for GPU usage, which is nearly negligible compared to manual efforts. Their annotation quality reaches about 50\% to 80\% of human performance, while achieving over 90\% performance on downstream training tasks.Motivated by these findings, we propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations. TagLLM comprises two components: Candidates generation, which employs structured group-wise prompting to efficiently produce a compact candidate set that covers as many true labels as possible while reducing subsequent annotation workload; and label disambiguation, which interactively calibrates the semantic concept of categories in the prompts and effectively refines the candidate labels. Extensive experiments show that TagLLM substantially narrows the gap between MLLM-generated and human annotations, especially in downstream training performance, where it closes about 60\% to 80\% of the difference.