# Webly-Supervised Image Manipulation Localization via Category-Aware Auto-Annotation

## TL;DR

This paper introduces a webly-supervised framework for image manipulation localization that leverages large-scale web data and auto-annotation techniques to significantly improve detection accuracy and address data scarcity issues.

## Contribution

It presents CAAAv2 for category-aware auto-annotation, QES for quality filtering, and Web-IML for leveraging web data, resulting in a large high-quality dataset and improved localization performance.

## Key findings

- Achieved a 31% performance gain over previous methods.
- Created MIMLv2, a dataset over 120 times larger than existing datasets.
- Surpassed state-of-the-art models by 21.6 IoU points.

## Abstract

Images manipulated by image editing tools can mislead viewers and pose significant risks to social security. However, accurately localizing manipulated image regions remains challenging due to the severe scarcity of high-quality annotated data, which is laborious to create. To address this, we propose a novel approach that mitigates data scarcity by leveraging readily available web data. We utilize a large collection of manually forged images from the web, as well as automatically generated annotations derived from a simpler auxiliary task, constrained image manipulation localization.Specifically, we introduce CAAAv2, a novel auto-annotation framework that operates on a category-aware, prior-feature-denoising paradigm that notably reduces task complexity. To further ensure annotation reliability, we propose QES, a novel metric that filters out low-quality annotations. Combining CAAAv2 and QES, we construct MIMLv2, a large-scale, diverse, and high-quality dataset containing 246,212 manually forged images with pixel-level mask annotations. This is over 120 times larger than existing handcrafted datasets like IMD20. Additionally, we introduce Object Jitter, a technique that further enhances model training by generating high-quality manipulation artifacts. Building on these advances, we develop Web-IML, a new model designed to effectively leverage web-scale supervision for the task of image manipulation localization. Extensive experiments demonstrate that our approach substantially alleviates the data scarcity problem and significantly improves the performance of various models on multiple real-world forgery benchmarks. With the proposed web supervision, our Web-IML achieves a striking performance gain of 31% and surpasses the previous state-of-the-art SparseViT by 21.6 average IoU points. The dataset and code will be released at https://github.com/qcf-568/MIML.