Webly-Supervised Image Manipulation Localization via Category-Aware Auto-Annotation

TL;DR

This paper introduces a webly-supervised framework for image manipulation localization that leverages large-scale web data and auto-annotation techniques to significantly improve detection accuracy and address data scarcity issues.

Contribution

It presents CAAAv2 for category-aware auto-annotation, QES for quality filtering, and Web-IML for leveraging web data, resulting in a large high-quality dataset and improved localization performance.

Findings

Achieved a 31% performance gain over previous methods.

Created MIMLv2, a dataset over 120 times larger than existing datasets.

Surpassed state-of-the-art models by 21.6 IoU points.

Abstract

Images manipulated by image editing tools can mislead viewers and pose significant risks to social security. However, accurately localizing manipulated image regions remains challenging due to the severe scarcity of high-quality annotated data, which is laborious to create. To address this, we propose a novel approach that mitigates data scarcity by leveraging readily available web data. We utilize a large collection of manually forged images from the web, as well as automatically generated annotations derived from a simpler auxiliary task, constrained image manipulation localization.Specifically, we introduce CAAAv2, a novel auto-annotation framework that operates on a category-aware, prior-feature-denoising paradigm that notably reduces task complexity. To further ensure annotation reliability, we propose QES, a novel metric that filters out low-quality annotations. Combining CAAAv2 and QES, we construct MIMLv2, a large-scale, diverse, and high-quality dataset containing 246,212 manually forged images with pixel-level mask annotations. This is over 120 times larger than existing handcrafted datasets like IMD20. Additionally, we introduce Object Jitter, a technique that further enhances model training by generating high-quality manipulation artifacts. Building on these advances, we develop Web-IML, a new model designed to effectively leverage web-scale supervision for the task of image manipulation localization. Extensive experiments demonstrate that our approach substantially alleviates the data scarcity problem and significantly improves the performance of various models on multiple real-world forgery benchmarks. With the proposed web supervision, our Web-IML achieves a striking performance gain of 31% and surpasses the previous state-of-the-art SparseViT by 21.6 average IoU points. The dataset and code will be released at https://github.com/qcf-568/MIML.

Tables

Table 1. TABLE I: Comparison between different handcrafted image manipulation localization datasets. ’Num.’ denotes number.

Name		Year		Real Num.		Forged Num.		(Height, Width) Range
CASIAv1 [46]		2013		800		921		(246, 384)-(500, 334)
CASIAv2 [46]		2013		7,461		5,123		(160, 240)-(901, 600)
Coverage [47]		2016		100		100		(190, 334)-(472, 752)
NIST16 [48]		2016		224		564		(500, 500)-(3744, 5616)
In Wild [55]		2018		0		201		(650, 650)-(2736, 3648)
MISD [51]		2021		618		300		(160, 240)-(901, 600)
IMD20 [45]		2020		414		2,010		(193, 260)-(4437, 2958)
CIMD [50]		2024		102		102		(2048, 1365)-(2048, 1365)
MIMLv1 [24]		2024		0		123,150		(45, 120)-(13846, 9200)
MIMLv2 (Ours)		2025		63,847		246,212		(45, 120)-(13846, 9200)

Table 2. TABLE II: Comparison on image manipulation localization. ‘WS‘ is our proposed web supervision, which trains models using data from the MIMLv2 and Object Jitter. ‘Avg*‘ is the average score excluding IMD20. ‘Param.‘ is the number of model parameters. CAT-Net, TruFor and SparseViT were trained with the IMD20 dataset and therefore not tested on it.

Method		CASIAv1 [46]				Coverage [47]				NIST16 [48]				IMD20 [45]				CocoGlide [49]				CIMD [50]				MISD [51]				Avg.*				Average				Param.
		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1
ManTra-Net [59]		.086		.130		.181		.271		.040		.062		.098		.146		.155		.203		.058		.096		.105		.171		.104		.156		.103		.154		8M
RRU-Net [60]		.330		.380		.165		.260		.080		.129		.169		.256		.223		.304		.036		.068		.440		.570		.212		.285		.206		.281		5M
MVSS-Net [35]		.403		.435		.389		.454		.243		.294		.200		.260		.276		.357		.016		.021		.120		.175		.235		.284		.235		.285		146M
PSCC-Net [56]		.410		.463		.340		.446		.067		.110		.115		.192		.333		.422		.212		.290		.400		.491		.294		.370		.268		.345		4M
CAT-Net [19]		.684		.738		.238		.292		.238		.302		-		-		.290		.366		.282		.344		.314		.394		.341		.406		-		-		114M
IF-OSN [61]		.465		.509		.181		.268		.247		.326		.259		.364		.207		.264		.195		.295		.375		.464		.278		.354		.341		.406		124M
EVP [62]		.438		.502		.078		.114		.188		.239		.177		.268		.084		.118		.224		.283		.392		.531		.234		.298		.276		.356		63M
TruFor [49]		.630		.692		.446		.522		.279		.348		-		-		.294		.362		.322		.418		.426		.549		.400		.482		-		-		67M
SparseViT [18]		.768		.819		.456		.504		.322		.381		-		-		.327		.389		.257		.322		.421		.538		.425		.492		-		-		50M
PIM [1]		.512		.566		.188		.251		.225		.280		.340		.419		.327		.404		.177		.229		.389		.519		.303		.375		.308		.381		168M
APSC-Net [24] (Conf.)		.799		.837		.490		.523		.398		.436		.339		.391		.243		.283		.517		.565		.420		.524		.477		.528		.458		.508		143M
Web-IML (Ours)		.823		.857		.508		.551		.412		.465		.370		.422		.285		.339		.629		.673		.430		.545		.515		.572		.494		.550		140M
APSC-Net (w/ MIMLv1)		.810		.848		.498		.568		.525		.590		.679		.760		.292		.355		.652		.720		.484		.587		.543		.611		.562		.632		143M
Web-IML (w/ MIMLv1)		.821		.854		.561		.616		.520		.587		.692		.768		.275		.325		.789		.845		.496		.610		.578		.640		.594		.658		140M
APSC-Net (w/ WS)		.832		.865		.594		.642		.578		.661		.690		.766		.389		.451		.660		.725		.485		.588		.598		.655		.604		.671		143M
Web-IML (w/ WS)		.846		.879		.715		.768		.591		.670		.698		.776		.439		.507		.759		.821		.495		.612		.641		.709		.649		.719		140M

Table 3. TABLE III: Ablation study on our Web-IML model. ‘MP‘ is the Multi-Scale Perception module. ‘SR‘ is the Self-Rectification module. ‘NA‘ is our Nested Channel Attention. ‘RF‘ denotes the second round self-rectification. ‘WS‘ denotes the proposed Web Supervision.

Num.		Ablation Settings										CASIAv1 [46]				Coverage [47]				NIST16 [48]				IMD20 [45]				CocoGlide [49]				CIMD [50]				MISD [51]				Average
		MP		SR		NA		RF		WS		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1
(1)		$\times$		$\times$		$\times$		$\times$		$\times$		.711		.779		.361		.430		.346		.410		.273		.342		.345		.417		.262		.338		.362		.471		.380		.455
(2)		✓		$\times$		$\times$		$\times$		$\times$		.775		.814		.440		.487		.359		.415		.351		.406		.275		.324		.450		.502		.375		.482		.432		.490
(3)		✓		✓		$\times$		$\times$		$\times$		.802		.836		.523		.576		.385		.441		.364		.415		.287		.343		.557		.610		.382		.493		.471		.530
(4)		✓		✓		✓		$\times$		$\times$		.816		.848		.549		.598		.394		.449		.369		.419		.291		.345		.593		.649		.386		.496		.486		.543
(5)		✓		✓		✓		✓		$\times$		.823		.857		.508		.551		.412		.465		.370		.422		.285		.339		.629		.673		.430		.545		.494		.550
(6)		✓		✓		✓		✓		✓		.846		.879		.715		.768		.591		.670		.698		.776		.439		.507		.759		.821		.496		.612		.649		.719

Table 4. TABLE IV: Robustness evaluation with AUC metric on NIST16 dataset.

				Resize			Blur			JPEG
Method		Ori		.78x	.25x		k=3	k=15		q=100	q=50
ManTra-Net [59]		.795		.774	.755		.774	.746		.779	.744
SPAN [63]		.840		.832	.802		.831	.792		.836	.807
PSCC-Net [56]		.855		.853	.850		.854	.800		.854	.854
ObjectFormer [37]		.872		.872	.863		.860	.803		.864	.862
SparseViT [18]		.888		.884	.869		.881	.877		.886	.881
NCL [17]		.912		.856	.831		.840	.806		.843	.819
ERMPC [64]		.895		.893	.877		.892	.871		.894	.888
UnionFormer [65]		.881		.873	.872		.865	.843		.880	.880
APSC-Net+MIMLv1		.928		.917	.888		.907	.900		.922	.907
Web-IML (Ours)		.942		.935	.904		.914	.900		.940	.909

Table 5. TABLE V: Ablation study on our Web Supervision method. ‘PSCC-Synthetic‘ denotes the synthetic data from the PSCC-Net  [ 56 ] . ‘O.J.‘ denotes the proposed Object Jitter method. ‘C.G.‘ denotes the CocoGlide  [ 49 ] . ‘ Δ \Delta ‘ denotes improvement.

				CASIAv1 [46]				Coverage [47]				NIST16 [48]				IMD20 [45]				C.G. [49]				CIMD [50]				MISD [51]				Average (#$\Delta$)
Num.		Training Data		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1
(1)		Web-IML Baseline		.823		.857		.508		.551		.412		.465		.370		.422		.285		.339		.629		.673		.430		.545		.494		.550
(2)		+PSCC-Synthetic [56]		.813		.845		.503		.546		.348		.392		.361		.410		.177		.211		.598		.654		.330		.441		.447 (-9.5%)		.500 (-9.1%)
(3)		+DEFACTO [20]		.809		.843		.354		.380		.412		.465		.365		.424		.359		.418		.394		.438		.461		.572		.451 (-8.7%)		.506 (-8.0%)
(4)		+AIGC-Synthetic [21]		.817		.849		.273		.296		.220		.276		.284		.337		.394		.462		.287		.340		.373		.429		.376 (-24.%)		.420 (-23.%)
(5)		+WSCL [66]		.815		.849		.467		.510		.415		.468		.411		.464		.293		.347		.630		.672		.431		.544		.495 (+0.2%)		.551 (+0.2%)
(6)		+MIMLv1 [24] (Conf.)		.821		.854		.561		.616		.520		.587		.692		.768		.275		.325		.789		.845		.496		.610		.594 (+20.%)		.658 (+20.%)
(7)		+MIMLv2		.849		.883		.694		.755		.518		.584		.697		.772		.376		.435		.716		.776		.448		.564		.614 (+24.%)		.681 (+24.%)
(8)		+MIMLv1 [24]+O.J.		.833		.867		.587		.643		.549		.620		.693		.768		.344		.407		.793		.849		.487		.602		.612 (+24.%)		.679 (+23.%)
(9)		+O.J.		.822		.853		.575		.616		.270		.310		.422		.482		.308		.362		.698		.760		.413		.521		.501 (+1.4%)		.558 (+1.4%)
(10)		+MIMLv2+O.J. (Ours)		.846		.879		.715		.768		.591		.670		.698		.776		.439		.507		.759		.821		.496		.612		.649 (+31.%)		.719 (+31.%)

Table 6. TABLE VI: Ablation study on our Object Jitter method. ‘COCO‘ denotes using the COCO  [ 67 ] images for Object Jitter. ‘SA‘ denotes using 400k random images from the SA-1B  [ 57 ] dataset for Object Jitter. ‘C.G.‘ denotes the CocoGlide dataset  [ 49 ] .

Num		Object Jitter Operation						Image Source		CASIAv1 [46]				Coverage [47]				NIST16 [48]				IMD20 [45]				C.G. [49]				CIMD [50]				MISD [51]				Average
		Size		Texture		Exposure				IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1		IoU		F1
(1)		$\times$		$\times$		$\times$		SA		.849		.883		.694		.755		.518		.584		.697		.772		.376		.435		.716		.776		.448		.564		.614		.681
(2)		$\times$		✓		✓		SA		.839		.872		.634		.687		.528		.592		.703		.778		.396		.456		.748		.798		.487		.605		.619		.684
(3)		✓		$\times$		✓		SA		.844		.878		.669		.729		.554		.618		.701		.778		.427		.491		.780		.834		.501		.616		.640		.706
(4)		✓		✓		$\times$		SA		.839		.872		.699		.754		.560		.627		.699		.774		.413		.479		.802		.861		.512		.628		.646		.714
(5)		✓		$\times$		$\times$		SA		.841		.875		.673		.732		.540		.602		.701		.777		.432		.496		.758		.816		.516		.631		.637		.704
(6)		✓		✓		✓		COCO		.848		.881		.645		.711		.584		.657		.699		.776		.489		.549		.803		.864		.496		.612		.652		.721
(7)		✓		✓		✓		SA		.846		.879		.715		.768		.591		.670		.698		.776		.439		.507		.759		.821		.496		.613		.649		.719

Table 7. TABLE VII: Comparison study of our web-supervision on IML task. Blue number for improvement and Red number for decrease . ‘B.L.‘ denotes baseline setting. ‘PSCC-Synth.‘ denotes the synthetic dataset of PSCC-Net  [ 56 ] , ‘AIGC-Synth.‘ denotes the AIGC forgery dataset OpenSDID  [ 21 ] . ‘O.J.‘ denotes our Object Jitter method.

Training Data		CASIAv1 [46]								Coverage [47]								NIST16 [48]								IMD20 [45]								CIMD [50]								Average
		IoU		$\Delta$		F1		$\Delta$		IoU		$\Delta$		F1		$\Delta$		IoU		$\Delta$		F1		$\Delta$		IoU		$\Delta$		F1		$\Delta$		IoU		$\Delta$		F1		$\Delta$		IoU		$\Delta$		F1		$\Delta$
PSCC-Net B.L.		.401		0		.430		0		.197		0		.218		0		.247		0		.295		0		.125		0		.156		0		.161		0		.221		0		.226		0		.264		0
+PSCC-Synth.		.352		12%		.379		12%		.147		25%		.159		27%		.230		7%		.274		7%		.101		19%		.125		20%		.101		37%		.124		44%		.186		18%		.212		20%
+DEFACTO[20]		.394		2%		.429		1%		.231		17%		.256		17%		.223		10%		.270		8%		.157		26%		.171		10%		.120		25%		.159		28%		.225		0%		.257		3%
+AIGC-Synth.		.346		16%		.377		14%		.109		81%		.123		77%		.091		171%		.142		108%		.110		14%		.144		8%		.124		30%		.170		30%		.156		45%		.191		38%
+WSCL [66]		.439		9%		.464		8%		.201		2%		.220		1%		.253		2%		.300		2%		.192		54%		.235		51%		.177		10%		.239		8%		.252		12%		.293		11%
+MIMLv1 (Conf.)		.609		52%		.649		51%		.395		101%		.477		119%		.402		63%		.476		61%		.470		276%		.541		247%		.206		28%		.251		14%		.416		84%		.479		81%
+MIMLv2		.602		50%		.642		49%		.402		104%		.498		128%		.405		64%		.485		64%		.575		360%		.662		324%		.354		120%		.483		119%		.468		107%		.554		110%
+MIMLv2+O.J		.609		52%		.651		51%		.451		129%		.550		152%		.406		64%		.486		65%		.570		356%		.658		322%		.335		108%		.464		110%		.474		110%		.562		113%
CAT-Net B.L.		.660		0		.703		0		.245		0		.286		0		.239		0		.287		0		.157		0		.192		0		.229		0		.263		0		.306		0		.346		0
+PSCC-Synth.		.410		38%		.437		38%		.215		12%		.232		19%		.288		21%		.331		15%		.121		23%		.148		23%		.160		30%		.189		28%		.239		22%		.268		23%
+DEFACTO[20]		.673		2%		.715		2%		.200		18%		.230		20%		.220		8%		.261		9%		.164		4%		.200		4%		.130		43%		.156		41%		.277		9%		.312		10%
+AIGC-Synth.		.646		2%		.685		3%		.136		80%		.147		95%		.170		41%		.242		19%		.130		21%		.165		16%		.088		160%		.117		125%		.234		31%		.271		28%
+WSCL [66]		.668		1%		.710		1%		.240		2%		.281		2%		.245		3%		.292		2%		.204		30%		.246		29%		.238		4%		.273		4%		.319		4%		.361		4%
+MIMLv1 (Conf.)		.691		5%		.728		4%		.302		23%		.389		36%		.353		48%		.422		47%		.547		248%		.629		228%		.397		73%		.477		81%		.458		50%		.529		53%
+MIMLv2		.684		4%		.720		2%		.368		50%		.443		55%		.338		41%		.416		45%		.562		258%		.644		235%		.470		105%		.548		108%		.484		58%		.554		60%
+MIMLv2+O.J		.703		7%		.742		6%		.355		45%		.437		53%		.358		50%		.440		53%		.575		266%		.656		242%		.497		117%		.575		119%		.498		63%		.570		65%
TruFor B.L.		.755		0		.794		0		.363		0		.411		0		.320		0		.377		0		.248		0		.289		0		.296		0		.346		0		.396		0		.443		0
+PSCC-Synth.		.750		1%		.798		1%		.340		7%		.384		7%		.325		2%		.373		1%		.253		2%		.298		3%		.278		6%		.331		5%		.389		2%		.437		2%
+DEFACTO[20]		.752		0%		.791		0%		.322		13%		.357		15%		.214		50%		.260		45%		.209		19%		.247		17%		.184		61%		.217		59%		.336		18%		.374		18%
+AIGC-Synth.		.746		1%		.791		0%		.220		65%		.261		57%		.213		50%		.263		43%		.221		12%		.267		8%		.121		146%		.156		122%		.304		30%		.348		28%
+WSCL [66]		.758		1%		.797		1%		.362		0%		.410		0%		.326		1%		.373		1%		.307		24%		.352		22%		.310		5%		.359		4%		.412		4%		.458		3%
+MIMLv1 (Conf.)		.755		0%		.794		0%		.385		6%		.458		10%		.352		9%		.408		8%		.596		158%		.678		157%		.340		13%		.401		14%		.486		18%		.548		19%
+MIMLv2		.767		2%		.794		0%		.387		6%		.458		10%		.407		21%		.472		20%		.618		160%		.696		158%		.370		20%		.422		18%		.510		22%		.568		22%
+MIMLv2+O.J		.770		2%		.811		2%		.466		22%		.511		20%		.439		27%		.502		25%		.619		160%		.700		159%		.309		4%		.357		3%		.521		24%		.576		23%
SparseViT B.L.		.702		0		.747		0		.320		0		.352		0		.288		0		.343		0		.199		0		.239		0		.177		0		.232		0		.337		0		.383		0
+PSCC-Synth.		.627		11%		.677		9%		.304		5%		.333		5%		.256		11%		.311		9%		.196		1%		.243		1%		.169		5%		.232		0%		.310		8%		.359		6%
+DEFACTO[20]		.692		1%		.730		2%		.257		20%		.283		20%		.310		8%		.364		6%		.186		7%		.225		6%		.155		12%		.198		15%		.320		5%		.362		6%
+AIGC-Synth.		.533		24%		.583		22%		.119		63%		.141		60%		.133		54%		.174		49%		.088		56%		.110		54%		.033		81%		.045		81%		.181		46%		.211		45%
+WSCL [66]		.709		1%		.755		1%		.324		1%		.356		1%		.301		4%		.352		3%		.267		34%		.314		31%		.184		4%		.241		4%		.357		6%		.404		6%
+MIMLv1 (Conf.)		.714		2%		.762		2%		.394		23%		.458		30%		.306		6%		.359		5%		.535		169%		.617		158%		.215		21%		.270		16%		.433		28%		.493		29%
+MIMLv2		.723		3%		.771		3%		.441		38%		.509		45%		.338		17%		.396		15%		.544		173%		.624		161%		.234		32%		.289		25%		.456		35%		.518		35%
+MIMLv2+O.J		.720		3%		.767		3%		.477		49%		.547		55%		.346		20%		.409		19%		.551		178%		.633		165%		.252		42%		.325		40%		.469		40%		.536		40%
IMDPrompter B.L.		.718		0		.765		0		.463		0		.522		0		.346		0		.419		0		.261		0		.310		0		.354		0		.414		0		.428		0		.486		0
+PSCC-Synth.		.682		5%		.729		5%		.473		2%		.531		2%		.295		15%		.356		15%		.262		0%		.312		1%		.286		19%		.337		19%		.400		7%		.453		7%
+DEFACTO[20]		.701		2%		.749		2%		.439		5%		.508		3%		.284		18%		.347		17%		.243		7%		.286		8%		.281		21%		.329		21%		.390		9%		.444		9%
+AIGC-Synth.		.630		12%		.678		11%		.254		45%		.289		45%		.218		37%		.263		37%		.222		15%		.264		15%		.263		26%		.309		25%		.317		26%		.361		26%
+WSCL [66]		.701		2%		.750		2%		.485		5%		.543		4%		.360		4%		.433		3%		.285		9%		.336		8%		.356		1%		.417		1%		.437		2%		.496		2%
+MIMLv1 (Conf.)		.724		1%		.769		1%		.542		17%		.610		17%		.418		21%		.501		20%		.601		130%		.696		125%		.392		11%		.456		10%		.535		25%		.606		25%
+MIMLv2		.729		2%		.773		1%		.564		22%		.635		22%		.426		23%		.512		22%		.614		135%		.708		128%		.418		18%		.490		18%		.550		29%		.624		28%
+MIMLv2+O.J		.735		2%		.778		2%		.626		35%		.706		35%		.433		25%		.521		24%		.635		143%		.721		133%		.425		20%		.498		20%		.571		33%		.645		33%

Table 8. TABLE VIII: Comparison study of CIML on SPG (left) and SDG (right). ’*’ denotes model trained with both SDG and SPG data. ’Nonzero’ denotes using the non-zero region of the difference map as mask. ’QES’ denotes the predictions filtered by our QES metric.

CIML on SPG										CIML on SDG
Method		P		R		F		IoU		Method		P		R		F		IoU
Nonzero		.159		.987		.273		.079		DMVN* [40]		.553		.357		.434		.276
OTSU [43]		.812		.589		.683		.497		DMVN [40]		.438		.568		.495		.317
DMVN* [40]		.329		.618		.430		.276		DMAC* [71]		.583		.538		.559		.410
DMVN [40]		.682		.781		.728		.578		DMAC [71]		.703		.622		.660		.518
DMAC* [71]		.617		.624		.620		.432		SACM [24] (Conf.)		.778		.819		.798		.702
DMAC [71]		.746		.710		.728		.573		Corr-DINO (Ours)		.816		.862		.838		.744
DASS (Ours)		.860		.921		.889		.835		Corr-DINO (QES)		.939		.971		.954		.912

Table 9. TABLE IX: Ablation study of CIML on SPG. ’Forged’ denotes using the forged image as input, ’Auth.’ denotes using the authentic image as input, ’Diff.’ denotes using the difference map between the forged and authentic image as input.

Num.		Backbone		Forged		Auth.		Diff.		P		R		F1		IoU
(1)		VGG [72]		✓		$\times$		$\times$		.423		.365		.392		.236
(2)		VGG [72]		✓		✓		$\times$		.800		.857		.827		.718
(3)		VGG [72]		$\times$		$\times$		✓		.771		.857		.812		.741
(4)		VGG [72]		✓		✓		✓		.825		.899		.860		.781
(5)		VAN [44]		✓		$\times$		$\times$		.573		.533		.552		.408
(6)		VAN [44]		✓		✓		✓		.860		.921		.889		.835

Table 10. TABLE X: Ablation study of CIML on SDG. ‘Freeze‘ denotes freezing the backbone and ‘L‘ denotes LoRA  [ 73 ] fine-tuning. ‘MAD‘ denotes the Multi-Aspect Denoiser. ‘FSR‘ denotes the Feature Super Resolution. ‘LA‘ denotes the Learnable Aggregation. ‘QES‘ denotes filtering the prediction with QES.

Num.		Backbone		Freeze		MAD		FSR		LA		QES		P		R		F1		IoU
(1)		ConvNeXt-B [68]		$\times$		✓		✓		✓		$\times$		.740		.690		.714		.595
(2)		ConvNeXt-B [68]		✓		✓		✓		✓		$\times$		.635		.554		.592		.414
(3)		Eff.SAM [75]		$\times$		✓		✓		✓		$\times$		.483		.624		.544		.381
(4)		Eff.SAM [75]		✓		✓		✓		✓		$\times$		.606		.502		.549		.403
(5)		CLIP-L [76]		$\times$		✓		✓		✓		$\times$		.665		.509		.577		.453
(6)		CLIP-L [76]		L		✓		✓		✓		$\times$		.744		.736		.740		.587
(7)		CLIP-L [76]		✓		✓		✓		✓		$\times$		.749		.721		.735		.596
(8)		DINOv1-B [77]		$\times$		✓		✓		✓		$\times$		.693		.512		.589		.456
(9)		DINOv1-B [77]		L		✓		✓		✓		$\times$		.693		.742		.717		.595
(10)		DINOv1-B [77]		✓		✓		✓		✓		$\times$		.700		.700		.700		.556
(11)		DINOv2-S [42]		$\times$		✓		✓		✓		$\times$		.739		.653		.693		.584
(12)		DINOv2-S [42]		L		✓		✓		✓		$\times$		.783		.733		.758		.637
(13)		DINOv2-S [42]		✓		✓		✓		✓		$\times$		.777		.781		.772		.667
(14)		DINOv2-B [42]		$\times$		✓		✓		✓		$\times$		.705		.574		.633		.472
(15)		DINOv2-B [42]		L		✓		✓		✓		$\times$		.794		.861		.826		.727
(16)		DINOv2-B [42]		✓		✓		✓		✓		$\times$		.816		.862		.838		.744
(17)		DINOv2-L [42]		$\times$		✓		✓		✓		$\times$		.667		.517		.582		.454
(18)		DINOv2-L [42]		L		✓		✓		✓		$\times$		.803		.843		.823		.725
(19)		DINOv2-L [42]		✓		✓		✓		✓		$\times$		.809		.850		.829		.732
(20)		DINOv3-B [78]		$\times$		✓		✓		✓		$\times$		.692		.588		.636		.478
(21)		DINOv3-B [78]		L		✓		✓		✓		$\times$		.690		.753		.720		.598
(22)		DINOv3-B [78]		✓		✓		✓		✓		$\times$		.709		.787		.746		.605
(23)		DINOv3-L [78]		$\times$		✓		✓		✓		$\times$		.718		.593		.650		.496
(24)		DINOv3-L [78]		✓		✓		✓		✓		$\times$		.722		.791		.755		.621
(25)		DINOv2-B [42]		✓		$\times$		$\times$		$\times$		$\times$		.644		.605		.624		.469
(26)		DINOv2-B [42]		✓		$\times$		✓		✓		$\times$		.699		.666		.682		.526
(27)		DINOv2-B [42]		✓		✓		$\times$		✓		$\times$		.822		.728		.772		.664
(28)		DINOv2-B [42]		✓		✓		✓		$\times$		$\times$		.804		.838		.821		.719
(29)		DINOv2-B [42]		✓		✓		✓		✓		$\times$		.816		.862		.838		.744
(30)		DINOv2-B [42]		✓		✓		✓		✓		✓		.939		.971		.954		.912

Table 11. TABLE XI: Ablation study on the threshold of QES.

QES Threshold		0.0		0.1		0.2		0.3		0.4		0.5		0.6		0.7		0.8		0.9
kept sample ratio		2.45		1.90		1.84		1.72		1.51		1.00		0.73		0.50		0.31		0.06
IMD20 SDG IoU		.744		.825		.842		.865		.889		.912		.924		.957		.967		.968

Table 12. TABLE XII: Classification results for SDG, SPG and misaligned SPG. ‘Ensemble‘ denotes the ensemble of the models.

method		SDG				SPG				Misaligned
		P	R	F		P	R	F		P	R	F
DiNAT [79]		.992	.992	.992		1	.996	.998		.867	.929	.897
SwinTrans [80]		.996	.981	.988		1	.996	.998		.737	1	.849
ConvNeXt [68]		.996	.992	.994		1	.996	.998		.875	1	.933
Ensemble		.996	1	.998		1	.996	.998		1	1	1

Table 13. TABLE XIII: Image manipulation localization performance on SACP  [ 81 ] dataset. ‘Forgery Baseline‘ denotes using the tampCOCO  [ 19 ] and CASIAv2  [ 46 ] datasets for model pretraining

Num.		Methods		Pretraining data		IoU		F1
(1)		DFCN [82]		ImageNet [83]		.466		.607
(2)		MVSS-Net [35]		ImageNet [83]		.401		.534
(3)		PSCC-Net [56]		ImageNet [83]		.482		.620
(4)		RRU-Net [60]		ImageNet [83]		.517		.651
(5)		CFL-Net [84]		ImageNet [83]		.433		.571
(6)		DTD [85]		ImageNet [83]		.588		.712
(7)		TIFDM [86]		ImageNet [83]		.576		.703
(8)		Web-IML (Ours)		ImageNet [83]		.773		.862
(9)		Web-IML (Ours)		ADE20k [87]		.774		.863
(10)		Web-IML (Ours)		Forgery Baseline		.778		.865
(11)		Web-IML (Ours)		(10) + DEFACTO [20]		.767		.856
(12)		Web-IML (Ours)		(10) + PSCC-Synthetic [56]		.762		.853
(13)		Web-IML (Ours)		(10) + AIGC Synthetic [21]		.756		.851
(14)		Web-IML (Ours)		(10) + MIMLv1 [24] (Conference)		.792		.872
(15)		Web-IML (Ours)		(10) + Web Supervision (Ours)		.800		.880

Table 14. TABLE XIV: Image manipulation localization performance on RTM  [ 88 ] dataset. ‘Forgery Baseline‘ denotes using the tampCOCO  [ 19 ] and CASIAv2  [ 46 ] datasets for model pretraining.

Num.		Methods		Pretraining data		IoU		F1
(1)		RRU-Net [60]		ImageNet [83]		.037		.072
(2)		PSCC-Net [56]		ImageNet [83]		.033		.064
(3)		MVSS-Net [35]		ImageNet [83]		.051		.097
(4)		CAT-Net [19]		ImageNet [83]		.113		.203
(5)		DTD [85]		ImageNet [83]		.065		.122
(6)		Mask2Former [89]		ADE20k [87]		.124		.172
(7)		ASCFormer [88]		ADE20k [87]		.198		.329
(8)		Web-IML (Ours)		ImageNet [83]		.192		.323
(9)		Web-IML (Ours)		ADE20k [87]		.207		.343
(10)		Web-IML (Ours)		Forgery Baseline		.194		.325
(11)		Web-IML (Ours)		(10) + DEFACTO [20]		.186		.313
(12)		Web-IML (Ours)		(10) + PSCC-Synthetic [56]		.189		.318
(13)		Web-IML (Ours)		(10) + AIGC Synthetic [21]		.185		.311
(14)		Web-IML (Ours)		(10) + MIMLv1 [24]		.226		.369
(15)		Web-IML (Ours)		(10) + Web Supervision (Ours)		.241		.389

Full text

††footnotetext: This research is supported in part by NSFC (Grant No.: 62476093) and GD-NSF (No.2021A1515011870). (Corresponding author: Lianwen Jin.) Chenfan Qu, Huiguo He, and Lianwen Jin are with the School of Electronics and Information Engineering, South China University of Technology, Guangdong 510641, China. (Email: 202221012612@mail.scut.edu.cn, hehuiguo@scut.edu.cn, eelwjin@scut.edu.cn) Yiwu Zhong is with the School of Intelligence Science and Technology, Peking University, Beijing 100091, China. (Email: yiwu-zhong@outlook.com) Bin Li is with the College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, China. (Email: libin@szu.edu.cn)

I Introduction

The misuse of manipulated images can lead to fraud and the spread of misinformation, posing serious threats to social media security [1, 2, 3, 4, 5, 6, 7, 8]. To this end, Image Manipulation Localization (IML) has garnered increasing attention in recent years [9, 10, 11, 12, 13, 14, 15, 16]. A central challenge in IML lies in the scarcity of large-scale high-quality data [17], which is essential for preventing model overfitting. Creating such data is extremely labor-intensive and time-consuming [18], requiring not only the crafting of convincing forgeries but also the precise pixel-level annotation of manipulated regions. Although alternatives like synthetic data [19, 20] or AI-generated forgeries [21] have been explored, they often exhibit significant discrepancies from real-world manipulations, thereby limiting model generalization to practical scenarios [17].

Meanwhile, numerous unlabeled, manually forged images are publicly available over the Internet, often accompanied by their corresponding originals. To address the scarcity of annotated data, we propose to leverage these unlabeled forgeries by automatically generating pixel-level mask annotations using Constrained Image Manipulation Localization (CIML) models. Our key insight is that CIML can reduce the complexity of manipulation localization by comparing the forged image with its original, resulting in more accurate and reliable annotations. Therefore, CIML offers a practical solution for automatically generating large-scale, high-quality training datasets for IML.

However, despite recent progress with CIML on simple synthetic images, existing methods are unreliable for complex, real-world forgeries due to two major limitations:

First, previous methods employ a suboptimal paradigm that relies on a single correlation-based model for all input types [22, 23]. In practice, image pairs consisting of forged images and their original counterparts can be categorized into two distinct groups (see Fig. 1): (1) Shared Donor Group (SDG) denotes image pairs that share foreground, with forged images generated by copying objects from the original image; (2) Shared Probe Group (SPG) denotes image pairs that share background, where forged images are obtained by modifying the original image in-place. While correlation-based methods perform reasonably well on SDG data, they falter on SPG data because the shared region (the background) often lacks distinctive features. Forcing a single model to handle both groups leads to confusion and degraded performance.

Second, the CIML task itself is challenging and also suffers from a scarcity of high-quality training data. Previous works provide the model with only the image pair, relying entirely on the network to learn the complex correspondence between the images in each pair. Without clear and explicit guidance, models struggle to learn this correspondence effectively and are prone to overfitting on limited training data.

To overcome these challenges, we propose a novel paradigm: Category-Aware Auto-Annotation v2 (CAAAv2). It achieves superior generalization via two key innovations:

(1) Category awareness. We treat image pairs in SDG and SPG separately through a classifier at start. This separation avoids the confusion of joint training and allows us to deploy distinct, optimized pipelines that leverage the unique correspondence of each category. This classifier can be effectively trained on unlabeled images.

(2) Task transformation. We reframe the challenging CIML problem as a far simpler denoising task. This is achieved by introducing novel prior features that provide strong initial guidance for localization. Crucially, these prior features are extracted via zero-shot frozen functions that are resistant to overfitting. For SDG pairs, where the forged regions are typically shared objects, we extract features from both images using a frozen DINOv2 encoder and compute their correlation map. The resulting map, named DINO correlation, serves as a prior that roughly highlights the manipulated regions (Fig. 1). The model’s task is reduced to simply denoising this prior to produce a precise mask. For SPG pairs, where the forged regions are the difference between the images, we obtain the prior map by subtracting the forged image from its original. Through task transformation, we significantly reduce task difficulty and bridge the gap between synthetic and real-world data, leading to substantially better generalization.

Subsequently, we collect a large amount of manually forged images from the Internet and then annotate their forged regions with the proposed CAAAv2. This approach considerably alleviates the data scarcity in image manipulation localization. To ensure the reliability of the annotations, we propose Quality Evaluation Score (QES), a novel metric that automatically assesses and filters out inadequate annotations.

While this pipeline yields a high-quality dataset, we observed a relative lack of sophisticated copy-move forgeries that feature only subtle edge artifacts. Direct copy-move synthesis is not effective since it brings obvious semantic artifacts. To address this, we introduce Object Jitter, a data synthesis method that simulates these challenging cases by applying minor perturbations (e.g., enlargement) to random objects on authentic images, effectively supplementing our dataset.

In our CVPR 2024 paper, we propose to address the data scarcity of IML by auto-annotating images from the web with a category-aware approach CAAA. However, CAAA did not incorporate a prior-feature-denoising paradigm for its SDG branch, leading to overfitting on limited training data. In this paper, we propose CAAAv2, an improved version of our CAAA that resolves this issue by leveraging frozen DINO-based prior features. The enhanced accuracy of CAAAv2 ensures that a greater number of samples are retained after filtering. Consequently, the collected dataset MIMLv2 is both larger and more diverse than our conference version MIMLv1 [24], better supporting the development of IML models (Figure 1). Moreover, different from our conference paper that only uses MIMLv1 dataset to improve IML models, our Object Jitter approach further effectively supplements MIMLv2 with high-quality hard samples.

To better leverage our high-quality web-scale supervision for IML, we further propose a new model named Web-IML, which comprises a Multi-Scale Perception module and a Self-Rectification module. The former enables the model to better integrate information from multiple perspectives; the latter allows the model to learn to identify and correct its mistakes. Our Web-IML significantly outperforms previous methods on various widely used benchmarks, being smaller yet more effective than the APSC-Net in our conference version.

In summary, our main contributions are as follows:

• •

We propose to address the scarcity of IML training data by leveraging the less challenging CIML task to auto-annotate forged images collected from the web.

• •

We introduce CAAAv2, a novel CIML paradigm that significantly enhances generalization. This is achieved through a category-aware design, which treats distinct image groups separately, and a prior-denoising approach, which reframes CIML as a simpler denoising task.

• •

With CAAAv2 and a novel metric QES, we automatically annotate manually forged web images, constructing a large-scale, diverse, high-quality dataset MIMLv2. This dataset considerably alleviates the data scarcity in IML.

• •

We introduce Object Jitter, which generates high-quality, challenging-to-localize artifacts, effectively supplementing our dataset. To better leverage the web-scale supervision, we design a novel model Web-IML. Extensive experiments have validated its effectiveness.

II Related Works

II-A Image Manipulation Localization (IML)

Image Manipulation Localization aims to identify forged regions within images [25, 26, 27, 28, 29, 30, 31, 32, 33]. The scale of existing handcrafted datasets is limited due to the high cost of manual collection and annotation, inevitably leading to overfitting [18, 17]. To mitigate this issue, some studies have incorporated handcrafted features. For instance, RGB-N [34] applied SRM [34] filters to suppress semantic bias. MVSS-Net [35] and MGQFormer [36] further exploited learnable noise domain modeling. CAT-Net [19], ObjectFormer [37] and HiFi-Net [38] leveraged frequency features to localize manipulations. However, the handcrafted features are noisy and unstable, hindering further improvements in model performance. Recently, pure vision models have made notable progress [39, 1]. NCL [17] employed contrastive learning to improve generalization. SparseViT [18] utilized dilated attention to reduce semantic bias. PIM-Net [1] explicitly modeled the super-pixel inconsistency to alleviate biased learning. Despite these advances, overfitting remains a significant challenge due to the fundamental issue: the scarcity of high-quality training data.

To address this fundamental issue, we propose to construct a large-scale, high-quality dataset by automatically annotating Internet images with a less challenging task, CIML.

II-B Constrained Image Manipulation Localization (CIML)

Constrained Image Manipulation Localization aims to identify forged regions in an image by leveraging its original counterpart [40]. Previous works relied on correlation matching and handled SDG and SPG image pairs uniformly. For instance, DMVN [40] computed correlation maps to localize similar objects across image pairs. DMAC [22] incorporated dilated convolution to enhance spatial information. AttentionDM [23] employed an attention mechanism to improve performance. MSTAF [41] performed correlation in both the encoder and decoder to extract richer features. While these methods achieved notable progress on less challenging datasets (e.g., synthetic COCO [40]), their performance is limited in real-world forgeries, which often involve high-resolution images, significant variations, and complex details.

In contrast, our approach reduces task difficulty and addresses overfitting by (1) explicitly harnessing the distinct characteristics of SPG and SDG pairs; and (2) transforming the CIML problem into a denoising task through prior features extracted from frozen functions.

III Category-Aware Auto-Annotation v2

III-A Motivation and Framework Overview

For CIML, previous works overlooked the critical discrepancy and characteristics between SPG and SDG images, treating them using a single fully-tunable correlation model. We argue that this paradigm is suboptimal for two reasons:

1. Conflicting Correlation Signals. The nature of the ”shared region” differs drastically between the two categories. In SDG pairs, the shared region is salient foreground objects with distinctive features (e.g., the lion in SDG of Fig. 2), providing a strong signal for correlation matching. Conversely, in SPG pairs, the shared region is the background, which lacks the unique features required for reliable matching (e.g., the sky in SPG of Fig. 2). Forcing a single correlation model to simultaneously handle both groups creates a conflicting objective that degrades performance.

2. Lack of Explicit Guidance. Learning to comprehend and fully leverage the correspondence between original and manipulated images is non-trivial. With scarce training data, CIML models tend to find shortcuts to minimize training loss, often overfitting to either forgery-agnostic semantics in small-scale handcrafted datasets or monotonous patterns in synthetic data. Without explicit guidance on how to utilize the correspondence within an image pair, models often fail to generalize to diverse, real-world manipulations.

To overcome these challenges, we propose Category-Aware Auto-Annotation v2 (CAAAv2), a novel paradigm for CIML. Our core idea is to process SDG and SPG images independently, while explicitly integrating image correspondence as a guiding prior feature. Specifically, as shown in Fig. 2, CAAAv2 introduces a classifier to categorize the input as either SPG or SDG, and then applies a category-specific prior to guide localization. For SDG pairs, we introduce DINO Correlation, a prior that captures shared foreground objects by correlating dense features from a frozen DINOv2 encoder. For SPG pairs, a simple Difference Map effectively identifies manipulated regions by subtracting the two images.

These priors embed domain-specific knowledge about forgery patterns into strong, zero-shot signals. By generating these priors with frozen, non-trainable functions, we reframe the complex task of open-ended forgery localization into a simpler, more constrained denoising problem. Here, the model’s objective is to refine the coarse prior into a precise mask, using the input images as context. This paradigm shift significantly reduces task difficulty and minimizes the generalization gap between synthetic and real-world forgeries, leading to substantial performance improvements.

Importantly, the trained models are further used to auto-annotate large-scale, unlabeled manual forgeries from the web, thereby addressing the data scarcity issue in IML.

Whereas CAAA overfits on SDG data, CAAAv2 provides explicit guidance by introducing DINO Correlation as a strong prior. This prior is generated using a frozen DINOv2, whose global receptive field naturally aligns multi-scale semantics, yielding substantial gains in real-world generalization.

III-B Self-Supervised Classification

Training a classifier to distinguish between SDG and SPG is a non-trivial task due to the lack of labeled data. To address this challenge, we propose training a classifier via self-supervised learning on unlabeled images. Given an image, we apply random augmentations (e.g., color jitter, image compression) and in-place manipulations (copy-paste, splicing, or removal). The manipulated image and the original image form an SPG pair. To construct an SDG pair, we copy 1-3 random regions from the original image, resize them, and paste them into another image. With the obtained image pairs, we can effectively train our classifier. Our classifier actually only needs to discern whether the main area of the input image pair is identical or not, without considering which image or region is fake. For example, in Fig. 6, the two images in any SPG pair are nearly identical and spatially aligned, whereas images in any SDG pair are distinctly different. These characteristics are universally applicable and strong enough. Therefore, this classification task is quite simple, enabling accurate separation of images into the two groups.

III-C Corr-DINO

For Shared Donor Group (SDG) pairs, the manipulated regions are mostly the objects copied from the original image. The localization task therefore reduces to identifying these shared foreground objects. To exploit this correspondence, we introduce a novel prior feature, termed DINO Correlation. This prior is generated by extracting dense features from both images using a frozen DINOv2 encoder and then computing their cross-correlation. The resulting map provides a strong yet coarse signal that highlights the shared objects, which often are the forged regions. The task is thus simplified into refining this coarse prior to a precise segmentation mask. Based on this, we propose the Corr-DINO model. As illustrated in Figure 3, it consists of four components:

1. Prior Feature Extractor for prior feature extraction. We extract object features from both the tampered image $I_{a}$ and its original image $I_{b}$ with a frozen DINOv2 [42] backbone, which has learned generalized representations through self-supervised learning on vast-scale web images. The features from the last DINO layer, denoted as $F_{a1}$ and $F_{b1}$, contain rich semantic information and are used to compute correlation features $F_{a\_corr}$ and $F_{b\_corr}$. $F_{a\_corr}=[Corr(F_{a1},F_{b1}),Corr(F_{a1},F_{a1})]$, $F_{b\_corr}=[Corr(F_{b1},F_{a1}),Corr(F_{b1},F_{b1})]$. The same correlation function $Corr$ as most previous works [23, 41, 24] is adopted.

2. Learnable Aggregation module for dimension reduction. Previous works simply select the top-K activated channels with the highest average values in $F_{a\_corr}$ and $F_{b\_corr}$ to reduce dimension. This approach works for trainable encoders but is suboptimal for our frozen encoder, whose correlated object features are relatively sparse. Selecting only the top-K channels may discard important features and compromise the integrity of $F_{a\_corr}$ and $F_{b\_corr}$. To overcome this, we propose a learnable aggregation module that uses trainable parameters to reduce the channel count to K. We also include statistics of average and maximum values across the channel dimension. $F_{a\_aggr}=[C_{1\times 1}(R(C_{1\times 1}(F_{a\_corr}))),Avg(F_{a\_corr}),Max(F_{a\_corr})]$. Here, $C_{1\times 1}$ is $1\times 1$ conv-layer, $R$ is ReLU. $Avg$ and $Max$ are average and maximum pooling respectively. A similar process is used for $F_{b\_corr}$ to generate $F_{b\_aggr}$.

3. Feature Super Resolution module for image detail reconstruction. The features from the last DINOv2 layer are rich in semantics but lack fine image details [42], making $F_{a\_aggr}$ and $F_{b\_aggr}$ too coarse for fine-grained mask predictions. To reconstruct image details, we leverage features from the last 2-4 layers of DINO, which are known to be more sensitive to edges and fine details [42]. We concatenate the features from the last four encoder layers $F_{a}=Concat(F_{a1,2,3,4})$ and interpolate $F_{a\_aggr}$ to different scales ($1\times,2\times,4\times,8\times$) to obtain $F_{aggr\_a1,2,3,4}$. We then fuse them with $1\times 1$ conv-layers. A similar process is applied to $F_{b\_aggr}$.

4. Multi-Aspect Denoiser for feature denoising. Since the DINOv2 encoder is frozen, its output features inevitably contain significant task-irrelevant noise. To achieve precise mask prediction, we propose denoising the correlation features by integrating multi-perspective information. This module fuses multi-scale features $F_{asr\_1,2,3,4}$ in a top-down manner. The fused features are further concatenated and denoised using a set of dilated conv-layers. The output is concatenated to $F^{\prime}1$, which is used to yield the mask prediction. More details are provided in the Appendix.

III-D Difference-Aware Semantic Segmentation (DASS)

SPG pairs are characterized by in-place manipulation, where the manipulated image is generated by directly altering its original. This process inherently ensures precise spatial alignment and a shared background, meaning that the forged regions are always highlighted by the image difference. Leveraging this strong correspondence, we use the difference map as a prior feature to guide model learning. Given a manipulated image $I_{m}$ and its original $I_{o}$, the absolute difference map $D_{mo}=abs(I_{m}-I_{o})$ can roughly highlight forged regions.

However, transmission noise and image degradation often contaminate the difference map, making it too noisy for precise localization. As illustrated in Fig. 4, nearly every area exhibits nonzero values due to noise, and even adaptive binarization [43] fails to denoise. To address this issue, we propose denoising the difference map by incorporating semantic information from the images. Specifically, we concatenate $I_{m}$, $I_{o}$, and $D_{mo}$ along the channel dimension, and feed them into a semantic segmentation model. This model integrates a VAN [44] encoder for extracting rich semantic features and employs the same Multi-Aspect Denoiser (III-C) as in our Corr-DINO for denoising the difference map.

IV MIMLv2 Dataset

In this section, we introduce MIMLv2, a large-scale, diverse, and high-quality dataset. The key idea is to leverage CIML models trained on existing datasets to auto-annotate manual forgeries collected from the web, thereby addressing the data scarcity in IML. To ensure high annotation quality, we propose a novel metric that filters out inadequate annotations without requiring ground-truth for evaluation.

Our MIMLv2 is annotated using our CAAAv2, which is significantly more accurate than its conference version, CAAA [24]. As a result, more samples are retained as high quality after filtering, as shown in Table I. Compared to MIMLv1 [24], MIMLv2 offers greater scale and diversity, making it a more effective resource for training IML models.

IV-A Dataset Construction

As illustrated in Fig. 5, we construct MIMLv2 through the following steps:

1. Image Collection. We collect forged images and their corresponding originals from imgur.com. On this website, the images are manually manipulated and thus have high-quality, diverse forged regions. Additionally, we exclude damaged, duplicated images and those that already appear in the evaluation datasets [45, 46, 47, 48, 49, 50, 51] with MD5 [52] and pHash [53].

2. Auto-Annotation. We utilize the CAAAv2 models, as proposed in Section III, to obtain mask annotations for the collected images. The models are effectively trained on synthetic COCO [40], CASIAv2 [46] and IMD20 [45] datasets.

3. Quality Evaluation. After auto-annotation, SPG annotations are already high-quality, whereas a subset of SDG annotations remains unsatisfactory. We empirically found that inaccurate predictions mostly have smooth edges and low confidence. Inspired by this, we propose Quality Evaluation Score (QES), a novel metric that assesses both confidence and edge sharpness to evaluate annotation quality and filter out inadequate ones. Specifically, given a prediction mask with shape (H, W) and normalized probability, we compute the QES as follows: QES=$\frac{\sum_{i,j}^{H,W}p_{i,j}>(1-T_{h})}{\sum_{i,j}^{H,W}p_{i,j}>T_{l}}$, where $\sum_{i,j}^{H,W}p_{i,j}>(1-T_{h})$ denotes the area of prediction with a high confidence greater than $(1-T_{h})$, and $\sum_{i,j}^{H,W}p_{i,j}>T_{l}$ denotes the total predicted potentially manipulated area. We empirically set $T_{h}$ and $T_{l}$ to $\frac{1}{16}$ and only retain samples with QES$>$0.5. In this way, QES can effectively filter unreliable mask annotations.

IV-B Dataset Highlights

We present a few samples of our MIMLv2 dataset in Fig. 6. The forged regions are precisely annotated with high-quality masks, demonstrating the effectiveness of our auto-annotation approach. The main highlights of our dataset include:

• •

High quality. The proposed dataset contains high-quality image manipulations crafted by humans. Such data is crucial to training models that can detect real-world forgeries rather than overfit to synthetic data.

• •

Large Scale. The proposed dataset is substantially larger than existing handcrafted IML datasets. As shown in Table I, it comprises 246,212 manually forged images, which is about 120 times more than IMD20 [45].

• •

Broad Diversity. MIMLv2 comprises images of various sizes, styles, and manipulation types (e.g., copy-move, splicing, removal, AIGC), created by more than 10k individuals and various software. This diversity significantly enhances the generalization of IML models.

• •

Updated Forgeries. MIMLv2 contains numerous modern images that were recently captured and forged, reflecting current photography and forgery technologies. In contrast, the CASIA dataset [46] proposed over a decade ago, features blurred small images and outdated forgeries. Therefore, our MIMLv2 better suits current IML requirements.

• •

Strong Scalability. With the rising popularity of online image manipulation competitions (e.g., PS-Battles [54], which attracts over 20 million participants), new manually forged images are continuously generated. Our dataset construction approach is ready to harness this growing cheap web data. Therefore, our dataset can be easily expanded in the future.

V Object Jitter

Sophisticated copy-move forgeries, where an object is copied and pasted within the same image, are difficult to detect. They introduce only subtle edge artifacts while preserving source texture and noise patterns. Their scarcity in web-sourced data leads models to rely more on common noise and compression anomalies, causing them to fail on these hard cases. Existing synthesis methods, designed to generate training data for such forgeries, are flawed: random copy-paste techniques [56, 34] often create semantically incoherent artifacts (e.g., a train on food, Fig. 7), while more constrained approaches like DEFACTO limit tampering to a few objects of consistent shape, lacking object diversity. Both issues can cause models to overfit to unrealistic cues.

To address this gap, we propose Object Jitter, a novel strategy that simulates these forgeries by subtly distorting existing objects. The core operation is Size Jitter: we slightly enlarge a randomly selected object, covering its original footprint. This mimics a realistic copy-move forgery by preserving semantics and texture while introducing only faint boundary artifacts. As illustrated in Fig. 9, the process involves (1) segmenting an object using ground truth or SAM [57], (2) scaling the object and its mask by a small random factor, and (3) applying a smooth alpha-blended transition at the new boundary to minimize extraneous visual cues.

To further enhance data diversity, we introduce two additional minor perturbations: Exposure Jitter (slight overexposure) and Texture Jitter (subtle modifications via JPEG compression, reverse JPEG [58], and blur operations). By preserving semantic plausibility and being applicable to any object, Object Jitter provides a source of challenging training examples. This compels the model to learn the subtle boundary artifacts characteristic of sophisticated forgeries, thereby improving its robustness and generalization.

VI Web-IML

To better leverage the web-scale supervision, we propose a new model called Web-IML. As illustrated in Fig. 9, Web-IML comprises a feature extractor, a Multi-Scale Perception module to integrate information from multiple perspectives, and a Self-Rectification module that enables the model to identify and correct its mistakes.

VI-A Multi-Scale Perception Module

Motivation. This module is designed to mimic the meticulous image forensic analysis, where humans often zoom in and out the image repeatedly, comparing diverse observations to assist their final prediction.

Pipeline. Given the feature maps ${F_{1},F_{2},F_{3},F_{4}}$ extracted from the encoder, we first extract global features of different scales through multi-scale average pooling. The global features are then fused and dimension-reduced to get $F^{\prime}4$. Next, the high-level features are fused with the low-level features recursively. This process fully integrates high-level semantics with low-level image details, resulting in the multi-scale features ${F^{\prime}_{1},F^{\prime}_{2},F^{\prime}_{3},F^{\prime}_{4}}$. More details are provided in the Appendix.

VI-B Self-Rectification Module

Motivation. This module is designed to make the model learn to check and correct its initial predictions, thereby minimizing the overlooking of key information and mistakes. The Self-Rectification (SR) module improves upon the Self-Calibration (SC) module in our conference paper from two aspects: First, in SC, we map the initial prediction mask to a convolution kernel, use it to convolve the mask and multiply the result by the multi-scale features. Despite its effectiveness, learning the map function to a convolution kernel is difficult, and directly multiplying the convolved mask often causes information loss in the black mask region. In our SR, we map the initial prediction mask to a feature map, and concatenate it with the multi-scale features, addressing the issues. Second, we introduce a Nested Channel Attention module that enables more meticulous analyses of the rectification feature.

Pipeline. As shown at the bottom of Fig. 9, we obtain the initial mask prediction $P_{n}$ from $F^{\prime}_{1}$. We then adaptively obtain a rectification feature $F_{r}$ through a tiny Rectification Feature Mapping module ($RFM$), which consists of 5 cascaded conv-layers. The $F_{r}$ informs the model of suspected tampered regions and potentially incorrect prediction regions. $F_{r}$ is fused with $F_{a}$ to get $F_{c}$. Then, in the Nested Channel Attention module, we obtain the global vectors $G_{1}$ from $F_{c}$, and subsequently higher-level global vectors $G_{2}$ from $G_{1}$. Then $G_{1}$ is weighted by the attention map calculated by $G_{2}$ and the results are further used to calculate the channel attention weights for $F_{c}$. The output $F_{o}$ is further processed by four dilated conv-layers, the output features are concatenated and fused to produce the mask prediction $P_{f}$. To get the refined final prediction $P_{f}$, we replace $P_{n}$ with $P_{f}$, replace $F_{a}$ with $F_{o}$, then repeat the self-rectification once more.

Our Web-IML is optimized with cross entropy loss in an end-to-end manner.

VII Experiments

VII-A Experiments on the Web-IML Model (IML Task)

This subsection evaluates the effectiveness of the proposed Web-IML model by comparing it with other IML models on multiple real-world handcrafted and AIGC forgery benchmarks. We also perform an ablation study to provide an in-depth analysis of the Web-IML model.

Implementation Details. For Web-IML, we adopt ConvNeXt-Base [68] as the feature extractor. The model is trained for 200k iterations with a batch size of 16, using a 512$\times$512 input size, consistent with previous works [18, 49]. We employ cross entropy loss and AdamW [69] optimizer, with the learning rate linearly decaying from 1e-4 to 1e-6. CASIAv2 [46] and tampCOCO [19] serve as baseline training sets. For fair comparison, mask predictions are binarized with a fixed threshold of 0.5 and evaluated using IoU and binary F1-score.

Comparison Study. Table II shows that our Web-IML significantly outperforms existing methods. For example, it achieves an average IoU that is 21.6 points higher than SparseViT. Furthermore, Web-IML consistently outperforms our conference version, APSC-Net, across various settings. These results demonstrate the strong generalization capabilities of our Web-IML model. Qualitative comparison is illustrated in Fig. 10.

Ablation Study. The ablation results are shown in Table III. Setting (2), which includes the proposed Multi-Scale Perception (MP) module, achieves 5.2 points higher IoU than the baseline (1). This indicates that the proposed MP module effectively integrates high-level and low-level features to detect diverse image artifacts. Setting (3), which further incorporates the Self-Rectification (SR) module, yields an additional 3.9 points IoU improvement over Setting (2), as SR enables the model to detect and correct its errors. Setting (4) outperforms (3) via our nested attention module that enables in-depth analysis. Setting (5) outperforms (4) by including a second round of self-rectification refinement, demonstrating that further refinement can minimize incorrect predictions.

Robustness Evaluation. We evaluate Web-IML under various distortions, including image resizing (scale factors 0.78 and 0.25), Gaussian blur (kernel sizes 3 and 15), and JPEG compression (quality factors 50 and 100). The results are shown in Table IV. Our model exhibits stable performance under these conditions, demonstrating strong robustness.

VII-B Experiments on the Web-Supervision Method (IML Task)

This subsection evaluates the effectiveness of our proposed web-supervision approach in addressing the data scarcity issue for IML. Our web-supervision encompasses both the MIMLv2 dataset and the Object Jitter method. We compare the performance of the same IML model trained on data created by our web-supervision approach against other synthetic methods and WSCL [66], a representative weakly supervised framework.

Implementation Details. The implementation details remain the same as in the previous subsection VII.A. When supplementing the baseline training sets with additional IML training sets, the total training volume is kept constant with a uniform sampling ratio across all datasets. For models integrated with WSCL, training involved two types of data: direct training with the fully (pixel-level) annotated tampCOCO [19] and CASIAv2 [46] datasets, and WSCL-specific training with weakly (image-level) annotated MIMLv2 and SA [57] dataset images. The training volume and other configurations remain the same as ours.

Comparison Study for the MIMLv2 Dataset. Table V presents the IML performance of the Web-IML model when trained on different datasets. The baseline (1) uses the two most common training sets: tampCOCO [19] and CASIAv2 [46]. Incorporating synthetic training data from PSCC-Net [56], DEFACTO [20] or OpenSDID [21] diminishes the average model performance (settings 2, 3, 4). This is because the first two synthetic datasets do not notably surpass the baseline data in quality or diversity, while the AIGC forgeries from OpenSDID [21] differ significantly from non-AIGC forgeries. These results confirm that simply increasing the scale of training data does not lead to improved performance. In contrast, including the MIMLv1 (setting 6) or MIMLv2 (setting 7) datasets significantly improves model performance by an average of 10 and 12 points, respectively, without increasing training or inference burden. Additionally, although WSCL (setting 5) offers improvements over baselines, its effectiveness is considerably lower than that of our web-supervision. This is because our method explicitly produces accurate mask annotations, by leveraging the image correspondence that substantially reduces task difficulty. Conversely, WSCL lacks the ability to leverage this valuable information, and relies heavily on unstable noise domain features. These results confirm that our MIMLv1 and MIMLv2 datasets considerably alleviate data scarcity and overfitting issues in real-world scenarios. This is attributable to their large scale, manual forgery, and considerably higher quality and diversity compared to other datasets. Notably, MIMLv2 yields greater improvement due to its larger quantity and superior diversity compared to MIMLv1.

Comparison Study for the Object Jitter. Table V shows that including the training data generated by our Object Jitter method further improves the model when combined with MIMLv1 (setting 8 outperforms 6) or MIMLv2 (setting 10 outperforms 7) datasets. This improvement is attributed to Object Jitter’s consistent generation of high-quality hard samples, which effectively supplements the training data by compensating for the relative scarcity of copy-move forgeries.

Ablation Study for the Object Jitter. Table VI presents the ablation study for Object Jitter operations and image source. Setting (1) is the baseline, where Web-IML is trained with MIMLv2 but without Object Jitter. Setting (7) represents the full Object Jitter method. Settings (2, 3, 4) involve removing the size jitter, texture jitter, and exposure jitter operations, respectively; all of them result in worse performance than setting (7). These results demonstrate that all three operations contribute to improved performance through increased data diversity. Notably, Setting (2), which lacks size jitter, is significantly worse than (7). Meanwhile, Setting (5), which uses only size jitter, performs close to (7). This indicates that the size jitter operation is the primary contributor to the improvement. Since most Internet forgeries are not generated by copy-move, they exhibit exposure and texture artifacts. The size jitter operation, without altering the object’s exposure or texture, reduces the model’s reliance on exposure or texture cues, thus better supplementing MIMLv2. Furthermore, Setting (6) applies the full Object Jitter method using COCO [67] as the image source, and setting (7) uses 400k random images from SA [57]. The similar performance of Settings (6) and (7) demonstrates that the Object Jitter method is robust to the choice of image source.

Verification of Conclusions on Other IML Models. We extended our verification process to other IML models. Specifically, we evaluated MIMLv2 and Object Jitter on PSCC-Net [56], CAT-Net [19], TruFor [49], SparseViT [18] and IMDPrompter [70] (a large model based on SAM [57]) using the same training data and configuration as ours. The results in Table VII clearly indicate that MIMLv1, MIMLv2, and Object Jitter all lead to significantly higher scores. MIMLv2, in particular, consistently yields significantly greater improvement than MIMLv1 (conference version). Moreover, Object Jitter further boosts performance when integrated with MIMLv2. These results robustly confirm our previous analyses and conclusions.

VII-C Experiments on the CAAAv2 Models (CIML task)

This subsection evaluates the effectiveness of the proposed CAAAv2 in the CIML task by comparing its DASS and Corr-DINO models against other CIML models. A higher CIML score means more accurate auto-annotation. We also conduct detailed ablation studies for in-depth analysis.

Implementation Details. Given that real-world forgeries in the IMD20 [45] dataset closely resemble the target images for annotation, we use a portion of this dataset to evaluate model performance using IoU and binary F1-score metrics. We categorize the forged images in IMD20 into SPG or SDG and randomly split them into training and test sets with a 3:1 ratio. CASIAv2 [46] and synthetic COCO [67] datasets are also used for training. Input images are resized to 512$\times$512, and a consistent training pipeline is applied across all methods.

Comparison Study. Table VIII presents comparison results for SPG (left) and SDG (right) separately. Our method clearly outperforms these previous methods because our category-aware and prior-feature-denoising paradigm significantly reduces task difficulty. As shown in the right side of Table VIII, our Corr-DINO, prior to QES filtering, achieves an IoU of 0.744. This is 4.2 points higher than the SACM [24] (our conference version), demonstrating that our frozen-denoising paradigm notably alleviates the overfitting issue when training on limited data. Notably, DMVN [40] and DMAC [71] models trained on both SPG and SDG data perform worse than those trained exclusively on one category. This is attributed to the confusion caused by similar backgrounds in SPGs, confirming the necessity of our category-aware approach.

Ablation Study for the DASS Model Input. Table IX presents the results, confirming that both the prior feature difference map and the image pair are essential for accurate prediction in SPG. The difference map between forged and authentic images can roughly indicate the forged regions, while the images themselves provide semantic information to help the model denoise the difference map.

Ablation Study for the Corr-DINO Model Prior Feature Extractor. The results are shown in Table X, settings (1-24). For ConvNeXt, the trainable version performs better. For all ViT-based models, however, frozen versions perform significantly better. Since ViT models tend to overfit with limited training data, freezing the backbone alleviates catastrophic forgetting and overfitting. LoRA [73] fine-tuned versions exhibit similar performance to their frozen counterparts, as LoRA also mitigates forgetting. The frozen DINOv2-Base [42] backbone performs optimally and is therefore adopted.

Ablation Study of the Corr-DINO Model Modules. The results are shown in the bottom of Table X, setting (25) is the baseline without any proposed modules, setting (29) is the full model with all proposed modules. In settings (26, 27, 28), we remove the proposed Multi-Aspect Denoiser, Feature Super Resolution and Learnable Aggregation modules from the full model (29) respectively. Setting (29) significantly outperforms (26). This validates that correlation features extracted from the frozen DINOv2 are highly noisy, and that denoising through multi-aspect feature integration is essential for accurate predictions. Setting (29) also significantly outperforms (27). This confirms that initial correlation features are too coarse for precise mask predictions, and our Feature Super Resolution module effectively addresses this by reconstructing fine-grained details. Finally, Setting (29) is superior to (28) because our learnable aggregation mitigates information loss inherent in previous rule-based aggregation.

Ablation Study of the QES. The last two rows of Table X show the results. The results of setting (30) are obtained by evaluating only the predictions from setting (29) where QES$>$0.5. QES-filtered predictions from setting (30) achieve an IoU of 0.912, representing a 16.8 point increase over setting (29). This confirms that QES effectively filters out unsatisfactory predictions, thereby notably improving annotation quality. Table XI confirms QES’s high relevance to the IoU metric without relying on ground-truth for evaluation.

SPG/SDG Classification. We evaluate the performance of the SPG/SDG classification model within the proposed CAAAv2 on IMD20 [45] dataset. Considering that a very small proportion of SPG image pairs are not spatially aligned, which could negatively impact the prediction’s quality, we also include an additional linear binary classification layer to filter them out. The classification results are presented in Table XII. It is evident that the voting ensemble of the classification models we used is accurate enough for SPG/SDG classification, as analyzed in Section III.A. In practice, to further ensure accuracy, we only keep images with a classification confidence above 99% and discard the rest. Our QES also effectively filters the annotations of potentially misclassified samples.

Our Models are Adequate Auto-Annotators. In Table VIII, both our DASS and Corr-DINO achieve high IoU scores. Considering the errors in IMD20’s ground-truth [74], especially in SPG images, our actual accuracy is higher. Clearly, our methods are sufficient for obtaining accurate auto-annotations.

VII-D Experiments on the Downstream Document IML Task

This subsection evaluates the generalization of the Web-IML model and the universal applicability of our web-supervision method on an important downstream task: document IML.

We fine-tuned the Web-IML model, pretrained under various configurations, on the handcrafted real-world document IML benchmarks, SACP [81] and RTM [88], adopting the same training and test configurations as previous studies [86, 88]. Across both benchmarks, our Web-IML model consistently and notably outperforms existing methods. A key observation from Tables XIII and XIV is that settings 11, 12, and 13, which augmented the baseline (Setting 10) with additional pretraining data from DEFACTO [20], PSCC [56], and OpenSDID [21], respectively, exhibited degraded performance. This confirms that merely increasing the scale and diversity of pretraining data does not lead to improved real-world generalization. In contrast, the model pretrained on our web-supervision data (Setting 15) demonstrates significant improvement. It achieved 2.2 points higher IoU on SACP and 4.7 points on RTM compared to the baseline (Setting 10). Furthermore, it also surpassed the model pretrained on MIMLv1 (Setting 14) by 0.8 points higher IoU on SACP and 1.5 points on RTM. These results demonstrate that the cost-effective data created by our web-supervision method possesses sufficient quality and relevance to enable models to learn robust semantic-agnostic forensic features, thereby generalizing effectively across diverse and unseen real-world IML tasks.

VIII Limitations

Although our QES metric is highly effective in ensuring overall annotation quality, a small number of correct annotations may also be filtered out. In addition, our Web-IML is designed to maximize accuracy by leveraging web-scale supervision with a medium-sized model (140M). As a result, it is not highly computationally efficient (achieving 10 frames per second on a single 3090 GPU). Figure 11 presents some failure cases, showing that both Web-IML and previous methods may struggle to produce precise masks when the manipulated region exhibits very few visual artifacts.

IX Conclusion

In this paper, we addressed the critical challenge of data scarcity in image manipulation localization through a scalable web-supervision approach. At the core of our approach is CAAAv2, a novel category-aware and prior-denoising auto-annotation paradigm. This innovative design strategically separates manipulations into Shared Donor Group and Shared Probe Group, effectively converting the challenging forensic task into a prior-denoising task. This dramatically reduces task complexity and mitigates overfitting. Leveraging this automatic annotation engine and our proposed QES metric for quality control, we constructed MIMLv2, a diverse, high-quality dataset of 246,212 manually forged images—over 120× larger than existing handcrafted datasets. We further enhanced this web-scale supervision with Object Jitter, a novel technique for generating challenging training examples from authentic images. To fully capitalize on this vast data, we designed Web-IML, a new model featuring multi-scale perception and self-rectification modules. Extensive experiments demonstrate that our web-supervision method provides unprecedented performance gains across multiple models on a wide array of real-world forgery benchmarks. Additionally, Web-IML establishes a new state-of-the-art, significantly outperforming previous methods and demonstrating remarkable generalization. We believe that this work establishes a scalable framework for continuously expanding training data as new manually forged images emerge online. This research transforms image forensics, shifting reliance from limited, expensive handcrafted datasets to abundant, continuously growing web resources, thus paving the way for more robust image manipulation localization systems.