First-Place Solution to NeurIPS 2024 Invisible Watermark Removal Challenge

TL;DR

This paper presents a winning solution to the NeurIPS 2024 challenge on removing invisible watermarks, demonstrating near-perfect removal under various attack scenarios and highlighting the need for more robust watermarking techniques.

Contribution

The paper introduces novel attack methods for watermark removal in both black-box and beige-box settings, combining VAE-based evasion and diffusion models with semantic priors.

Findings

Achieved 95.7% watermark removal success rate.

Demonstrated minimal impact on image quality.

Provided insights for developing more robust watermarking methods.

Abstract

Content watermarking is an important tool for the authentication and copyright protection of digital media. However, it is unclear whether existing watermarks are robust against adversarial attacks. We present the winning solution to the NeurIPS 2024 Erasing the Invisible challenge, which stress-tests watermark robustness under varying degrees of adversary knowledge. The challenge consisted of two tracks: a black-box and beige-box track, depending on whether the adversary knows which watermarking method was used by the provider. For the beige-box track, we leverage an adaptive VAE-based evasion attack, with a test-time optimization and color-contrast restoration in CIELAB space to preserve the image's quality. For the black-box track, we first cluster images based on their artifacts in the spatial or frequency-domain. Then, we apply image-to-image diffusion models with controlled noise injection and semantic priors from ChatGPT-generated captions to each cluster with optimized parameter settings. Empirical evaluations demonstrate that our method successfully achieves near-perfect watermark removal (95.7%) with negligible impact on the residual image's quality. We hope that our attacks inspire the development of more robust image watermarking methods.

Tables

Table 1. Table 1: Black-box Track Final Leaderboard

Black-box Track
Rank	Participant	Detection	Quality	Total
	Team-MBZUAI	0.043	0.136	0.143
	Team-SHARIF	0.063	0.158	0.170
	Team-UFL	0.087	0.177	0.197

Table 2. Table 3: Performance comparison of our watermark removal pipeline. Det. refers to the detection metric (lower is better), Qual. measures image quality. Progressive enhancements through test-time optimization and color/contrast adjustment improve quality.

Method	Det.	Qual.	Overall
VAE Finetune	0.023	0.192	0.193
+ Test-Time Opt.	0.033	0.161	0.165
+ Color/Contrast	0.037	0.153	0.157

Table 3. Table 4: Quantitative comparison of our proposed quality-preserving watermark removal pipeline against VAE fine-tuning alone. Higher PSNR and SSIM, and lower LPIPS, FID, NMI, and CLIPFID indicate better perceptual quality and watermark suppression. The results demonstrate that integrating test-time optimization with color/contrast transfer yields substantial gains across both pixel-level and perceptual metrics.

Method	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	FID$\downarrow$	NMI$\uparrow$	CLIPFID$\downarrow$
VAE Finetune	21.899	0.647	0.264	85.383	0.231	11.805
Ours	28.061	0.823	0.078	30.786	0.326	2.831

Table 4. Table 5: Cluster assignments for all 300 black-box images , grouped by manually identified artifact type: boundary artifacts, circular or square Fourier magnitude patterns, or no visible artifacts. Each block lists the image indices belonging to that cluster, enabling targeted strategy design and facilitating quantitative comparisons of removal performance.

Cluster 1 (No Artifacts)
0, 2, 3, 4, 10, 13, 14, 16, 19, 21, 24, 29, 30, 33, 36, 40, 41, 43, 50, 51, 56, 58, 60, 61, 62, 67, 71, 75, 77, 80, 83, 86, 93, 94, 95, 96, 97, 98, 104, 107, 109, 110, 112, 113, 114, 123, 125, 126, 130, 137, 138, 146, 149, 154, 155, 157, 159, 161, 162, 166, 167, 171, 175, 178, 179, 187, 196, 198, 199, 200, 201, 205, 209, 210, 215, 216, 217, 218, 220, 226, 227, 233, 235, 241, 245, 247, 249, 250, 254, 260, 262, 266, 268, 275, 279, 280, 281, 288, 291, 292, 294, 296
Cluster 2 (Boundary Artifacts)
1, 6, 11, 23, 35, 42, 48, 49, 57, 65, 68, 69, 78, 82, 84, 85, 91, 100, 102, 105, 119, 120, 121, 131, 132, 140, 141, 142, 143, 148, 169, 170, 180, 186, 203, 204, 207, 231, 240, 246, 256, 257, 261, 267, 269, 274, 283, 297, 298
Cluster 3 (Circular Patterns in Fourier Magnitude)
9, 17, 25, 28, 31, 32, 38, 39, 47, 53, 70, 79, 87, 88, 99, 115, 117, 127, 128, 133, 147, 151, 163, 165, 172, 174, 177, 183, 185, 192, 193, 195, 197, 202, 211, 232, 242, 243, 244, 251, 258, 263, 265, 271, 276, 277, 278, 287, 289
Cluster 4 (Square Patterns in Fourier Magnitude)
5, 7, 8, 12, 15, 18, 20, 22, 26, 27, 34, 37, 44, 45, 46, 52, 54, 55, 59, 63, 64, 66, 72, 73, 74, 76, 81, 89, 90, 92, 101, 103, 106, 108, 111, 116, 118, 122, 124, 129, 134, 135, 136, 139, 144, 145, 150, 152, 153, 156, 158, 160, 164, 168, 173, 176, 181, 182, 184, 188, 189, 190, 191, 194, 206, 208, 212, 213, 214, 219, 221, 222, 223, 224, 225, 228, 229, 230, 234, 236, 237, 238, 239, 248, 252, 253, 255, 259, 264, 270, 272, 273, 282, 284, 285, 286, 290, 293, 295, 299

Table 5. Table 6: Quantitative comparison of our Beige-box removal strategy for TreeRing watermarks with and without pixel restoration. Incorporating selective restoration improves both fidelity metrics (FID, CLIP-FID) and perceptual similarity (SSIM, LPIPS), demonstrating that boundary-aware restoration enhances watermark removal without compromising quality.

Method	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	FID$\downarrow$	NMI$\uparrow$	CLIPFID$\downarrow$
w/o Pixels Restoration	14.878	0.428	0.133	15.773	0.212	3.199
Ours	14.353	0.434	0.124	12.531	0.218	1.622

Equations

$$\mathcal{L}(\theta ,\varphi )=\Vert D_{\varphi }(E_{\theta }(x_w))-x_i{\Vert }_2^2,$$

$${\mathcal{L}}_{\text{total}}=\underset{\text{MSE Loss}}{\underbrace{\Vert D_{\varphi }(E_{\theta }(x_r))-x_w{\Vert }^2}}+\underset{\text{Perceptual Loss}}{\underbrace{{\mathcal{L}}_{\text{LPIPS}}(D_{\varphi }(E_{\theta }(x_r)),x_w)}}+\underset{\text{Structural Similarity Loss}}{\underbrace{0.5(1-\text{SSIM}(D_{\varphi }(E_{\theta }(x_r)),x_w))}}$$

$$x_{\text{final}}={\mathcal{F}}_{\text{RGB}}(L_{\text{final}},a_w,b_w).$$

$$x_{\text{final}}(i,j)=\{\begin{array}{ll}{x}_w(i,j)& \text{if }j<\Delta x\\ x_{\text{shifted}}(i,j)& \text{otherwise}\end{array}$$

Full text

1 Introduction

Content watermarking is a widely used technique for embedding imperceptible information into digital media to ensure provenance verification [13], copyright protection [45; 59], and content traceability [27]. With generative AI systems now capable of producing high-fidelity synthetic content at scale, watermarking serves as an essential safeguard for content owners and organizations to combat unauthorized distribution [65; 56] and forgery [42; 5]. The goal of a watermarking method is to hide a signal (message) in generated content that can only be detected with a secret watermarking key [1; 30]. However, despite its widespread deployment, watermarking systems remain vulnerable to both unintentional distortions (e.g., blur, resizing) and targeted adversarial attacks that intentionally suppress the watermark signal without perceptually degrading the content [4; 9].

A robust watermarking scheme must ensure that successful removal is only possible at the cost of significant perceptual degradation [65; 31; 18]. However, recent studies have exposed critical vulnerabilities in existing methods, where adversaries can remove or spoof watermark signals by exploiting adaptive attacks [32; 9], latent-space priors [36; 22], or even simple averaging techniques [60]. These attacks preserve visual fidelity, often introducing no perceptible artifacts, thereby undermining the reliability of watermark detectors. As watermarking becomes increasingly important for ensuring the integrity, attribution, and traceability of AI-generated content, addressing these weaknesses is imperative. Progress in this area requires rigorous evaluation under realistic threat models to inform the design of watermarking schemes that are not only imperceptible but also resilient to both adversarial manipulations and common distortions.

To gauge the robustness of invisible watermarking methods against realistic threats, the NeurIPS 2024 Erasing the Invisible: A Stress-Test Challenge for Image Watermarks [10] introduced a rigorous benchmark targeting watermark removal. This paper presents our first-place winning solution to this challenge under two practical threat models: beige-box, where the watermarking methodology was known, and black-box, where no prior knowledge was available. For the beige-box scenario, we design an adaptive VAE-based attack for effective watermark removal, coupled with test-time optimization and frequency-aware color restoration to preserve image quality. We also show that simple spatial-domain translations can effectively disrupt TreeRing-based watermarks. In the black-box setting, we design targeted, cluster-specific removal attacks by grouping images based on spatial and spectral artifacts, followed by diffusion-based purification guided by semantic captions. Our approach outperformed the runner-ups by 26% and 31.7% in detection score on the beige-box and black-box tracks respectively, while preserving high perceptual quality (see Table 2 and Table 2). By exposing vulnerabilities in existing watermarking methods, we aim to inspire the development of more robust defenses against such attacks.

2 Related Work

Image Watermarking. Image watermarking embeds imperceptible signals into digital images to support authentication, copyright enforcement, and forensic traceability [8; 41; 38; 11]. Traditional image watermarking approaches operate in the spatial or frequency domains by modifying pixel values or transform-domain coefficients such as Discrete Cosine Transform [7], Discrete Wavelet Transforms [3], or Discrete Fourier Transform [23], often trading off between imperceptibility and robustness. More recently, deep learning-based watermarking schemes leverage convolutional neural networks and generative models to embed watermarks via learned feature spaces [62; 19]. Methods such as StegaStamp [53], Gaussian Shading [61], IConMark [43], Robin [20], and TreeRing [58] improve robustness under common corruptions (e.g., JPEG compression, resizing) and are increasingly adopted to tag AI-generated content. Despite their effectiveness in benign settings, these techniques remain vulnerable to both incidental degradation and targeted removal attacks [4; 66], underscoring the need for watermarking methods with stronger resilience.

Robustness of Image Watermarks. Image watermarks are vulnerable to degradation from common distortions such as Gaussian noise, blurring, and compression. More critically, adversarial attacks intentionally exploit model vulnerabilities to remove the watermark signal while maintaining high perceptual quality [21; 60; 14]. To mitigate these threats, recent approaches employ adversarial training, where watermarking models are optimized to withstand a range of perturbations [20; 54]. Generative models, including autoencoders and diffusion models, have also been leveraged to embed more resilient watermarks by aligning with the natural image manifold [20]. Despite these advances, achieving robustness without compromising imperceptibility remains an open challenge, especially with partial or complete knowledge of the watermarking algorithm [34; 12]. The NeurIPS 2024 Erasing the Invisible challenge [10] provides a benchmark for evaluating the resilience of state-of-the-art image watermarking methods under beige-box and black-box threat models.

Generative Models. Pretrained generative models have demonstrated strong performance across a range of vision tasks, including image restoration [6; 67; 46; 49; 47; 48], privacy preservation [50; 51], and adversarial purification [37; 44]. Leveraging their powerful priors over natural images, models such as VAEs [24], GANs [15], and diffusion models [17] can reconstruct high-fidelity content from corrupted or perturbed inputs. These capabilities have recently been explored in the context of watermarking, where the goal is to remove imperceptible signals embedded within images while preserving visual quality [29; 66]. For example, VAEs can project watermarked images into clean latent spaces that suppress hidden signals, while diffusion-based regeneration/rinsing methods have shown promise in erasing invisible perturbations [28; 4]. Recently, inspired by the inductive bias of untrained neural networks [55; 40; 39], such architectures have also been explored for invisible image watermark removal [26], demonstrating surprising effectiveness.

3 Proposed Approach

Challenge Overview: The NeurIPS 2024 competition Erasing the Invisible: A Stress-Test Challenge for Image Watermarks [10] rigorously assesses invisible watermarking robustness under realistic threat models. The benchmark comprises 300 watermarked images per track across two settings: a beige-box scenario, where the attacker knows the watermarking algorithm but not its parameters, and a black-box scenario, where no prior knowledge is available. Participants were tasked with removing invisible watermarks while preserving perceptual image quality. Evaluation was based on two criteria:

• •

Detection score measures watermark removal success as the true positive rate (TPR) at a 0.1% false positive rate (FPR). For each method, a detection threshold is set using the 0.001st percentile of distances $d(m,m^{\prime})$ between known messages $m$ and decoded outputs $m^{\prime}$ from 10,000 unwatermarked images. An attacked image is flagged as watermarked if its decoded message falls below this threshold. The detection score is then the fraction of images that were originally watermarked still flagged as containing a watermark.

• •

Image quality score is a weighted combination of low-level fidelity metrics (PSNR, SSIM [57], NMI) and high-level perceptual metrics (FID, CLIP-FID [25], LPIPS [64], Delta Aesthetics [10], and Delta Artifacts [10]). Each metric is normalized based on attack sensitivity and weighted accordingly: positive weights for metrics where lower is better (e.g., FID), and negative for those where higher is better (e.g., SSIM). This comprehensive multi-metric formulation ensures consistency and fairness across both pixel-level and perceptual domains.

We can query the competition leaderboard up to five times per day for both beige-box and black-box tracks to evaluate our submission performance. Below, we provide details of our proposed attack for both the beige-box and black-box tracks.

3.1 Beige-Box Track

In the beige-box track, we had access to the algorithmic description of the watermarking method used to embed invisible watermarks in the provided images, but not its hyperparameters or the underlying generative model. This partial prior knowledge enabled the design of an adaptive attack strategy that targeted the known watermarking scheme, akin to recent efforts [32; 9]. The organizers released 300 watermarked images, equally divided between two watermarking methods: a modified StegaStamp algorithm [53] and a variant of the TreeRing method [58]. We designed a dedicated watermark removal pipeline for each method.

3.1.1 Stegastamp Watermark Removal

For the StegaStamp-based watermark removal, we developed a three-stage pipeline: (i) paired data generation, (ii) adaptive VAE fine-tuning, and (iii) post-processing for quality restoration. This structured methodology enabled effective suppression of the embedded watermark while preserving the perceptual quality of the images. The overall pipeline is shown in Figure 2.

Paired Dataset Generation: We first curated a comprehensive training dataset leveraging 1,000 text prompts from the Hugging Face Stable-Diffusion-Prompts dataset [16]. Using these prompts, we generated corresponding images via Stable Diffusion 2-1 with a guidance scale of 7.5 and 50 inference steps. Each generated $512^{2}$ image was resized to $400^{2}$ pixels using bilinear interpolation before being processed through a pretrained StegaStamp model from the WAVES repository [4]. The key aspect of our dataset preparation involved creating image pairs where each original image was encoded with both a 100-bit binary message $m$ sampled uniformly at random and its inverse $1-m$, resulting in a dataset of 1,000 paired examples that captured watermarking artifacts.

Adaptive VAE Finetuning: The core of our attack framework centers on a Variational Autoencoder (VAE) that was adaptively tuned to perform watermark removal via supervised reconstruction. Given a watermarked image $x_{w}$ and its inverse message counterpart $x_{i}$, the VAE - consisting of encoder $E_{\theta}$ and decoder $D_{\phi}$-was optimized to reconstruct $x_{i}$ from $x_{w}$, using the following MSE loss:

[TABLE]

where $D_{\phi}(E_{\theta}(x_{w}))$ represents the reconstructed image from the watermarked input, and $x_{i}$ is the target image containing the inverted message. We optimize this objective using Adam optimizer with learning rate $\alpha=1\times 10^{-5}$ for 10 epochs with a batch size of 16. To stabilize training, we employ gradient clipping with a maximum norm of 1.0. Model training was performed on an NVIDIA A6000 GPU (48GB VRAM) and completed in under two GPU hours. We adapted a pretrained SDXL VAE model. This adaptive fine-tuning stage enabled the model to effectively strip away the watermark while preserving image structure. The VAE Finetuning algorithm is given in Algorithm. 1.

Quality-Preserving Post-Processing: Despite strong watermark removal performance, VAE reconstructions degraded color and contrast fidelity as shown in Figure. 3. To address this issue, we introduced a two-stage post-processing pipeline (Algorithm 2) with the aim to enhance the image quality without re-introducing the removed watermark.

• •

Test-Time VAE Optimization: Using the SDXL Refiner VAE, we performed image-specific optimization by fine-tuning the VAE parameters $\{\theta,\phi\}$ to better align with the original watermarked input $x_{w}$. The loss function combined pixel-wise, perceptual (LPIPS), and structural (SSIM) terms:

[TABLE]

While this step effectively removes the watermark signal, it can introduce slight degradation in visual quality, motivating the need for the second stage.

• •

Color and Contrast Transfer: The second stage restores perceptual quality by adjusting color and contrast in the CIELAB color space. Let $x_{\text{opt}}$ denote the output from test-time optimization, with CIELAB components $\{L_{\text{opt}},a_{\text{opt}},b_{\text{opt}}\}$, and let $\{L_{w},a_{w},b_{w}\}$ be those of the original watermarked image $x_{w}$. For color transfer, we preserve the luminance $L_{\text{opt}}$ from the optimized image and adopt the chrominance components from the watermarked image, yielding an intermediate image $x_{c}=\mathcal{F}_{\text{RGB}}(L_{\text{opt}},a_{w},b_{w})$, where $\mathcal{F}_{\text{RGB}}$ denotes conversion from CIELAB to RGB space. Next, for contrast transfer, we match the statistical moments of the luminance channel. Let $\mu_{c},\sigma_{c}$ and $\mu_{w},\sigma_{w}$ represent the mean and standard deviation of the luminance channels of $x_{c}$ and $x_{w}$, respectively. The adjusted luminance is computed as $L_{\text{final}}=\frac{\sigma_{w}}{\sigma_{c}}(L_{c}-\mu_{c})+\mu_{w}$. The final image is reconstructed as:

[TABLE]

The color and contrast transfer step enhances visual fidelity without reintroducing the watermark. For an ablation of this stage, see Table 3.

As shown in Table 4, the complete quality-preserving post-processing pipeline significantly outperforms VAE fine-tuning alone across all evaluation metrics. Specifically, PSNR improves by over 6 dB and SSIM increases by 0.176, while perceptual distance measures such as LPIPS, FID, and CLIPFID are markedly reduced. These gains highlight the effectiveness of combining image-specific test-time optimization with CIELAB-based color and contrast transfer, which restores visual fidelity without reintroducing the removed watermark.

3.1.2 TreeRing Watermark

For images embedded with TreeRing watermarks [58], we discovered a notable vulnerability to phase perturbations in the frequency domain. Specifically, modifying the phase component of the Fourier spectrum induces a spatial translation in the image domain—an operation that can effectively disrupt the watermark signal without introducing noticeable perceptual changes.

Building on this insight, we implemented a lightweight spatial-domain defense by applying a horizontal shift to each image as $x_{\text{shifted}}=\mathcal{T}(x_{w},\Delta x)$, where $x_{w}$ is the input watermarked image, $\mathcal{T}$ denotes a horizontal translation operator, and $\Delta x=7$ pixels is the empirically determined optimal shift that achieves a favorable trade-off between watermark removal and perceptual fidelity (see Figure 4).

However, spatial translation alone introduce visual artifacts near image boundaries. To counter this, we selectively restore the leftmost $\Delta x$ columns from the original image:

[TABLE]

This simple yet effective approach removes TreeRing watermarks while maintaining high image quality and requires no training or additional models, making it computationally efficient.

3.2 Black-Box Track

In the black-box track, no prior information was provided about the underlying watermarking algorithm. We adopted a data-driven strategy to infer potential watermarking mechanisms by systematically analyzing signatures in the spatial and frequency domains. Since different watermarking schemes inherently leave characteristic artifact patterns, either as spatial distortions or as structured frequency components, these observations directly guided our clustering process. Through comprehensive analysis, we partitioned the 300 watermarked images into four distinct clusters, as shown in Figure 5: Cluster-1: no discernible artifacts, Cluster-2: border-like artifacts in the spatial domain, Cluster-3: circular patterns in the Fourier magnitude spectrum, Cluster-4: square patterns in the Fourier magnitude spectrum. This clustering allowed us to develop targeted removal strategies tailored to each artifact signature, significantly improving both the efficacy of watermark removal and the preservation of image quality compared to uniform parameter settings.

3.2.1 Image-to-Image Diffusion Models for Watermark Removal

We leveraged the image-to-image capabilities of the Stable Diffusion Refiner model [35] as a core component of our black-box watermark removal strategy. This approach exploits diffusion models’ learned prior over natural images to project watermarked inputs onto the clean image manifold, effectively suppressing embedded signals while preserving semantic content. The process comprises two stages: (i) forward diffusion, which progressively corrupts an image with Gaussian noise, and (ii) reverse diffusion, where a denoising network reconstructs the image by iteratively removing noise.

Given a watermarked image $x_{w}$, the forward process adds noise according to schedule $\{\alpha_{t}\}_{t=1}^{T}$, where strength parameter $s\in[0,1]$ controls the initial corruption level via $\alpha_{1}=1-s$ as as $x_{t}=\sqrt{\alpha_{t}}x_{w}+\sqrt{1-\alpha_{t}}\epsilon_{t}$, where $\epsilon_{t}\sim\mathcal{N}(0,I)$. Here, the parameter $s$ critically balances watermark removal efficacy against content preservation: lower values ($s\approx 0.04$) maintain fine details but may insufficiently suppress watermarks, while higher values ($s\approx 0.25$) remove watermarks more aggressively at the risk of semantic drift (see Figure 6). Subsequently, the reverse diffusion process iteratively removes noise through a learned denoising function $\epsilon_{\theta}$, reconstructing the final refined image as $x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(x_{t},t)\right)$.

We configured the diffusion process with $T=500$ inference steps to enable sufficient iterative refinement for thorough watermark suppression. To preserve semantic content, we further introduced semantic guidance via ChatGPT-4-generated captions [2] that provide accurate, content-specific descriptions of each image (e.g., “a medieval castle on a hilltop surrounded by forests”). These captions anchor the denoising trajectory, ensuring that reconstructed regions align with natural image statistics rather than residual watermark patterns. The classifier-free guidance scale was set to $w=1.0$ to maintain photorealism: higher values ($w>2$) increase text condition strength but tend to introduce stylization artifacts inconsistent with the original content [52]. By tuning the strength parameter $s$ for each cluster, we achieved a balanced trade-off between aggressive watermark removal and high perceptual fidelity.

3.3 Cluster-Specific Solutions

Our empirical clustering analysis revealed distinct artifact patterns, enabling the design of adaptive, cluster-tailored watermark removal strategies rather than relying on a single, global configuration. This specialization significantly improved both perceptual quality and watermark suppression efficacy.

Cluster 1: No discernible artifacts. In the absence of visible spatial or frequency-domain cues, we adopted a high-strength image-to-image diffusion configuration ($s=0.16$). This choice prioritizes aggressive denoising to suppress potential low-energy watermark embeddings, while relying on semantic guidance to recover fine details without introducing perceptual distortions.

Cluster 2: Boundary artifacts. For images exhibiting border-like spatial patterns, we employed a three-stage removal pipeline, as in the Beige-box track (detailed in Sec. 3.1.1):

1. (i)

Paired Synthetic dataset generation, where each pair consists of two images watermarked with the same spatial pattern but opposite bit sequences. 2. (ii)

VAE fine-tuning to minimize reconstruction loss between watermarked inputs and clean target images, thereby learning an implicit watermark removal prior. 3. (iii)

Post-processing enhancement via CIELAB-space color transfer and local contrast adjustment to restore natural appearance without reintroducing watermark signals.

Cluster 3: Circular patterns in the Fourier spectrum. This cluster was treated using the same three-stage beige-box track pipeline as Cluster 2.

Cluster 4: Square patterns in the Fourier spectrum. We adopted a hybrid approach combining low-strength ($s=0.04$) image-to-image diffusion to minimally perturb fine details, followed by a 7-pixel horizontal translation with selective restoration, similar to the TreeRing watermark removal method (detailed in Sec. 3.1.2).

Across clusters, hyperparameters (especially $s$) were tuned to balance removal aggressiveness vs. content fidelity. This cluster-specific methodology outperformed uniform parameter settings, yielding notable improvements in both qualitative assessments and quantitative metrics. Details for Black-box approach are provided in Algorithm 3.

4 Additional Experiments

To further understand the trade-off between watermark removal strength and perceptual quality, we conducted a set of controlled experiments across different diffusion models and sampling configurations. These results, while not part of the main leaderboard evaluations, provide quantitative support for the hyperparameter choices.

Quality Metrics Trends Across Tracks and Clusters. Figure 8 evaluates our optimized pipeline across both beige-box tracks and four black-box clusters. The results highlight distinct quality–removal trade-offs. Beige-box StegaStamp achieves the most balanced outcome (PSNR: 28.06$\pm$1.2, SSIM: 0.82$\pm$0.04), with high semantic fidelity (CLIP-FID $<$ 5) and strong pixel-wise alignment—attributable in part to our CIELAB-based post-processing stage, which restores fine-scale color consistency after watermark removal. In contrast, TreeRing and Black-box Cluster with square Fourier artifacts show markedly lower PSNR and SSIM, a direct consequence of the 7-pixel horizontal translation that disrupts watermark embeddings; despite this, semantic similarity remains high (CLIP-FID $<$ 5). Finally, Black-box Cluster with no artifacts exhibits the largest perceptual gap, indicating that its imperceptible watermark necessitated more aggressive interventions, with some loss of fine-grained detail.

Impact of Diffusion Strength and Model Choice.

Figure 9 presents evaluation of four Stable Diffusion variants across strength parameters $s\in[0.04,0.25]$ and inference steps $T\in\{25,100\}$. Quality degradation exhibits a sharp nonlinearity at $s\approx 0.10$, where PSNR drops from 26 to 22 dB, motivating our conservative $s=0.04$ for structured artifacts versus aggressive $s=0.16$ for hidden watermarks. Similarly, extended denoising (100 vs. 25 steps) yields marginal improvements ($\sim$ 0.5 dB PSNR, 0.02 LPIPS). In addition, from our leaderboard submissions we observe that stable-diffusion-xl-refiner-1.0 is more effective at watermark removal than other tested diffusion models.

5 Exploratory but Ineffective Approaches

In parallel to developing our primary solution, we systematically explored multiple alternative strategies for watermark removal. While each was grounded in plausible hypotheses, none yielded competitive results in terms of the removal–quality trade-off. Below, we summarize these unsuccessful attempts along with representative qualitative outcomes.

Watermark Removal via ControlNet-Based Edge Guidance. We explored a ControlNet-based [63] strategy where the Canny edges of each watermarked image were extracted and used as structural guidance. To provide semantic conditioning, we generated descriptive captions using ChatGPT, explicitly incorporating both scene content and color attributes (e.g., “a red vintage car parked in front of a rustic wooden cabin, surrounded by snow”). These captions aimed to preserve fine-grained semantic and chromatic fidelity in the regenerated output. The edge maps and captions were then fed into ControlNet with Stable Diffusion to synthesize new images that aligned closely with the original structure and color scheme. Despite producing visually coherent outputs, as shown in Figure 10, the watermarks were largely retained. This outcome aligns with recent findings that watermark signal components often survive structural-guided regeneration when the conditioning enforces strong pixel-level consistency with the original image [33]. In the qualitative example, the top row shows the original watermarked images, and the bottom row shows the ControlNet-generated outputs, which maintain the global scene layout but fail to remove the embedded watermark.

Aggressive Diffusion Rinsing. We applied aggressive “diffusion rinsing,”,where the image is repeatedly passed through forward–reverse diffusion cycles in latent space, effectively purifying the image by destroying and reconstructing its content multiple times. A four-pass rinsing cycle removed most visible watermark traces; however, as shown in the first row of Figure 11, this came at the cost of severe perceptual degradation, with washed-out textures and loss of fine details. Pixel-space diffusion models were also tested but did not yield noticeable improvements in removal performance.

White-Boxes Overlay. In the early competition phase, we experimented with directly adding white-box perturbations to the input images with the aim of disrupting watermark patterns. This not only failed to erase the watermarks but also caused quality degradation, producing visually unnatural textures, as evident in the second row of Figure 11.

Untrained Neural Network Reconstruction. Motivated by the premise that untrained neural networks capture natural image priors [55], we attempted to reconstruct the original watermarked images from scratch using an untrained convolutional decoder. This approach succeeded in attenuating most watermarks, but reconstruction artifacts, oversmoothed textures, and loss of color fidelity were evident, as seen in the third row of Figure 11. The method’s inherent slow convergence and inability to perfectly reproduce fine structures limited its practicality for competition-scale deployment.

Additional Unsuccessful Attempts. Beyond the above failed strategies, we explored several other methods that ultimately proved ineffective. We implemented adversarial attacks on an ensemble of latent-space encoders, following the methodology outlined in ding2024waves, but these did not disrupt the embedded watermarks in a meaningful way. We also attempted super-resolution and inpainting-based perturbations, hypothesizing that re-synthesizing high-frequency details or contextually filling regions might break watermark structures; however, these approaches only introduced visual artifacts without consistent removal. Other tested methods included color space perturbations, frequency domain masking, and hybrid denoising–sharpening pipelines, all of which degraded perceptual quality while leaving partially intact watermark traces.

6 Conclusion

In conclusion, our work provides a comprehensive examination of the vulnerabilities in current invisible watermarking schemes, demonstrating that even under restrictive threat models, including the challenging black-box setting, determined attackers can effectively strip watermarks while preserving perceptual quality. Our results, which topped Beige-box and Black-box NeurIPS 2024 competition tracks, highlight not only the feasibility but also the generality of such removal strategies across fundamentally different watermark methods. This calls for a rethinking of watermark robustness, emphasizing the urgent need for schemes that can withstand adaptive, high-capacity generative manipulation without sacrificing utility, thereby ensuring reliable provenance tracking in the era of powerful image synthesis models.

Acknowledgment

We would like to thank Mucong Ding, Tahseen Rabbani, Bang An, and Chenghao Deng for their consistent availability and support throughout the competition, particularly in resolving Codabench-related issues. We are also grateful to Tom Goldstein and Furong Huang for their efforts in organizing the NeurIPS 2024 watermarking competition and for fostering a platform to advance research on robustness and provenance in generative models.

Appendix

This appendix provides background information and extensive qualitative results. For clarity, we summarize the contents and their page numbers below.

Background . Background

Figure 12 Qualitative results of Beige-box track on StegaStamp images. 12

Figure 13 Qualitative results of Beige-box track on TreeRing images. 13

Figure 14 Qualitative results of Black-box track on images with no artifacts. 14

Figure 15 Qualitative results of Black-box track with boundary artifacts images. 15

Figure 16 Qualitative results of Black-box track with circular Fourier artifacts images. 16

Figure 17 Qualitative results of Black-box track with square Fourier artifacts images. 17

Figure 18 Qualitative results of Black-box track with varying diffusion strength. 18

Figure 19 Captions generated by ChatGPT for watermarked images. 19

Figure 20 Captions generated by ChatGPT for watermarked images. 20

Figure 21 Effect of spatial translation on TreeRing watermarks. 19

Table 6 Quantitative results of our approach with and w/o pixel restoration. 6

Figure 22 Captions generated by ChatGPT for ControlNet model. 22

Background

6.1 StegaStamp: Invisible Hyperlinks in Photographs

StegaStamp [53] introduced one of the first end-to-end learned approaches for embedding invisible information into images that remain robust under real-world conditions. The key idea is to train an encoder–decoder system jointly while simulating the distortions introduced by printing and re-capturing photographs, such as perspective changes, blur, noise, color variation, and compression. The encoder embeds a short bitstring into the image with minimal perceptual difference, while the decoder is trained to recover the message reliably after these transformations. To further improve resilience, error-correcting codes are used to ensure accurate recovery even when parts of the image are degraded. Experiments demonstrated that StegaStamp achieves high decoding accuracy across different combinations of printers, displays, and cameras, while maintaining near-identical image quality. This work showed that imperceptibility and robustness can be achieved together, moving beyond earlier digital-only steganography methods and establishing a framework for watermarking that extends into practical physical-world settings.

In the context of the NeurIPS watermarking competition, StegaStamp is particularly relevant because the Beige-box track uses a variant of its algorithm to generate watermarked images. Understanding its design principles and robustness objectives provides a clear basis for developing effective removal strategies, since the competition task required eliminating such watermarks while maintaining image quality.

6.2 Tree-Ring Watermarks: Fingerprints for Diffusion Images

Tree-Ring watermarks [58] addressed the challenge of watermarking in the context of diffusion-based generative models. The approach embeds circular “tree-ring” patterns in the Fourier phase of generated images, producing signals that are imperceptible in the spatial domain but remain stable under common transformations such as resizing, compression, cropping, and mild editing. By embedding during the image generation process itself, rather than applying marks after generation, the method ensures that the watermark is more deeply integrated and harder to remove. Detection is carried out in the frequency domain using statistical classifiers, enabling reliable identification of watermarked content at scale. This design makes tree-ring watermarks more robust than many prior spatial-domain techniques and provides a practical mechanism for verifying the provenance of AI-generated content. The work highlights the importance of designing watermarking strategies that align with the properties of modern generative models, establishing a foundation for authenticity and accountability in synthetic media.

For the competition, Tree-Ring Watermarks formed the basis of another watermarking method in the beigebox track. Their reliance on structured frequency-domain perturbations made removal more challenging, requiring participants to design attacks that disrupt the Fourier-phase patterns while preserving spatial fidelity. Understanding the strengths of tree-rings thus directly shaped the strategies needed for effective watermark removal.