NIC-RobustBench: A Comprehensive Open-Source Toolkit for Neural Image Compression and Robustness Analysis
Georgii Bychkov, Khaled Abud, Egor Kovalev, Alexander Gushchin, Sergey Lavrushkin, Dmitriy Vatolin, Anastasia Antsiferova
TL;DR
NIC-RobustBench is an open-source framework that evaluates the adversarial robustness of neural image compression methods, addressing a gap in existing benchmarks by including attacks, defenses, and downstream impact analysis.
Contribution
It introduces a comprehensive benchmark for NIC robustness, integrating multiple attacks, defenses, and evaluation tools, and provides extensive empirical analysis of NIC vulnerabilities and resilience.
Findings
Identified failure modes of NIC models under attack
Highlighted architectures with highest resilience
Provided insights into robustness trade-offs
Abstract
Neural image compression (NIC) is increasingly used in computer vision pipelines, as learning-based models are able to surpass traditional algorithms in compression efficiency. However, learned codecs can be unstable and vulnerable to adversarial attacks: small perturbations may cause severe reconstruction artifacts or indirectly break downstream models. Despite these risks, most NIC benchmarks only emphasize rate-distortion (RD) performance, focusing on model efficiency in safe, non-adversarial scenarios, while NIC robustness studies cover only specific codecs and attacks. To fill this gap, we introduce \textbf{NIC-RobustBench}, an open-source benchmark and evaluation framework for adversarial robustness of NIC methods. The benchmark integrates 8 attacks, 9 defense strategies, standard RD metrics, a large and extensible set of codecs, and tools for assessing both the robustness of the…
Peer Reviews
Decision·Submitted to ICLR 2026
a) Compared with existing NIC libraries that mostly focus on RD (CompressAI, TFC, NeuralCompression), this is the first one that systematically bakes in both attacks and defenses, and it clearly has the widest model set so far. b) Many “benchmark” papers stop at attacks; this one also implements reversible/geometric defenses and diffusion-style purification (DiffPure), so it supports end-to-end studies.
a) Novelty is mainly engineering/integration. The paper stands on putting things together, not on a new attack/defense or a new robustness metric. That’s OK for a benchmark, but then it must be very explicit in positioning vs. CompressAI (Bégain et al. 2020), TFC (Ballé et al. 2024), NeuralCompression (Muckley et al. 2021). b) Only white-box attacks. The authors argue black-box is costly, but for a benchmark that wants to be comprehensive, skipping black-box / transfer attacks is a real limita
1. This paper provides a comprehensive compilation of adversarial attack methods, defense strategies, and evaluation metrics for NIC methods, offering a valuable resource for future research in this domain. 2. The experimental section is extensive, covering 10 mainstream NIC models, 6 attack methods, 7 defense strategies, and multiple datasets, offering numerous case studies. 3. The modular design in the framework diagram reflects excellent engineering, facilitating reproducibility and ensuring
1. The paper focuses more on the practical tool aspect and lacks theoretical discussions, particularly regarding the relationship between adversarial robustness and compression mechanisms. 2. The attacks and defense strategies presented in the paper are derived from existing research; the paper mainly integrates these methods rather than proposing novel algorithms, lacking a degree of innovation.
1. The paper includes an open-source package for experimenting with many state-of-the-art NIC codecs. 2. The paper finds interesting trends about types of models and robustness (e.g., larger models are less secure, higher compression rates are the most robust).
1. The paper has limited novelty and contribution. Specifically, all of the NIC models and attacks are from prior work. Most of the metrics are also borrowed from prior work, with the exception of $\delta$, which the paper does not motivate well. Finally, there is very limited discussions or theoretical insight into the empirical findings. 2. The evaluation datasets are limited and not ideal for image compression tasks. The Cityscapes and NIP 2017 datasets have very low resolution images and Kod
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage Processing Techniques and Applications · Neural Networks and Applications · Image and Signal Denoising Methods