A Comprehensive Real-World Assessment of Audio Watermarking Algorithms: Will They Survive Neural Codecs?

TL;DR

This paper presents RAW-Bench, a standardized benchmark for evaluating deep learning-based audio watermarking algorithms under realistic distortions, revealing neural codecs as a major challenge and highlighting the importance of attack-aware training.

Contribution

Introduction of RAW-Bench, a comprehensive benchmark with a systematic attack pipeline for assessing audio watermarking robustness in real-world scenarios.

Findings

Neural compression techniques significantly challenge watermarking robustness.

Training with audio attacks improves robustness but is not always sufficient.

Certain distortions like reverb and time stretching severely impact specific methods.

Abstract

We introduce the Robust Audio Watermarking Benchmark (RAW-Bench), a benchmark for evaluating deep learning-based audio watermarking methods with standardized and systematic comparisons. To simulate real-world usage, we introduce a comprehensive audio attack pipeline with various distortions such as compression, background noise, and reverberation, along with a diverse test dataset including speech, environmental sounds, and music recordings. Evaluating four existing watermarking methods on RAW-bench reveals two main insights: (i) neural compression techniques pose the most significant challenge, even when algorithms are trained with such compressions; and (ii) training with audio attacks generally improves robustness, although it is insufficient in some cases. Furthermore, we find that specific distortions, such as polarity inversion, time stretching, or reverb, seriously affect certain methods. The evaluation framework is accessible at github.com/SonyResearch/raw_bench.