Revisiting the Generalization Problem of Low-level Vision Models Through the Lens of Image Deraining

TL;DR

This paper investigates why low-level vision models, especially in image deraining, struggle to generalize to unseen degradations, revealing that shortcut learning driven by content complexity is a key factor and proposing strategies to mitigate it.

Contribution

It uncovers the mechanism behind generalization failure in low-level vision models and introduces data balancing and prior-based methods to enhance robustness.

Findings

Shortcut learning causes overfitting to simple degradation patterns.

Balancing training data complexity improves generalization.

Pre-trained generative models help constrain outputs to high-quality image manifolds.

Abstract

Generalization to unseen degradations remains a fundamental challenge for low-level vision models. This paper aims to investigate the underlying mechanism of this failure, using image deraining as a primary case study due to its well-defined and decoupled structure. Through systematic experiments, we reveal that generalization issues are not primarily caused by limited network capacity, but rather by a ``shortcut learning'' phenomenon driven by the relative complexity between image content and degradation patterns. We find that when background content is excessively complex, networks preferentially overfit the simpler degradation characteristics to minimize training loss, thereby failing to learn the underlying image distribution. To address this, we propose two principled strategies: (1) balancing the complexity of training data (backgrounds vs. degradations) to redirect the network's focus toward content reconstruction, and (2) leveraging strong content priors from pre-trained generative models to physically constrain the network onto a high-quality image manifold. Extensive experiments on image deraining, denoising, and deblurring validate our theoretical insights. Our work provides an interpretability-driven perspective and a principled methodology for improving the robustness and generalization of low-level vision models.