EnsIR: An Ensemble Algorithm for Image Restoration via Gaussian Mixture Models

TL;DR

This paper introduces EnsIR, a model-agnostic, training-free ensemble method for image restoration that uses Gaussian mixture models and EM algorithm to improve prediction accuracy across multiple tasks.

Contribution

It reformulates ensemble learning for image restoration into GMMs and employs an EM-based approach for efficient, inference-stage ensemble weight estimation, outperforming existing methods.

Findings

Outperforms regression-based and averaging ensembles on 14 benchmarks

Efficient, model-agnostic, and training-free ensemble inference method

Effective across super-resolution, deblurring, and deraining tasks

Abstract

Image restoration has experienced significant advancements due to the development of deep learning. Nevertheless, it encounters challenges related to ill-posed problems, resulting in deviations between single model predictions and ground-truths. Ensemble learning, as a powerful machine learning technique, aims to address these deviations by combining the predictions of multiple base models. Most existing works adopt ensemble learning during the design of restoration models, while only limited research focuses on the inference-stage ensemble of pre-trained restoration models. Regression-based methods fail to enable efficient inference, leading researchers in academia and industry to prefer averaging as their choice for post-training ensemble. To address this, we reformulate the ensemble problem of image restoration into Gaussian mixture models (GMMs) and employ an expectation maximization (EM)-based algorithm to estimate ensemble weights for aggregating prediction candidates. We estimate the range-wise ensemble weights on a reference set and store them in a lookup table (LUT) for efficient ensemble inference on the test set. Our algorithm is model-agnostic and training-free, allowing seamless integration and enhancement of various pre-trained image restoration models. It consistently outperforms regression based methods and averaging ensemble approaches on 14 benchmarks across 3 image restoration tasks, including super-resolution, deblurring and deraining. The codes and all estimated weights have been released in Github.