Few-Shot Meta-Denoising

TL;DR

This paper introduces a meta-learning approach for few-shot denoising that improves generalization by training on synthetic noise models and fine-tuning on small real datasets, outperforming traditional methods.

Contribution

The work proposes a novel meta-learning training method for few-shot denoising, enabling models to generalize better from limited data by leveraging synthetic noise during meta-training.

Findings

Meta-learning improves denoising accuracy over supervised and transfer learning.

The approach generalizes well to different noise types and signals.

Meta-learning shows potential as a primary method for denoising tasks.

Abstract

We study the problem of few-shot learning-based denoising where the training set contains just a handful of clean and noisy samples. A solution to mitigate the small training set issue is to pre-train a denoising model with small training sets containing pairs of clean and synthesized noisy signals, produced from empirical noise priors, and fine-tune on the available small training set. While such transfer learning seems effective, it may not generalize well because of the limited amount of training data. In this work, we propose a new meta-learning training approach for few-shot learning-based denoising problems. Our model is meta-trained using known synthetic noise models, and then fine-tuned with the small training set, with the real noise, as a few-shot learning task. Meta-learning from small training sets of synthetically generated data during meta-training enables us to not only generate an infinite number of training tasks, but also train a model to learn with small training sets -- both advantages have the potential to improve the generalisation of the denoising model. Our approach is empirically shown to produce more accurate denoising results than supervised learning and transfer learning in three denoising evaluations for images and 1-D signals. Interestingly, our study provides strong indications that meta-learning has the potential to become the main learning algorithm for denoising.

Figures

Tables

Table 1. Table 1: CT-Scan Evaluation. We examine two types of noise models, namely Gaussian and Poisson. We present the results for Gaussian and / or Poisson noise models. We evaluate on tube current with dose 10 % percent 10 10\% and 5 % percent 5 5\% . The evaluation metric is the peak signal-to-noise ratio (PSNR), where larger is better. Bold numbers indicate the best performance per column.

	# Tasks		Tube current
	Poisson	Gaussian	10%	5%
Initial Noise	-	-	38.28	35.18
Supervised Learning	4	0	36.50	34.15
	4	4	35.50	33.80
	8	0	36.31	33.94
	8	8	35.01	33.42
Transfer Learning	4	0	38.98	36.96
	4	4	37.91	35.99
	8	0	38.39	36.70
	8	8	39.19	36.76
Meta-Denoising	4	0	39.77	37.37
	4	4	39.42	37.50
	8	0	39.99	37.96
	8	8	39.84	37.49

Table 2. Table 2: Comparison with Non-learning Algorithms on CT-Scan. We compare the three learning-based algorithms with two classic denoising approaches (BM3D and LGP-PCA). The evaluation metric is the peak signal-to-noise ratio (PSNR), where larger is better. The execution time is shown in seconds. Bold numbers indicate the best performance per column.

	Tube current		Execution (sec.)
	10%	5%
Initial Noise	38.28	35.18	-
BM3D	38.70	36,27	2.42
LGP-PCA	38.98	36,29	206.84
Supervised Learning	36.50	34.15	0.0008
Transfer Learning	39.19	36.96	0.0008
Meta-Denoising	39.99	37.96	0.0008

Table 3. Table 3: Natural Image Evaluation. We evaluate on the smartphone dataset  [ 1 ] our meta-denoising approach, supervised- and transfer- learning. The evaluation metric is the peak signal-to-noise ratio (PSNR) and tasks are generated from a Gaussian and Poisson distributions. The best result per column is in bold font.

	# Tasks		PSNR
	Gaussian	Poisson	DnCNN	xDnCNN
Initial Noise	-	-	27.61
Supervised Learning	3	0	22.23	26.71
	3	3	25.74	26.25
Transfer Learning	3	0	28.54	29.11
	3	3	30.01	28.30
Meta-Denoising	3	0	29.88	29.58
	3	3	31.71	29.51

Table 4. Table 4: Comparison with Non-learning Algorithms on Natural Images. We compare the three learning-based algorithms with two classic non-learning denoising approaches (BM3D and LGP-PCA). The execution time in seconds is reported too. The best result per column is in bold font.

	PSNR	Execution (sec.)
Initial Noise	27.61	-
BM3D	34.21	273.7
LGP-PCA	32.41	11009
Supervised Learning	26.71	0.001
Transfer Learning	30.01	0.001
Meta-Denoising	31.71	0.001

Table 5. Table 5: We evaluate our approach on ECG sequences. The supervised-learning is trained on the same data as our approach. The transfer-learning has been fine-tuned on the same k-shot data that we use during fine-tuning. The evaluation metric is the signal-to-noise ratio (SNR), larger is better. Best result is shown in bold font.

	# Tasks	SNR(db)
Initial Noise	-	-6.16
Supervised Learning	50	-5.80
	100	-5.45
Transfer Learning	50	0.53
	100	2.82
Meta-Denoising	50	2.00
	100	2.97

Table 6. Table 6: K-shot learning experiment. We evaluate our approach and transfer learning for 1-, 3-, 5-, 7- and 10-shot learning on the CT-Scan database. The evaluation metric is PSNR. Best result per column is shown in bold font.

	k-shot	10%	5%
Initial Noise	-	38.28	35.18
Transfer Learning	1	36.82	34.94
	3	38.43	36.21
	5	39.28	36.79
	7	38.87	36.88
	10	38.98	36.96
Meta-Denoising	1	35,94	33,82
	3	38.90	36.90
	5	39.30	37.40
	7	39.72	37.87
	10	39.99	37.96

Equations

$$\mathcal{L}(\theta ,\mathbf{x},\mathbf{y})={\mathbb{E}}_{\mathbf{x},\mathbf{y}}[f_{\theta }(\mathbf{x})-\mathbf{y}{]}^2.$$

$${\theta }_i^{\prime }=g(\mathcal{L}(\theta ,h_{\tau }(\mathbf{y}),\mathbf{y}),\theta ,s),$$

$$\theta \leftarrow \theta +ϵ\frac{1}{n}\sum _{i=1}^n({\theta }_i^{\prime }-\theta ),$$

Full text

1 Introduction

Learning-based denoising methods typically require a large training set with pairs of clean and noisy samples, where the simultaneous access to the pair is usually challenging. In this framework, training sets are often built using synthetic noise that is added to images based on a prior noise model [42]. The use of prior noise models for training data generation models is also common in medical imaging, where the noise distribution is usually assumed to be known [45, 48]. In other cases, such as with motion or image signals [1, 9], it is possible to employ multiple sensors for simultaneous capturing of clean and noisy signals. However, this setup cannot be easily replicated in other real-world scenarios. Consequently, the availability of training sets that contain a large amount of pairs of clean and real noisy samples is limited, making the study of few-shot learning-based denoising an important research topic.

To frame our problem, we assume that: 1) we have access to clean samples (i.e. noiseless), 2) we have empirical prior knowledge of the real noise distribution, and 3) we have a small training set containing pairs of clean and noisy samples affected by real noise. In this context, we introduce a new meta-learning training approach for few-shot learning-based denoising problems. In our proposed training approach, the meta-training stage consists of learning how to quickly denoise from a limited number of synthetically generated noisy samples. Afterwards, the fine-tuning stage comprises the learning from the limited training set containing pairs of clean and noisy samples affected by real noise. To the best of our knowledge, this paper is the first to propose a meta-learning method for few-shot learning in denoising problems.

Traditionally, signal denoising has been addressed by non-learning approaches [13, 26, 38], where solutions have only made the assumption (2) above. However, learning-based works have shown more effective denoising results, when there is sufficient amount of training data [49]. Denoising with supervised learning has been explored in three different ways. First, one can directly train a denoising model based on a small training set (assumption 3 above), but current deep learning models are likely to over-fit such small training sets. Second, the denoising model can be trained with supervision using samples formed from synthetic noise (assumptions 1 and 2), while it is tested on the real noise. The lack of fine-tuning on real noise typically results in low denoising accuracy [40]. Alternatively, in transfer learning, a denoising model is pre-trained with pairs of clean and synthetically generated noisy samples, where the noise is based on a known model (e.g. Gaussian or Poisson). Then, this pre-trained model is fine-tuned with the small training set containing real noise. Unfortunately, depending on the size of the small training set with real noise, transfer learning may not generalize well [46].

In this paper, we propose a new and effective meta-learning approach for learning-based few-shot denoising problems. Although we share the same assumptions of transfer learning, our paper is motivated by the success of meta-learning in few-shot classification [22]. The most successful few-shot approaches are based on meta-learning, where the model learns to quickly adapt to new learning tasks using small training sets [32, 37]. We argue that the success of meta-learning is due to: 1) the ability to generate potentially infinite training sets containing synthetic noise (like transfer learning); and 2) the ability to train a model that quickly adapts to new learning tasks with small training sets (unlike transfer learning). In our experiments, we empirically show that the denoising results produced by meta-learning are better than the ones from transfer learning, and other non-learning and learning-based denoising approaches.

We assess our work on three different data sets: low dose CT scans, natural images, and electrocardiogram (ECG) sequences. The demonstration of functionality in these diverse data sets shows the generalization ability of our proposed meta-denoising learning method, and allows us to compare it with relevant denoising methods. In all data sets, we rely on prior noise models for synthesizing noisy signals that partially capture the real noise affecting the image or signal. Our evaluations show that meta-learning outperforms supervised learning and transfer learning in few-shot denoising scenarios, given a sufficient number of learning tasks. Therefore, the main contribution of this paper is to be the first to show the potential of meta-learning to become a standard few-shot learning-based denoising algorithm.

2 Related Work

Traditional Denoising

This type of denoising has been performed with signal processing algorithms. On the image domain, it is common to assume that the real noise is related to the Gaussian and Poisson noise models [25]. Similarly, the variance-stabilizing transformation has been proposed for converting the noise to Gaussian noise and then denoise with standard algorithms such as BM3D [13] or [7]. In signal denoising, wavelets and filtering are the usual algorithms for denoising [26, 34, 38]. However, recent works have shown that learning-based approaches outperform such traditional denoising algorithms [1, 49].

Learning-based Denoising

Auto-encoders [41, 42] have standardized learning-based methods, followed by more complex network architectures [2, 44]. Convolutional neural networks [17, 33], multi-layer perceptron [8] and encoder-decoder [6, 23, 27, 49] architectures have also been proposed for image denoising. Unfortunately, since such methods are generally trained with supervision, they need large training sets of clean and noisy image pairs. However, as explained in Sec. 1, some denoising problems can have relatively small training sets and, consequently, be characterized as few-shot learning problem. Differently from previous learning-based denoising methods, our approach explicitly assumes that denoising is a few-shot learning problem. An alternative to be considered for solving this few-shot learning problem is transfer learning [5, 10], where the denoising model is pre-trained in some related denoising tasks (e.g., using large-scale data sets that may be available or building large-scale data sets by synthesizing noisy signals or images). Then, this pre-trained model is fine-tuned with the small training set. However, transfer learning does not usually generalize well from small data sets and requires substantial hyper-parameter tuning [46]. We show the transfer learning limitations in our evaluations, when compared with our meta-learning approach.

Meta-learning

The motivation for meta-learning is to train a model that is optimized to adapt to new learning tasks (sampled from a common, but latent, distribution of tasks), with relatively small training sets [16, 29, 36, 39]. Meta-learning happens at two levels. First, the meta-learner obtains knowledge across multiple tasks (sampled from the latent distribution of tasks) in order to improve the base learner to adapt to new tasks. Second, the base learner acquires fast knowledge of a specific task using a small number of annotated data. Based on this scheme, several approaches have been proposed, which explore: metric learning [43, 21, 37], memory augmentation (i.e. model-based) [35, 28] and optimization [14, 32, 30]. Meta-learning approaches that explore optimization methods learn how to update the base learner with an iterative method [3, 16, 24]. For example, the meta-learner in [32] is an LSTM that learns the update rule for the base learner. Instead of the recurrent net, the meta-learner is modelled as batch stochastic gradient descent in model agnostic meta-learning (MAML) [14] and Reptile [30]. Both approaches deliver state-of-the-art performance in few-shot classification. We have tried both algorithms for denoising and found that Reptile provides better training convergence and generalization than MAML. Furthermore, Reptile requires first-order gradients, which is computationally more efficient than MAML that works with second-order gradients [30]. For those reasons, we focus on Reptile in our study of meta-learning few-shot denoising algorithms.

To the best of our knowledge, this paper is the first to adapt meta-learning for the few-shot learning-based denoising problem.

3 Few-Shot Meta-Denoising Learning

We reformulate the optimization-based meta-learning method Reptile [30], which has been originally developed for few-shot classification, to work for few-shot image and signal denoising. Our meta-denoising approach is trained in two stages: 1) during meta-training, the tasks consist of small sets of synthesized noisy samples that have been formed by taking clean samples and applying synthetic noise sampled from known models; and then 2) a small training set of real noisy samples is used to fine-tune the denoising model. Note that the two stages listed above allow us to access an infinite number of tasks for the meta-training stage that can help in the learning of a generic denoising model that quickly adapts to new denoising tasks that contain small training sets.

3.1 Problem Definition

We define the clean signal as $\mathbf{y}\in\mathbb{R}^{D}$ and the corrupted signal as $\mathbf{x}\in\mathbb{R}^{D}$, where this corruption occurs due to noise that is specific to the signal formation. At each step of meta-learning, each clean signal is affected by a noise model, where task $\tau$ refers to the selected noise model and its parameters. For each task $\tau$, sampled from a latent distribution of tasks $p(\tau)$, we build a training set defined by $\{\mathcal{X}_{\tau},\mathcal{Y}_{\tau}\}$, where $\mathcal{X}_{\tau}=\{\mathbf{x}_{i}\}_{i=1}^{k}$, $\mathcal{Y}_{\tau}=\{\mathbf{y}_{i}\}_{i=1}^{k}$, and $\mathbf{x}_{i}=h_{\tau}(\mathbf{y}_{i})$, with $h_{\tau}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$ denoting the function that transforms the clean signal $\mathbf{y}_{i}$ into the noisy signal $\mathbf{x}_{i}$ by adding synthetic noise, according to the task $\tau$. The data set containing real noise is represented by $\{\mathcal{X}_{\tau^{*}},\mathcal{Y}_{\tau^{*}}\}$, where we assume that $\tau^{*}\sim p(\tau)$, and this set is divided into a training and a test set, denoted by $\{\mathcal{X}^{(t)}_{\tau^{*}},\mathcal{Y}^{(t)}_{\tau^{*}}\}$ and $\{\mathcal{X}^{(v)}_{\tau^{*}},\mathcal{Y}^{(v)}_{\tau^{*}}\}$, respectively.

3.2 Denoising Objective

The objective is to learn the parameters $\theta$ of a deep neural network that recovers the clean signal. The network is represented by $f_{\theta}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$ and is trained with back-propagation. Given a large training set of pairs of clean and noisy samples, the objective to minimize is

[TABLE]

This is the standard configuration for gradient-based supervised learning, which tends to work well when there is sufficient amount of training data. Since we assume that the available pairs of clean and noisy samples are limited, we define the problem of denoising within the context of few-shot meta-learning, which we refer to as meta-denoising.

3.3 Meta-denoising

We define the base learner to be the network $f_{\theta}(.)$. The learning process estimates the parameter $\theta$ that can be quickly adapted to denoising real signals, given a small training set, containing few pairs of clean and noisy signals. In the context of meta-denoising, a task corresponds to denoising pairs of clean and synthesized signals, formed by applying a known noise model to the clean signals. Note that a task $\tau$ is formed not only by noise models, but also by noise parameters. Given a sampled task $\tau_{i}\sim p(\tau)$, Reptile performs $s$ steps of gradient descent update on the loss from (1), described by:

[TABLE]

where $g(.)$ is the update operator that represents stochastic gradient descent or another optimization algorithm. This is the inner-loop update. After iterating over $n$ tasks, the model parameters $\theta$ are adapted towards the new parameters, defined as:

[TABLE]

where $\epsilon$ is the step-size. This is the outer loop of the algorithm, where across tasks knowledge is acquired. Note that instead of being explicitly modeled, the meta-learner is implicitly represented by the gradient-based learning in (3). The number of iterations for the inner and outer loops and the batch-size of the inner loop compose the hyper-parameters of the learning algorithm. At the end of the learning process, the model is fine-tuned with $\{\mathcal{X}^{(t)}_{\tau^{*}},\mathcal{Y}^{(t)}_{\tau^{*}}\}$ and tested with $\{\mathcal{X}^{(v)}_{\tau^{*}},\mathcal{Y}^{(v)}_{\tau^{*}}\}$ – this is the only step that involves annotated samples, i.e. pairs of clean and noisy samples, where the noise is real. The complete learning and assessment processes are presented in Algorithm 1.

4 Experiments

We compare the proposed meta-denoising few-shot learning algorithm with other non-learning and learning-based methods. We assess our few-shot meta-denoising approach in two image-based datasets and one 1-D dataset to demonstrate the generalization of our approach to different types of signals. Since this paper is the first to explore few-shot denoising with meta-learning, we propose a new assessment protocol.

Protocol. For meta-training, noise models are sampled and synthetic tasks are generated as a k-shot denoising problem, where k-pairs of clean and noisy samples are produced. During meta-testing, real tasks are used for k-shot fine-tuning. The evaluation is performed on the test set, which is composed of previously unseen pairs of clean and noisy samples from the real tasks. The evaluation metric is the peak signal-to-noise ratio (PSNR) for images and signal-to-noise ratio (SNR) for 1-D data. These are the standard metrics for evaluating denoising methods [18, 44].

Other Training Algorithms. We define two baselines for comparison with our approach. First, we train from scratch a supervised learning model, where the training set is composed of synthetically generated data and is as large as the training set of all tasks in meta-training. As a result, the amount of data processed by supervised learning is the same as by meta-training. Second, we perform transfer learning with the trained model using the same k-pairs of clean and real noisy samples that we employ for fine-tuning in meta-training. By fixing the model and data, we can compare meta-learning with supervised- and transfer-learning in a fair manner. Finally, we also report the initial noise for each evaluation that corresponds to the test set noise prior to applying a denoising algorithm.

Implementation. We present three experiments with different data types, so we employ different network architectures, optimization methods and hyper-parameter values. All employed architectures are taken from related publications, since proposing a new architecture is not the aim of this paper. The hyper-parameters have been chosen with grid-search optimization. The implementation of our approach, as well as the evaluation protocol will be made publicly available – both implementations were made using PyTorch [31].

Below, for each data set, we discuss the task generation, implementation details and denoising results. Finally, we examine the results significance using the t-test, different k-shots and we also provide results when training with the MAML algorithm.

4.1 CT-Scan Denoising Evaluation

Task Generation. The CT database in [45] contains CT scans of a deceased piglet. The scans are produced using a normal dose of tube current (100%) and reduced doses. We use the doses of 10% and 5% because they include substantial amount of noise, where each dose set has 850 images of size $D=512\times 512$. Following the data set protocol [45], 708 of those images are selected for training and 108 for testing. During training, each task consists of synthetic images generated by simulating reduced dose CT images by adding Poisson and Gaussian noise to the sinograms of the normal dose CT images, as in $\mathbf{x}=h_{\tau=\mathcal{P}(b),\mathcal{N}(0,\sigma^{2})}(\mathbf{y})$, where $\mathbf{x}=S^{-1}(\mathbf{z})$, $\mathbf{z}\sim\mathcal{P}(b\times\exp(-S(\mathbf{y})))+\mathbf{r}$, with $\mathcal{P}(.)$ denoting the Poisson distribution, $S(.)$ representing the process to generate the sinogram or the fan-beam projected data [48] from the clean CT image $\mathbf{y}$, $\mathbf{r}\sim\mathcal{N}(0,1)$ denoting the read-out electronic noise represented by $\mathcal{N}(.)$ the Gaussian distribution with mean $\mu$ and variance $\sigma$, and $b$ denoting the X-ray source intensity or a blank scan factor that controls the noise level. We create the tasks with $\mathbf{r}=0$, namely one per blank scan factor, where we use the following factors $b\in\{{10}^{4},{3\times 10}^{4},{5\times 10}^{4},{7\times 10}^{4},{10}^{5},{2\times 10}^{5},{3\times 10}^{5},$ ${5\times 10}^{5}\}$. Consequently, each factor contributes to the generation of a different task. To simulate the effect of the Gaussian noise, we add noise directly on the CT image $\mathbf{x}$. The Gaussian task $\tau$ is represented by Gaussian noise $\eta(i,j)\sim\mathcal{N}(0,\sigma^{2})$ with $\sigma^{2}\sim U(0.001,0.003)$, where $U(a,b)$ denoting uniform distribution in $[a,b]$. The result is the noisy image $\mathbf{x}(i,j)=\mathbf{y}(i,j)+\eta(i,j)$ for $i\in\{1,...,D_{height}\}$ and $j\in\{1,...,D_{width}\}$. Finally, we keep $k$ images for the meta-learning for each reduced dose set. The rest of the images are used for evaluation.

Model. We use a residual encoder-decoder architecture from [12] that has been proposed for low dose CT denoising [45]. The network consists of a 10-layer convolutional neural network with input and output sizes of $55\times 55$. The activation functions are all ReLU, except for the last layer that is linear. The optimization algorithm is Adam [19] with learning rate $10^{-4}$. The inner loop has 20 epochs with mini-batch size 20. The outer loop runs for 1000 epochs with step-size $\epsilon=0.1$ and the number of fine-tuning samples $k=10$.

Results. We evaluate CT-Scan denoising on tube current with doses $10\%$ and $5\%$, where smaller percentage denotes more noise – results are summarized in Table 1. We generate from 4 to 8 Gaussian and Poisson tasks, with each task representing an individual noise model. Notice that we also applied Gaussian and Poisson noise, at the same time, on an image as another task, but the performance was not as good as sampling independently. In the case of task combination (e.g. 4 Gaussian and 4 Poisson), we equally sample tasks from both noise models. We attempted to generate less than 4 and more than 8 tasks, but there was not a considerable performance difference.

The same evaluation is performed for the supervised- and transfer-learning. Comparing with these two algorithms, it is clear that our approach delivers the highest PSNR. Although the results from transfer learning are closer, the difference is significant based on the t-test as we show in Sec. 4.4. Since this is the only experiment where we have access to a relatively large training set containing pairs of clean and real noisy data, we check the upper bound performance for our approach, which is the supervised learning performance using this large training set. Note that such full supervision violates one of our three assumptions mentioned in Sec. 1 – in particular, the one regarding the availability of a large number of training samples. Supervised learning achieves 40.28dB for $10\%$ and 38.84dB for $5\%$, when trained with clean and real noisy training data. Our results in Table 1 indicate that we are close to this upper bound, although we train with synthetic training data. To further explore our approach, we meta-train it with clean and real noisy training data. Notably, our meta-denoising reaches 40.47dB for $10\%$ and 38.31dB for $5\%$. This result shows the potential of meta-learning to become the standard training algorithm for the task of denoising. We present visual results in Fig. 1.

Furthermore, we investigate how our method and the other learning-based algorithms from Table 1 compare to non-learning denoising algorithms. In Table 2, we present the best results from all learning-based approaches and compare with the non-learning approaches BM3D [13] and LGP-PCA [50]. It is clear that our meta-denoising, as well as, supervised- and transfer-learning outperform the non-learning approaches. Moreover, BM3D and in particular LGP-PCA are significantly slower in terms of execution time.

4.2 Natural Images Evaluation

Task Generation. We use the color data set BSD500 [4] as training set and evaluate on the SIDD database [1]. BSD500 data set is standard for image denoising [27, 49], while the SIDD has been recently introduced for denoising images from smartphones. The training and testing sets from BSD500 have 200 images which we employ as training data during meta-training. SIDD has 160 images from which we keep $10$ for the fine-tuning stage and the rest as test set. In BSD500, the size of images is $D=481\times 321\times 3$. In the training images, we add Gaussian noise in the spatial domain to generate tasks with synthetic noise, as in $\mathbf{x}=h_{\tau=\mathcal{N}(\mu,\sigma)}(\mathbf{y})$. This Gaussian noise has mean $\mu=0$ and standard deviation $\sigma\in\{15,25,50\}$, which have been chosen from the literature in image denoising [44, 49]). Second, we introduce Poisson noise in the image domain as in $\mathbf{x}=h_{\tau=\mathcal{P}(b)}(\mathbf{y})$, where $\mathbf{x}\sim\mathcal{P}(b\times 10^{12}\times\mathbf{x})/10^{12}$, with $\mathcal{P}(.)$ denoting the Poisson distribution, and $b\in\{10^{9},10^{9.5},10^{10}\}$ denotes the scale factor that controls the noise level. In total, we have three tasks for Poisson and three for Gaussian noise.

Model. The network is DnCNN [49], an encoder-decoder ConvNet that reconstructs the residual noisy image. We extend the original architecture from [49] to have a depth of $13$ layers to reduce the amount of resources required. The denoised signal is obtained by subtracting the predicted noise from the input noisy image. All layers of this network consists of convolutions, with batch normalization and ReLU activation, except the last layer, which is linearly activated. The input and output resolution image resolution is $55\times 55$ during training, while $40\times 40$ in testing. In addition, we evaluate a more recent version of the architecture DnCNN, xDnCNN [20] which uses the new activation function xUnit adapted for image restoration scenarios. The optimizer for the inner loop is Adam with learning rate $10^{-3}$, with 20 inner epochs. The step-size of the outer loop is $\epsilon=1e-4$ and the number of samples $k=10$.

Results. Although the noise of natural images is more complicated than the proposed Gaussian and Poisson tasks, we are able to denoise images with real noise [1] – Table 3 summarizes the results. We have used three Gaussian and Poisson tasks jointly, but independently. The results in Table 3 indicate that the supervised learning performs poorly because the generated tasks may not represent well the real noise in natural images. However, the tasks are enough for meta-learning, as shown by the results.

Similarly to the CT-Scans, we compare all learning-based algorithms to non-learning denoising approaches. In Table 4, we present the best results of meta-denoising and the other two learning algorithms (from Table 3), next to BM3D [13] and LGP-PCA [50]. In this evaluation, the non-learning algorithms perform considerably better at the cost of a significantly longer execution time. This finding suggests a limitation of learning-based denoising methods, which is that they depend on training sets that represent well the noise seen in the evaluation data set. A few visual results of our approach are presented in Fig. 2 and Fig. 3.

4.3 ECG Denoising Evaluation

Task Generation. We use the Physionet ECG-ID database [15], which includes 310 ECG records from 90 subjects. Each record contains the raw noisy ECG signal and the filtered version that represents the clean signal. The ECG signal is often corrupted by power line interference, contact noise and motion artefacts. We choose 217 ECG records (70% of the sequences) for training, 62 (20%) for validation and 31 (10%) for testing. The total number of samples is around 385k. We use the training set for generating the synthetic tasks, where in each task, we add Gaussian noise to the clean training samples, as in $\mathbf{x}=h_{\tau=\mathcal{N}(\mu,\sigma)}(\mathbf{y})$, where $h_{\tau}(.)$ is defined by $\mathbf{x}(j)=\mathbf{y}(j)+\eta(j)$ for $j\in\{1,...,D\}$, $\tau$ is represented by a Gaussian noise $\eta(j)\sim\mathcal{N}(\mu,\sigma)$, with $\mu\sim U(-0.1,0.1)$ and standard deviation $\sigma\sim U(0,0.3)$ for $U(a,b)$ denoting uniform distribution in $[a,b]$ and $\mathcal{N}(.)$ representing the Gaussian distribution with mean $\mu$ and variance $\sigma$. We generate the training samples per task, where each sample is represented by a window of $D=30$ consecutive signal measurements with stride one. To perform fine-tuning for transfer learning or meta-learning, we randomly draw $k$ samples from the test set and use the rest for evaluation.

Model. The network is a denoising autoencoder [11], consisting of 3 encode and 3 decoder linear layers with 150 units, and a latent representation layer with 25 units [11]. The activation function is ReLU, which is applied in all layers, except the last one that uses linear activation. The input and output for the model consists of a 30-element vector. During meta-training, we use the AdaDelta [47] optimization for the inner loop (i.e. , per task optimization). AdaDelta showed a more stable performance than stochastic gradient descent. The learning rate is 1.5 and the number of epochs is 10. For the outer loop, we iterate for 800 epochs with step-size $\epsilon=0.01$. The number of samples for few-shot evaluation is $k=10$.

Results. The results are summarized in Table 5. We evaluate for 50 and 100 generated tasks using the Gaussian noise model. We experimentally found that a much larger number of tasks is necessary for this experiment, compared to image-based denoising. The same experiments have been conducted for supervised- and transfer-learning. We observe that increasing the number of tasks improves the performance almost by 1dB. Generating more tasks for 10-shot learning did not have a large impact on the overall performance of our model. Comparing to the other two approaches, we see that only transfer learning achieves considerable denoising and is closer to our results. We also compare our approach with a non-learning algorithm, as with the earlier experiments, where we evaluate a Wavelets approach [38], resulting in -5.90dB, which is much worse than our approach. Here, learning-based denoising produces more accurate results than non-learning approaches.

4.4 Statistical Significance

To examine the statistical significance of our approach, we conduct one-tailed paired t-test for all experiments, comparing our approach to supervised and transfer learning. For all experiments, the test results show that under the significance level of $10^{-3}$, the performance of our approach is always significantly better.

4.5 K-shot Learning

We further study the performance of our approach under different k-shots of learning. The goal is to observe the performance drop when reducing the number fine-tuning samples $k$, which is a common experiment for few-shot classification. We selected as reference the CT-Scan data set with eight Poisson tasks, since this model achieve the best performance as it is shown in Table 1. We have trained this approach for 1-, 3-, 5-, 7- and 10-shot learning, and followed the same protocol for the transfer learning. Table 6 summarizes the results. It is clear that larger values for $k$ improves the results. Interestingly, we are able to denoise even with 3-shot learning. The same conclusion holds for transfer learning, where we chose the model with four Poisson tasks as reference. By comparing the two algorithms, we see that our method always reaches higher PSNR for both dose tube currents.

4.6 Meta-Learning with MAML

We have considered MAML [14] as a second meta-learning algorithm for the few-shot denoising problem. We have observed though that it is less stable during training and significantly slower in terms of convergence. MAML training has been at least four times slower than Reptile independently of the amount of generated tasks. Finally, it does not generalize as well as Reptile. In particular, we have again considered the CT-Scan data set where MAML performs at best when using eight Gaussian tasks. The PSNR is 35.08 for $5\%$ tube current and 37.68 for for $10\%$ tube current. These results are far behind Reptile results in Table 1.

5 Conclusion

We have introduced few-shot meta-learning for denoising. Our approach applies gradient-based meta-learning to signal denoising based on synthetically generated tasks. In our evaluation, we explored two image- and a sequence-based problems, where we demonstrate that: 1) meta-learning is effective given sufficient amount of synthetic tasks; 2) meta-learning works better than supervised- and transfer-learning under the assumptions listed in Sec. 1; 3) meta-learning can produce competitive denoising results, compared with a fully supervised training with real data; 4) it is more accurate than non-learning approaches, when the synthetic noise represents well the (latent) noise model used to form the real noisy image; and 5) learning-based denoising inference is significantly faster than non-learning models. Our study provides strong indication that meta-learning has the potential to become the standard learning algorithm for denoising.