Solar Image Restoration with the Cycle-GAN Based on Multi-Fractal Properties of Texture Features

TL;DR

This paper introduces a Cycle-GAN based method leveraging multi-fractal texture features to restore solar images, enhancing resolution without requiring paired datasets or additional hardware.

Contribution

The paper presents a novel, purely data-driven solar image restoration technique using Cycle-GANs and multi-fractal texture assumptions, applicable without paired training data.

Findings

Improves spatial resolution of solar images

Works with simulated and real data

Does not require paired datasets or extra instruments

Abstract

Texture is one of the most obvious characteristics in solar images and it is normally described by texture features. Because textures from solar images of the same wavelength are similar, we assume texture features of solar images are multi-fractals. Based on this assumption, we propose a pure data-based image restoration method: with several high resolution solar images as references, we use the Cycle-Consistent Adversarial Network to restore burred images of the same steady physical process, in the same wavelength obtained by the same telescope. We test our method with simulated and real observation data and find that our method can improve the spatial resolution of solar images, without loss of any frames. Because our method does not need paired training set or additional instruments, it can be used as a post-processing method for solar images obtained by either seeing limited telescopes or telescopes with ground layer adaptive optic system.

Tables

Table 1. Table 1: Structure of the Generator. Conv2d is standard convolutional layer. IN is a instance normalization layer which will normalize each image through I = ( I − I ¯ ) / V ​ a ​ r ​ ( I ) 𝐼 𝐼 ¯ 𝐼 𝑉 𝑎 𝑟 𝐼 I=(I-\overline{I})/Var(I) , where I ¯ ¯ 𝐼 \overline{I} and V ​ a ​ r ​ ( I ) 𝑉 𝑎 𝑟 𝐼 Var(I) are mean value and variance of I. ReLu is the activation function. ResidualBlock is special structure of neural network and the input will feed into its output and the first layer as defined in Table  3 . ConvT2d is a transposed convolutional layer which will up sample the input data through learnable weights.

Type	kernel size $\backslash$ stride	output
Conv2d	$7\times 7\backslash 1$	$256\times 256\times 64$
IN		$256\times 256\times 64$
ReLu		$256\times 256\times 64$
Conv2d	$3\times 3\backslash 2$	$128\times 128\times 128$
IN		$128\times 128\times 128$
ReLu		$128\times 128\times 128$
Conv2d	$3\times 3\backslash 2$	$64\times 64\times 256$
ResidualBlock		$64\times 64\times 256$
ResidualBlock		$64\times 64\times 256$
ResidualBlock		$64\times 64\times 256$
ResidualBlock		$64\times 64\times 256$
ResidualBlock		$64\times 64\times 256$
ConvT2d	$3\times 3\backslash 2$	$128\times 128\times 128$
IN		$128\times 128\times 128$
ReLu		$128\times 128\times 128$
ConvT2d	$3\times 3\backslash 2$	$256\times 256\times 64$
IN		$256\times 256\times 64$
ReLu		$256\times 256\times 64$
Conv2d	$7\times 7\backslash 1$	$256\times 256\times 1$

Table 2. Table 2: Structure of the Discriminator. LeakyReLu is a leakyReLu activation function, which has small slope for negative values, and in this paper we use the negative slope of 0.2. Sigmoid is the output layer of the discriminator and is used for classification of the input signals. All other layers have the same definitions as those in Table   1 .

Type	kernel size $\backslash$ stride	output	negative slope
Conv2d	$4\times 4\backslash 2$	$128\times 128\times 64$
LeakyReLU		$256\times 256\times 64$	0.2
Conv2d	$4\times 4\backslash 2$	$64\times 64\times 128$
IN		$64\times 64\times 128$
LeakyReLU		$64\times 64\times 128$	0.2
Conv2d	$4\times 4\backslash 1$	$64\times 64\times 256$
IN		$64\times 64\times 256$
LeakyReLU		$64\times 64\times 256$	0.2
Conv2d	$4\times 4\backslash 1$	$64\times 64\times 1$
Sigmoid

Table 3. Table 3: Structure of the ResidualBlock. The residualblock was firstly introduced by He et al. ( 2015 ) . All the layers have the same definitions as those in Table   1 . The input will feed into the first layer and the output simultaneously ( R ​ e ​ s ​ i ​ d ​ u ​ a ​ l ​ B ​ l ​ o ​ c ​ k ​ O ​ U ​ T = O ​ U ​ T + I ​ N ​ P ​ U ​ T 𝑅 𝑒 𝑠 𝑖 𝑑 𝑢 𝑎 𝑙 𝐵 𝑙 𝑜 𝑐 𝑘 𝑂 𝑈 𝑇 𝑂 𝑈 𝑇 𝐼 𝑁 𝑃 𝑈 𝑇 ResidualBlockOUT=OUT+INPUT , where O ​ U ​ T 𝑂 𝑈 𝑇 OUT is the output of the last I ​ N 𝐼 𝑁 IN layer, I ​ N ​ P ​ U ​ T 𝐼 𝑁 𝑃 𝑈 𝑇 INPUT is the input of the ResidualBlock and R ​ e ​ s ​ i ​ d ​ u ​ a ​ l ​ B ​ l ​ o ​ c ​ k ​ O ​ U ​ T 𝑅 𝑒 𝑠 𝑖 𝑑 𝑢 𝑎 𝑙 𝐵 𝑙 𝑜 𝑐 𝑘 𝑂 𝑈 𝑇 ResidualBlockOUT is the output of this ResidualBlock).

Type	kernel size $\backslash$ stride
Conv2d	$3\times 3\backslash 1$
IN
ReLu
Conv2d	$3\times 3\backslash 1$
IN

Equations

$$Img(x,y)=Obj(x,y)\ast PSF(x,y)+Noise(x,y),$$

$${\mathcal{L}}_{GAN}(Restore,D_O,I,O)={\mathbb{E}}_{O\sim p_{data}(O)}[log{D}_O(O)]+{\mathbb{E}}_{I\sim p_{data}(I)}[log(1-D_O(Restore(I)))],$$

$${\mathcal{L}}_{GAN}(PSF,D_I,I,O)={\mathbb{E}}_{I\sim p_{data}(I)}[log{D}_I(I)]+{\mathbb{E}}_{O\sim p_{data}(O)}[log(1-D_I(PSF(O)))],$$

$${\mathcal{L}}_{cyc}(Restore,PSF)={\mathbb{E}}_{O\sim p_{data}(O)}[\Vert Restore(PSF(O))-O{\Vert }_1]+{\mathbb{E}}_{I\sim p_{data}(I)}[\Vert PSF(Restore(I))-I{\Vert }_1],$$

$$O\to PSF(O)\to Restore(PSF(O))\approx O$$

$$I\to Restore(I)\to PSF(Restore(I))\approx I$$

$${\mathcal{L}}_{identity}(Restore,PSF)={\mathbb{E}}_{O\sim p_{data}(O)}[\Vert PSF(O)-O{\Vert }_1]+{\mathbb{E}}_{I\sim p_{data}(I)}[\Vert Restore(I)-I{\Vert }_1]$$

$${\mathcal{L}}_{TV}(Restore,PSF)={\mathbb{E}}_{I\sim p_{data}(I)}[\Vert {\nabla }_{h}Restore(I){\Vert }_2+\Vert {\nabla }_{w}Restore(I){\Vert }_2)]$$

$$+{\mathbb{E}}_{O\sim p_{data}(O)}[\Vert {\nabla }_{h}PSF(O){\Vert }_2+\Vert {\nabla }_{w}PSF(O){\Vert }_2)],$$

$$\mathcal{L}(Restore,PSF)={\mathcal{L}}_{GAN}(Restore,D_O,I,O)+{\mathcal{L}}_{GAN}(PSF,D_I,I,O)$$

$$+\lambda {\mathcal{L}}_{cyc}(Restore,PSF)+{\mathcal{L}}_{identity}(Restore,PSF)+{\mathcal{L}}_{TV}(Restore,PSF).$$

Full text

techniques: image processing – sun: general – atmospheric effects

1 Introduction

The imaging process of optical telescopes can be modeled by equation 1:

[TABLE]

where $Obj(x,y)$ and $Img(x,y)$ are the original and observed images, $\ast$ is the convolutional operator, $PSF(x,y)$ is the point spread function (PSF) of the whole optical system and $Noise(x,y)$ stands for the noise from the background and the detector. During real observations, many different effects will introduce variable $PSF(x,y)$ and $Noise(x,y)$. These effects will make the $Img(x,y)$ different from the $Obj(x,y)$ and strangle further scientific researches.

For ground based solar observations, because the exposure time is short (dozens of millisecond) and the field of view is big (comparing to the isoplanatic angle), the atmospheric turbulence, thermal and gravity deformations of the optical system will introduce $PSF(x,y)$ with highly spatial and temporal variations. Even with the help of adaptive optics and active optics systems, the residual error will still introduce a variable $PSF(x,y)$. The variable PSF of solar images is different from that of ordinary night time astronomy observations and it is often called short exposure PSF, because the exposure time is only dozens of milliseconds. The short exposure PSF can not be described by any contemporary analytical PSF models such as the Moffat or the Gaussian model and it is the main limitation for ground based solar observations.

Several different image restoration methods have been proposed to reduce the effects brought by the short exposure PSF and increase the spatial resolution of astronomical images, such as the blind deconvolution algorithm (Jefferies & Christou, 1993), the speckle reconstruction algorithm (Labeyrie, 1970; von der Luehe, 1993), the phase diversity algorithm (Paxman et al., 1992, 1996; Löfdahl & Scharmer, 1994) and the multi-object multi-frame blind deconvolution algorithm (van Noort et al., 2005). These methods have different hypothesis or prior value of the $PSF(x,y)$ or the $Img(x,y)$, including wavefront measurements or assuming the image is invariant between different frames, and have achieved remarkable performance.

Texture is fundamental characteristic of an image and it describes the grey scale spatial arrangement of images. Normally texture features are used to evaluate textures. In our recent paper (Huang et al., 2019), we have shown the multi-fractal properties of texture features in solar images of different wavelengths. Based on the results from that paper, we use a Cycle-Consistent Adversarial Network (CycleGAN) to restore solar images with the multi-fractal properties as regularized condition in this paper. Our method can restore arbitrary number of solar images obtained by the same telescope within a few days with only several high resolution images as references. We will discuss the multi-fractal property of texture features in Section 2 and introduce our method in Section 3. In Section 4, we will show the performance of our method with real and simulated observation data. We will make our conclusions and anticipate our future work in Section 5.

2 The Multi-Fractal Property of Texture Features In Solar Images

Textures are mostly related to spatially repetitive structures which are formed by several repeating elements (Castelli & Bergman, 2002). Similar to other natural images, solar images also have a lot of textures, such as the granulation in TiO and the filament in H-alpha, as shown in Figure 1. The texture feature is a description of the spatial arrangement of the gray scale in an image and it is usually used to describe the regularity or coarseness of an image. Manually designed texture features have been successfully used to describe the arrangement of texture constituents in a quantitative way (Tamura et al., 1978; Manjunath & Ma, 1996; Li-wen et al., 2019). However, textures in solar images are not arranged in a regular or periodic way, which makes it hard to design adequate texture features by hand. According to our experience, textures in solar images have the following properties:

1. In the same wavelength, textures from different solar images are similar.

2. For the same solar image, the shape variation of textures in the same spatial scale satisfies the same statistical law. For example, filaments are bending with a smooth curve instead of a polyline.

These properties indicate us that although texture features are not organized in a regular way for solar images, the relative weights of different texture features are stable, which means if we measure the relative weights of texture features in a statistical way, the probability distribution in the same wavelength should be the same. We can use multi-fractal properties to describe texture features in solar images (Jia et al., 2014; Peng et al., 2017). The multi-fractal property of texture features means the spatial distribution of textures in solar images (coded by texture features) satisfies the same continuous power spectrum and for different scales, the exponents of the spectrum are different.

Because textures are just appearances of the physical process behind and the physical process that generates the textures does not change, the multi-fractal property of texture features are valid for all solar images obtained in the same wavelength, i.e., texture features of solar images in the same wavelength satisfy the same power spectrum. Recent works suggest that neural network have very good property in representing complex functions.

In this paper, we further try to take advantage of this property and propose to use neural networks to evaluate multi-fractal properties from solar images. The multi-fractal properties are encoded in the neural network and can be visualized by feature maps of each layer. In our recent papers (Huang et al., 2019), we show the multi-fractal properties of G-band images. In this paper, the multi-fractal properties are used as regularized condition of the CycleGAN, just as other regularization conditions used in traditional deconvolution algorithm, such as: the total-variation condition in deconvolution algorithms. So we do not try to extract the multi-fractal property of texture features directly, instead we use many high resolution images to represent it in a statistical way.

In real observations, we can obtain a lot of high resolution solar images through speckle reconstruction, phase diversity, multi-object multi-frame deconvolution or observations with diffraction-limited adaptive optic systems, such as the single-conjugate or multi-conjugate adaptive optics systems (Rao et al., 2018). The spatial resolution of these images are around the diffraction limit of the telescope and can reveal the highest spatial frequency that the observation data have. As the texture features from high resolution solar images are just realization of the theoretical multi-fractal property, we can use multi-fractal properties of texture features from high resolution images as a restriction condition for image post-processing methods, as we will discuss in the next section.

3 The CycleGAN for Image Restoration with the Multi-fractal Property

3.1 Introduction of the CycleGAN

The deep convolutional neural network (DCNN) is a type of deep learning framework and is widely used in image restoration (Xu et al., 2014; Wieschollek et al., 2016; Zhang et al., 2017). For solar images, several different DCNNs have been proposed for image restoration or enhancing (Díaz Baso & Asensio Ramos, 2018; Asensio Ramos et al., 2018). These methods are based on supervised learning, which requires pairs of high resolution images and blurred images as training set to model the degradation process, i.e. the $PSF(x,y)$. However, for real observations, obtaining the training set is hard. Besides, the number and diversity of images in the training set are usually not large enough to represent different image degradation processes. A trained DCNN will output unacceptable results, when blurred images have a different $PSF(x,y)$ than that of the training set. The requirement of many paired images in the training set limits wider application of these image restoration methods.

The generative adversarial network (GAN) is a generative model (Goodfellow et al., 2014), which contains two DNNs: a generator G and a discriminator D. Given real data set R, G tries to create fake data that looks like the genuine data from R, while D tries to discriminate the fake data and the genuine data. The GAN can be trained with back-propagation algorithm effectively, when there are only limited training data. For image restoration of galaxies, the GAN has been successfully trained with only 4105 pairs of training images (Schawinski et al., 2017). However, as we discussed above, the GAN also models the degradation process from these training images. For solar observations, because the atmospheric turbulence induced short exposure PSF is much more complex than the long exposure PSF, the number of training images required in the GAN will be greatly increased and the performance of GAN will be strongly influenced by limited training data.

Limited training data is also a problem in other image related tasks. Zhu et al. (2017) propose the Cycle-Consistent Adversarial Network (CycleGAN) to solve this problem. The CycleGAN is an unsupervised learning algorithm, which contains a pair of Generative Adversarial Networks (GAN). Given two sets of images, one GAN learns the image mapping and the other GAN learns the inverse mapping. Under the constrain condition that the mapped image after inverse mapping should be similar to itself and vise versa (cycle consistency loss), the CycleGAN can restore blurred images directly with high resolution images as references. It should be noted that, when used for image restoration, although the supervised DCNN, the GAN and the CycleGAN all try to learn the restoration function, the CycleGAN has very different hypothesis than that of other methods. The CycleGAN is constrained by the probability distribution of data (multi-fractal property of texture features in this paper), while other methods are constrained by the blur properties contained in pairs of training images. The detail structure of the CycleGAN used in this paper will be discussed in the next subsection.

3.2 Structure of the CycleGAN for Solar Image Restoration

The structure of the CycleGAN used in this paper is shown in Figure 2111The complete code used in this paper is written in Python programming language (Python Software Foundation) with the package Pytorch and can be downloaded from http://aojp.lamost.org.. Because short exposure PSFs for solar observation have very complex structure, we use a very deep CNN as generator as shown in Figure 3 and Appendix A. This generator is inspired by Zhu et al. (2017). However, since the CycleGAN will learn the short exposure PSF which has complex structures, we modify it and use smaller convolution kernels here to increase its representation ability. For the discriminator, we use an ordinary CNN that is normally used in image style transfer (Isola et al., 2017; Yi et al., 2017; Zhu et al., 2017).

As shown in Figure 2, $I$ and $O$ are blurred images and high resolution images. One GAN in the CycleGAN tries to learn the restoration function and it has a generator $Restore$ and a discriminator $D_{O}$, which can be written as: $Restore:I\rightarrow O$. The other GAN tries to learn the PSF and it has a generator $PSF$ and a discriminator $D_{I}$, which can be written as $PSF:O\rightarrow I$. We apply ordinary adversarial loss to both of these two GANs as:

[TABLE]

where $\mathbb{E}_{I\sim p_{data}(I)}$ stands for the expectation in the probability space of $data(I)$ and vise versa. These two adversarial loss can make the distribution generated by the above two generators close to the real distribution of $I$ or $O$. However because the generator is very complex, with the adversarial loss along, the generator would learn other mappings that also match the distribution of $I$ or $O$. To further restrict the space of possible mapping functions, we use the cycle consistency loss $\mathcal{L}_{cyc}(Restore,PSF)$ to constrain the solution space:

[TABLE]

where $\|\|_{1}$ stands for the 1-norm. The cycle consistency loss guarantees that, for each image, the CycleGAN will bring it back to the original value:

[TABLE]

Because the image restoration algorithm should not flip the gray value between different pixels, we use the identity loss to constrain the contrast of the image, preventing from rapid change of gray scale between different pixels.

[TABLE]

At last, we calculate the total variation of $O$ and $I$ and use them as the total variation loss to improve the image quality and reduce the artifacts generated by the CycleGAN,

[TABLE]

where $\|\|_{2}$ stands for the 2-norm, $\nabla_{h}$ and $\nabla_{w}$ are horizontal and vertical gradient of these images. We calculate the weighted summation of the above loss functions and use the function defined below to train the CycleGAN, where $\lambda$ is the relative weight of the cycle consistent loss.

[TABLE]

3.3 Other Restriction Conditions for the CycleGAN in Image Restoration

Because the CycleGAN tries to model the degradation process according to the statistical probability distribution of texture features in images, the restriction of this model should lie both in the image and the degradation process. First of all, as the CycleGAN does not have any paired training images as supervisions and it is supervised in the form of $O$ and $I$, where $O$ and $I$ are high resolution images and blurred images respectively, they should satisfy the following properties. For $O$:

1. High resolution images need to have the same or smaller pixel scale than that of the blurred images. Then we will down sample all the high resolution images to images of the same pixel scale as that of blurred images in data set $O$.

2. The apparent structures should be removed from high resolution images, albeit keeping it small enough. Because the apparent structures, such as sunspots have different textures and will change the multi-fractal properties.

3. We need enough images with textures to represent the multi-fractals in a statistical way. According to our experience, at least 100 frames of reference images with $256\times 256$ pixels are required, however it is much smaller than ordinary DNN based image restoration method.

As the CycleGAN is to model the restoration function and the PSF, which have very strong spatial and temporal variations, we need to set several restrictions in $I$ to make the CycleGAN robust in real applications:

1. When restoring a single frame of solar image, it would be better to divide it into smaller images with size of around dozens of arcsec.

2. For several continuous frame of solar images, it would be better to cut the interested areas (with size of around dozens of arcsec) from these images and directly restore these temporal continuous small images with the CycleGAN.

3. To reduce image processing time, it would be better to use texture-rich small images that are cut from blurred images to train the CycleGAN. After training, the $Restore$ can be directly used to restore all the blurred images.

Last but not least, reference images and blurred images need to have the same multi-fractal property, so the images taken in the same band of wavelength within a few days is strict enough for our method. We tested our method with real observation data and found that the neural network used in this paper is complex enough for image restoration task, because we did not find any images that the CycleGAN failed to restore. With the above restrictions, the CycleGAN can be effectively trained through several thousand iterations. In the next Section, we will show implementations of our algorithm.

4 Implementations of the CycleGAN for Image Restoration

4.1 Performance Evaluation with Simulated Observation Data

Over-fitting is a major problem that would limit the performance of our algorithm. Because the CycleGAN is a generative model, overfitting will make $Restore$ remember the structure of high resolution images and generate fake structures during image restorations. To test our algorithm, we use the CycleGAN to restore simulated blurred images. There are two sets of images used in this paper: observations carried out in H-alpha wavelength between 02 April 2018 and 03 April 2018 and observations carried out in TiO band on 17 November 2014. These two sets of data are both observed by the NVST (Liu et al., 2014): the H-alpha data was observed in 656.28 nm and bandpass of 0.025 nm with pixelscale of 0.136 arcsec and the TiO data was observed in 705.8 nm and bandpass of 1 nm with with pixelscale of 0.052 arcsec 222For more details, please refer to: http://fso.ynao.ac.cn/cn/introduction.aspx?id=8. These images are restored by speckle reconstruction methods (Li et al., 2015).

From 5 high resolution H-alpha wavelength images, we crop 500 images with size of $256\times 256$ pixels as references and crop another 5 images with size of $256\times 256$ as test images. From 5 high resolution TiO solar images, we crop 500 images with size of $256\times 256$ pixels as references and crop another 5 images with size of $256\times 256$ as test images. According to real observation conditions, we use Monte Carlo method to simulate several high fidelity atmospheric turbulence phase screens with $D/r_{0}$ of 10 for H-alpha wavelength and 4 for TiO data (Jia et al., 2015a, b), where $D$ stands for the diameter of telescope and $r_{0}$ stands for the coherent length of atmospheric turbulence. We calculate simulated short exposure PSFs with these phase screens through far field propagation (Basden et al., 2018). At last, we convolve test images and temporal-continuous PSFs to generate 100 simulated blurred images as simulated blurred H-alpha and TiO data.

These simulated blurred images and high resolution images in the same wavelength are used to train the CycleGAN with 2000 iterations. Because all the simulated blurred images have the original high resolution images, we can compare them to test our algorithm. The results are shown in the figure 4. We carefully check these images and find that the resolution of the restored images has been improved and there are no observable difference between the restored images and the the original images. We have also calculated the median filter-gradient similarity – MFGS (Deng et al., 2017) of the simulated blurred images, the original high resolution images and the restored images to further test our method. For H-alpha data, the MFGS is increased from $0.75\pm 0.05$ to $0.81\pm 0.04$, while the mean MFGS of the original images is $0.81\pm 0.04$. For TiO data, the mean MFGS is increased from $0.81\pm 0.03$ to $0.82\pm 0.02$, while the mean MFGS of the original images is $0.82\pm 0.03$. According to these results, we can find that the image quality has been increased by our method.

4.2 Performance Evaluation with Real Observation Data

In this part, we use real observation data from NVST to evaluate performance of the CycleGAN and also show our recommendations of how to use the CycleGAN in real applications. According to the size of isoplanatic angle, an image should be cut into small images for restoration. However considering the processing speed, we use images with size of $256\times 256$ pixels (equivalent to around $0.5\times 0.5$ arcmin) for restoration. It is much larger than the isopanatic angle and the restoration results will drop down slightly, however it is a trade-off we have to make. In real applications, there are two scenes of image restoration: restoration of several continuous solar images in a small interested region or restoration of a single frame image with relative large size.

For the first scene, we use both H-alpha images observed between 02 April and 03 April 2018 and TiO images observed on 19 August 2017 with small size to test our algorithm. For each wavelength, 500 frames of images with size $256\times 256$ pixels are extracted from 5 frames of blurred image with size of $1024\times 1024$ pixels as $I$ and 500 frames of images with size $256\times 256$ pixels are extracted from one speckle reconstructed image with size of $1024\times 1024$ pixels as $O$. Because the CycleGAN is deep and complex, the maximal number and size of the reference image and the blurred image are actually limited by the computer333In this paper, we use a computer with two Nvidia GTX 1080 graphics cards, 128 GB memory and two Xeon E5 2650 processors. It will cost 4498 seconds to train the CycleGAN with 6000 iterations.. After training, we can use the $Restore$ to directly restore two interested regions with size of $256\times 256$ pixels in the blurred images. It costs around 1 minutes to process 333 frames of these blurred images. Two frames of restored images are shown in Figure  5 and interested readers can also find animated versions of these figures in the online version of this paper. They are 100 frames of blurred solar images from H-alpha and 79 frames from TiO before and after restoration, alone with their MFGS values. From these figures, we can find that the resolution of the restored images have been improved. The mean MFGS is increased from $0.65\pm 0.01$ to $0.89\pm 0.004$ for TiO data and from $0.75\pm 0.03$ to $0.89\pm 0.007$ for H-alpha data.

For the second scene, we test the performance of our algorithm in big images. A blurred image from the NVST is shown in the panel of Figure 6. The validation part is the image in the white box and the rest of this image is used as the training part. We extract 500 frames of images with size of $256\times 256$ from the training part as $I$ and use 500 frames of high resolution images used above as $O$ in the CycleGAN. After 6000 iterations, the images in $I$ are restored and we use $Restore$ to restore images in the white box. The results are shown in the right panel of Figure 6. We can find that the spatial resolution of the observed images has been improved and the difference of image quality between the validation part and the training part is very small. The MFGS is increased from 0.78 to around 0.89 for this image.

5 Conclusions

As more and more high resolution solar images are obtained, we propose a pure data based image restoration method to make better use of these data. We assume texture features of solar images in the same wavelength are multi-fractals and use a deep neural network – CycleGAN to restore blurred images, with several high resolution images from the same telescope as references. Our method does not need paired images as the training set. Instead, with only several high resolution images observed in the same wavelength, our method can give promising restoration results for every frame of real observation data without any additional instruments. We use simulated blurred images to test our algorithm. We compare the reconstructed images with real images and find that the MFGS has been increased with our method. Besides, we further use our algorithm to restore real observation images. Although the image quality increased by our method is slightly smaller than the speckle reconstructed images, our method can restore every frame of blurred images, while the speckle reconstructed method use a lot of blurred images (dozens or hundreds) and only obtains one frame of restored image. Our method is suitable for future observation data obtained by seeing-limited telescope or telescope with ground layer adaptive optic systems. Because our method does not have any prior assumption of the degradation process, it can also be used to restore images of other astronomical objects with features, such as galaxies, nebulae or supernova remnants.

The authors are grateful to the anonymous referee for his or her comments and suggestions, which have greatly improved the quality of this manuscript. The authors would like to thank Dr. Yongyuan Xiang and Professor Kaifan Ji from Yunnan Astronomical Observatory for their suggestions and providing solar observation data from the NVST. P.J. would like to thank Professor Hui Liu and Professor Zhong Liu from Yunnan Observatory, Professor Yong Zhang from Nanjing Institute of Astronomical Optics and Technology, Dr. Alastair Basden, Dr. Tim Morris, Dr. James Osborn and Dr. Matthew Townson from Durham University, Dr. Yang Guo and Dr. Qi Hao from Nanjing University and Dr. Qinmin Zhang from Purple Mountain Observatory who provide very helpful suggestions for this paper. This work is supported by National Natural Science Foundation of China (NSFC)(11503018), the Joint Research Fund in Astronomy (U1631133) under cooperative agreement between the NSFC and Chinese Academy of Sciences (CAS). PJ is supported by the China Scholarship Council to study at the University of Durham. The data used in this paper were obtained by the New Vacuum Solar Telescope in Fuxian Solar Observatory of Yunnan Astronomical Observatory, CAS.

Appendix A Detail Structure of the CycleGAN

\startlongtable

\startlongtable

\startlongtable