Convolutional Neural Network-Based Block Up-sampling for Intra Frame Coding

TL;DR

This paper introduces a CNN-based block up-sampling method for intra frame coding that improves compression efficiency by leveraging learned up-sampling and frame-level optimization, achieving significant bitrate reductions.

Contribution

The paper presents a novel CNN architecture for block up-sampling in intra coding, incorporating multi-scale fusion and residue learning, and integrates it into HEVC for improved compression.

Findings

Achieves 5.5% BD-rate reduction on standard sequences.

Achieves 9.0% BD-rate reduction on UHD sequences.

Significantly improves low bit rate compression efficiency.

Abstract

Inspired by the recent advances of image super-resolution using convolutional neural network (CNN), we propose a CNN-based block up-sampling scheme for intra frame coding. A block can be down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution. Different from previous studies on down/up-sampling-based coding, the up-sampling methods in our scheme have been designed by training CNN instead of hand-crafted. We explore a new CNN structure for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning, making the network both compact and efficient. We also design different networks for the up-sampling of luma and chroma components, respectively, where the chroma up-sampling CNN utilizes the luma information to boost its performance. In addition, we design a two-stage up-sampling process, the first stage being within the block-by-block coding loop, and the second stage being performed on the entire frame, so as to refine block boundaries. We also empirically study how to set the coding parameters of down-sampled blocks for pursuing the frame-level rate-distortion optimization. Our proposed scheme is implemented into the High Efficiency Video Coding (HEVC) reference software, and a comprehensive set of experiments have been performed to evaluate our methods. Experimental results show that our scheme achieves significant bits saving compared with HEVC anchor especially at low bit rates, leading to on average 5.5% BD-rate reduction on common test sequences and on average 9.0% BD-rate reduction on ultra high definition (UHD) test sequences.

Figures

Tables

Table 1. TABLE I: List of Abbreviations

CNN	Convolutional Neural Network
CTU	Coding Tree Unit
DCTIF	Discrete Cosine Transform based Interpolation Filter
HEVC	High Efficiency Video Coding
HR	High-Resolution
LR	Low-Resolution
MSE	Mean-Squared-Error
PSNR	Peak Signal-to-Noise Ratio
QP	Quantization Parameter
R-D	Rate-Distortion
ReLU	Rectified Linear Unit [17]
SR	Super-Resolution
SRCNN	Super-Resolution Convolutional Neural Network [12]
SSIM	Structural Similarity [18]
UCID	Uncompressed Colour Image Database [19]
UHD	Ultra High Definition
VDSR	Very Deep network for Super Resolution [15]

Table 2. TABLE II: Characteristics of the UHD Test Sequences

Source	Resolution	Name	Frame Rate
SJTU UHD	3840$\times$2160	Fountains	30 fps
		Runners
		Rushhour
		TrafficFlow
		CampfireParty

Table 3. TABLE III: BD-Rate Results of All Test Sequences

Class	Sequence	BD-Rate (Anchored on HEVC)				BD-Rate (Anchored on HEVC+DCTIF)
		Y	U	V	Y SSIM	Y	U	V	Y SSIM
Class A	Traffic	–10.1%	–3.5%	6.0%	–12.9%	–8.0%	–13.2%	–2.6%	–7.9%
	PeopleOnStreet	–9.7%	–14.8%	–14.5%	–12.9%	–8.5%	–20.4%	–18.5%	–9.7%
	Nebuta	–2.0%	–22.0%	3.1%	–4.4%	–1.7%	–22.5%	1.6%	–3.6%
	SteamLocomotive	–1.7%	–27.7%	–25.4%	–6.1%	–1.2%	–34.2%	-25.6%	–2.8%
Class B	Kimono	–7.7%	–5.5%	18.8%	–9.6%	–3.4%	–25.9%	–4.3%	–3.4%
	ParkScene	–7.1%	–14.4%	–2.3%	–11.3%	–5.0%	–25.2%	–14.6%	–6.6%
	Cactus	–6.6%	–2.5%	8.3%	–10.0%	–5.0%	–6.5%	0.9%	–6.7%
	BQTerrace	–3.7%	–7.6%	–9.1%	–9.6%	–3.1%	–8.2%	–7.1%	–6.5%
	BasketballDrive	–6.1%	–1.2%	3.2%	–10.8%	–3.4%	–5.8%	–2.5%	–3.8%
Class C	BasketballDrill	–4.9%	4.5%	8.1%	–7.9%	–4.0%	4.9%	2.1%	–6.6%
	BQMall	–2.9%	–7.2%	–7.2%	–6.2%	–2.3%	–10.6%	–9.1%	–5.3%
	PartyScene	–1.0%	–5.1%	–1.6%	–4.0%	–1.0%	–5.5%	–3.2%	–3.6%
	RaceHorsesC	–6.7%	4.6%	7.5%	–10.7%	–6.0%	1.9%	3.9%	–8.6%
Class D	BasketballPass	–2.0%	–3.7%	9.2%	–4.3%	–2.3%	–7.5%	12.3%	–4.4%
	BQSquare	–0.9%	–0.6%	–21.1%	–1.4%	–0.5%	1.7%	-16.7%	–1.2%
	BlowingBubbles	–3.2%	3.1%	–8.0%	–5.3%	–1.7%	0.5%	-9.6%	–3.8%
	RaceHorses	–9.9%	7.5%	6.4%	–12.6%	–9.6%	5.0%	6.6%	–11.1%
Class E	FourPeople	–7.2%	–10.5%	–11.0%	–11.0%	–7.2%	–14.7%	–14.5%	–9.5%
	Johnny	–9.0%	–3.2%	–3.2%	–11.1%	–7.1%	–6.0%	–8.3%	–5.6%
	KristenAndSara	–6.8%	–11.2%	–11.1%	–13.0%	–5.3%	–8.4%	–10.6%	–8.2%
Class UHD	Fountains	–4.0%	–12.9%	–11.2%	–7.4%	–2.0%	–16.1%	–9.2%	–2.0%
	Runners	–11.2%	22.8%	–0.1%	–12.4%	–7.0%	0.9%	–13.7%	–6.0%
	Rushhour	–8.5%	4.4%	1.8%	–10.3%	–3.2%	–9.2%	–9.5%	–3.0%
	TrafficFlow	–12.7%	–11.7%	–5.8%	–12.7%	–6.9%	–17.3%	–11.9%	–5.6%
	CampfireParty	–8.4%	–10.8%	–0.8%	–9.5%	–6.5%	–10.8%	–5.0%	–6.4%
Average of Classes A–E		–5.5%	–6.0%	–2.2%	–8.8%	–4.3%	–10.0%	–6.0%	–5.9%
Average of Class UHD		–9.0%	–1.6%	–3.2%	–10.5%	–5.1%	–10.5%	–9.9%	–4.6%

Table 4. TABLE IV: BD-Rate Results at Different QPs (Anchored on HEVC)

Class	BD-Rate (QP 22–37)				BD-Rate (QP 27–42)				BD-Rate (QP 32–47)
	Y	U	V	Y SSIM	Y	U	V	Y SSIM	Y	U	V	Y SSIM
Class A	–0.4%	–3.3%	–2.6%	–1.8%	–2.4%	–9.4%	–5.5%	–5.3%	–5.9%	–17.0%	–7.7%	–9.1%
Class B	–1.4%	–3.3%	–0.7%	–2.8%	–3.5%	–5.0%	0.6%	–6.7%	–6.2%	–6.2%	3.8%	–10.3%
Class C	–0.2%	–0.5%	0.3%	–0.5%	–1.3%	–0.4%	1.6%	–3.0%	–3.9%	–0.8%	1.7%	–7.2%
Class D	–0.3%	0.3%	–0.9%	–1.0%	–1.4%	1.0%	–2.3%	–3.7%	–4.0%	1.6%	–3.4%	–6.4%
Class E	–1.0%	–3.3%	–4.9%	–2.7%	–3.8%	–6.0%	–8.2%	–7.6%	–7.7%	–8.3%	–8.4%	–11.7%
Avg. Classes A–E	–0.7%	–2.0%	–1.6%	–1.7%	–2.5%	–3.9%	–2.3%	–5.0%	–5.5%	–6.0%	–2.2%	–8.8%
Class UHD	–2.1%	–6.6%	–4.5%	–4.0%	–5.6%	–6.8%	–4.9%	–7.8%	–9.0%	–1.6%	–3.2%	–10.5%

Table 5. TABLE V: Symbols for CTUs that Choose Different Modes

Symbol	Remark
$C_{Total}$	All CTUs in a frame
$C_{Hitting}$	CTUs selecting the mode of low-resolution coding
$C_{Luma}$	Low-resolution coded CTUs, whose luma component
	is up-sampled using CNN
$C_{Cb}$	Low-resolution coded CTUs, whose Cb component
	is up-sampled using CNN
$C_{Cr}$	Low-resolution coded CTUs, whose Cr component
	is up-sampled using CNN

Table 6. TABLE VI: Hitting Ratio Results on Different Classes of Test Sequences

Class	$P_{Hitting}$	$P_{Luma}$	$P_{Cb}$	$P_{Cr}$
Class A	72.2%	70.3%	71.2%	55.0%
Class B	68.4%	75.0%	65.1%	49.4%
Class C	48.1%	92.0%	68.5%	73.5%
Class D	42.4%	81.9%	51.6%	70.7%
Class E	68.7%	72.8%	54.4%	58.5%
Class UHD	85.2%	68.4%	54.2%	64.1%

Table 7. TABLE VII: BD-Rate Results of Using Trained CNN Models for Different QPs

Class	Anchored on HEVC		Anchored on HEVC+DCTIF
	QP$+2$	QP$-2$	QP$+2$	QP$-2$
Class A	–6.6%	–4.5%	–5.2%	–3.8%
Class B	–6.9%	–5.5%	–4.0%	–3.3%
Class C	–5.5%	–2.9%	–4.9%	–2.9%
Class D	–6.0%	–2.1%	–5.0%	–2.0%
Class E	–8.1%	–6.5%	–6.7%	–5.6%
Avg. Classes A–E	–6.6%	–4.3%	–5.0%	–3.4%
Class UHD	–9.0%	–8.5%	–4.9%	–5.0%

Table 8. TABLE VIII: PSNR Results of Different Up-sampling Methods for Luma

Class	Sequence	DCTIF	VDSR	Ours
Class B	Kimono	39.82	39.86	39.79
	ParkScene	34.23	34.26	34.63
	Cactus	33.01	33.59	33.86
	BQTerrace	28.50	29.61	30.20
	BasketballDrive	34.36	35.64	35.71
Class C	BasketballDrill	31.46	32.77	33.17
	BQMall	28.31	29.36	29.46
	PartyScene	24.94	26.00	26.16
	RaceHorsesC	30.06	31.09	31.18
Class D	BasketballPass	30.34	31.33	31.56
	BQSquare	23.48	25.95	25.80
	BlowingBubbles	28.52	29.50	29.57
	RaceHorses	29.59	31.34	31.56
Class E	FourPeople	36.45	37.58	37.74
	Johnny	34.30	36.90	36.64
	KristenAndSara	34.12	35.43	35.69
Average		31.34	32.51	32.67

Table 9. TABLE IX: PSNR Results of Different Up-sampling Methods for Chroma

Class	Sequence	DCTIF		CNN without luma		CNN with luma
		Cb	Cr	Cb	Cr	Cb	Cr
Class B	Kimono	40.60	41.75	40.60	41.70	41.13	41.86
	ParkScene	38.20	39.74	38.20	39.68	38.83	39.80
	Cactus	37.68	39.32	37.77	39.51	37.83	39.58
	BQTerrace	38.19	40.61	38.22	40.73	38.50	40.85
	BasketballDrive	41.86	41.71	42.00	42.02	41.90	42.23
Class C	BasketballDrill	37.41	37.51	37.63	37.79	37.49	38.34
	BQMall	38.36	39.34	38.66	39.79	39.08	40.16
	PartyScene	34.89	35.45	35.03	35.65	35.34	35.91
	RaceHorsesC	36.58	37.66	36.80	38.22	37.00	38.40
Class D	BasketballPass	37.72	36.94	37.88	37.41	38.02	37.80
	BQSquare	39.22	39.53	39.33	39.84	39.69	40.16
	BlowingBubbles	35.26	37.32	35.36	37.52	35.38	37.73
	RaceHorses	35.53	35.38	35.82	35.99	36.39	36.30
Class E	FourPeople	43.86	44.65	44.12	44.95	43.96	45.01
	Johnny	43.14	44.22	43.33	44.51	43.22	44.55
	KristenAndSara	42.19	43.51	42.43	43.84	42.62	44.04
Average		38.79	39.67	38.95	39.95	39.15	40.17

Table 10. TABLE X: Percentage of CTUs that Benefit from the Second Stage, and Average MSE Results (Luma, QP 37)

Class	Percentage	Average MSE	Average MSE
		(First Stage)	(Second Stage)
Class A	72.8%	39.95	39.73
Class B	72.1%	25.01	24.80
Class C	85.8%	36.04	34.70
Class D	76.9%	40.38	39.76
Class E	76.7%	15.17	14.84
Class UHD	70.2%	22.57	22.42
Average	75.3%	29.96	29.49

Table 11. TABLE XI: BD-Rate Results of Using The Second Stage of Up-sampling

Class	Sequence	Y	U	V
Class A	Traffic	–0.9%	–3.3%	–2.7%
	PeopleOnStreet	–1.0%	–5.4%	–5.2%
	Nebuta	–0.3%	–4.0%	–1.0%
	SteamLocomotive	–0.4%	–4.4%	–7.3%
Class B	Kimono	–0.7%	–5.6%	–2.4%
	ParkScene	–0.7%	–4.4%	–2.6%
	Cactus	–0.7%	–2.4%	–2.7%
	BQTerrace	–0.6%	–1.9%	–2.5%
	BasketballDrive	–0.7%	–1.5%	–2.2%
Class C	BasketballDrill	–0.9%	–1.1%	–2.2%
	BQMall	–0.6%	–2.4%	–3.6%
	PartyScene	–0.2%	–1.0%	–1.0%
	RaceHorsesC	–1.0%	–3.7%	–4.3%
Class D	BasketballPass	–0.6%	0.2%	–1.2%
	BQSquare	–0.2%	–1.5%	–2.7%
	BlowingBubbles	–0.5%	–0.1%	–3.1%
	RaceHorses	–1.2%	–2.8%	–2.5%
Class E	FourPeople	–1.2%	–3.0%	–4.3%
	Johnny	–1.1%	–2.9%	–2.9%
	KristenAndSara	–0.9%	–3.3%	–3.5%
Class UHD	Fountains	–0.6%	–5.1%	–4.3%
	Runners	–0.8%	–2.2%	–3.4%
	Rushhour	–0.7%	–3.2%	–3.7%
	TrafficFlow	–1.1%	–4.6%	–4.3%
	CampfireParty	–0.7%	–1.8%	–3.0%
Average of Classes A–E		–0.7%	–2.7%	–3.0%
Average of Class UHD		–0.8%	–3.4%	–3.7%

Table 12. TABLE XII: Computational Time Comparison of Our Scheme and HEVC (HEVC is 100%)

Class	Encoding Time	Decoding Time
Class A	713%	29572%
Class B	749%	29868%
Class C	726%	16754%
Class D	751%	13721%
Class E	779%	21502%
Class UHD	766%	44766%
Average	747%	25021%

Equations

$${\mathbf{F}}_1(\mathbf{X})=\max (0,{\mathbf{W}}_1\ast \mathbf{X}+{\mathbf{B}}_1)$$

$${\mathbf{F}}_2(\mathbf{X})=\{\begin{array}{l}{\mathbf{F}}_{21}(\mathbf{X})=\max (0,{\mathbf{W}}_{21}\ast {\mathbf{F}}_1(\mathbf{X})+{\mathbf{B}}_{21})\\ {\mathbf{F}}_{22}(\mathbf{X})=\max (0,{\mathbf{W}}_{22}\ast {\mathbf{F}}_1(\mathbf{X})+{\mathbf{B}}_{22})\end{array}$$

$${\mathbf{F}}_3(\mathbf{X})=\max (0,{\mathbf{W}}_3\star {\mathbf{F}}_2(\mathbf{X})+{\mathbf{B}}_3)$$

$${\mathbf{F}}_4(\mathbf{X})=\{\begin{array}{l}{\mathbf{F}}_{41}(\mathbf{X})=\max (0,{\mathbf{W}}_{41}\ast {\mathbf{F}}_3(\mathbf{X})+{\mathbf{B}}_{41})\\ {\mathbf{F}}_{42}(\mathbf{X})=\max (0,{\mathbf{W}}_{42}\ast {\mathbf{F}}_3(\mathbf{X})+{\mathbf{B}}_{42})\end{array}$$

$${\mathbf{F}}_5(\mathbf{X})={\mathbf{W}}_5\ast {\mathbf{F}}_4(\mathbf{X})+{\mathbf{B}}_5$$

$${\mathbf{F}}_D(\mathbf{X})=\mathrm{DCTIF}(\mathbf{X})$$

$$\hat{\mathbf{Y}}(\mathbf{X})={\mathbf{F}}_5(\mathbf{X})+{\mathbf{F}}_D(\mathbf{X})$$

$$J=\sum _{i=1}^N{D}_i+\lambda \sum _{i=1}^N{R}_i$$

$$D_{full}=\alpha \times D_{low}+\beta$$

$$\begin{array}[]{rl}J&=D_{full}+\lambda R\\ &=\alpha D_{low}+\beta+\lambda R\\ &=\alpha(D_{low}+\frac{\lambda}{\alpha}R)+\beta\end{array}$$

$$J_{low}=D_{low}+{\lambda }_{low}R$$

$$\lambda =c\times 2^{\frac{QP-12}{3}}$$

$$Q{P}_{low}=QP-3\times {\log }_2\alpha$$

$${\lambda }_{low}=\lambda /4$$

$$Q{P}_{low}=QP-6$$

$$P_{Hitting}=\frac{\#C_{Hitting}}{\#C_{Total}},P_{Luma}=\frac{\#C_{Luma}}{\#C_{Hitting}},$$

$$P_{Cb}=\frac{\#C_{Cb}}{\#C_{Hitting}},P_{Cr}=\frac{\#C_{Cr}}{\#C_{Hitting}}$$

Full text

I Introduction

Video resolution keeps increasing in the past three decades along with the development of new video capture and display devices. The International Telecommunication Union has approved ultra high definition (UHD) television as standard, defining both 4K and 8K that lead to a new level of spatial resolution [1]. While UHD video applications, such as home theater, provide users with further enhanced experience and become increasingly popular, they raise even bigger challenges to the video storage and transmission systems. Accordingly, video coding methods have been more and more focused on high definition videos. The state-of-the-art video coding standard, High Efficiency Video Coding (HEVC), supports up to 8K resolution [2]. However, there is still necessity to further increase the compression efficiency for UHD videos, especially in scenarios where bandwidth is limited for video transmission.

Although the video capture and display devices enable higher resolution, such resolution may not be necessary to carry the important visual information in videos. Thus, it is a well known strategy to down-sample videos prior to encoding and to up-sample the decoded videos for reconstruction [3, 4, 5, 6, 7, 8, 9, 10]. Previous studies have shown that using low-resolution version during coding performs better than direct coding of full-resolution videos in low bit rate scenarios [3, 4]. Moreover, the critical resolution for reconstructing signal is known to be dependent on the spatial frequency of image/video, but different regions of natural images/videos have very different spatial frequency components. Then, several researches have been performed on spatially variant sampling rates for down/up-sampling-based image/video coding [9, 10].

The up-sampling process plays a key role in down/up-sampling-based video coding as it immediately decides the quality of the final reconstructed videos. Some researches then have been focused on devising more efficient up-sampling methods [5, 6, 7, 8]. Actually, image up-sampling is a classic research topic and has been extensively studied in the literature of image processing, where it is also termed super-resolution (SR). Typical image SR methods can be categorized into interpolation-based, reconstruction-based, and learning-based [11], and some of these methods were borrowed into video coding. For example, Shen et al. proposed a down/up-sampling-based video coding scheme, where the up-sampling method is a learning-based one that enhances the current low-resolution reconstructed image from the information of an external high-resolution image set [7]. Nonetheless, most of the previous studies on down/up-sampling-based video coding adopt fixed, hand-crafted interpolation filters rather than many advanced SR methods, partially due to the consideration of computational complexity.

Recently, learning-based image SR using convolutional neural network (CNN) has demonstrated remarkable progress. Dong et al. first proposed a CNN-based SR method known as SRCNN, which clearly outperforms the previous rivals in the single image SR task [12]. Since then, several CNN-based SR methods have been developed and shown to achieve further performance boost [13, 14, 15, 16].

Inspired by the abovementioned advances, in this paper, we propose a CNN-based block up-sampling scheme for intra frame coding. While it is conceptually natural to replace the hand-crafted interpolation filters with the trained CNN models for better quality, there are lots of issues to investigate when implementing a down/up-sampling-based coding scheme with CNN. First of all, we propose to perform block-level down/up-sampling instead of the entire frame, since different regions have variant local features and then need different sampling rates. Specifically in this work, compliant with the HEVC standard, the basic unit for down/up-sampling is the coding tree unit (CTU). Each CTU can be compressed at its full resolution, or down-sampled by a factor of 2, compressed at low resolution, and then up-sampled. Note that we adopt two different sampling rates here, i.e. $1\times 1$ and $1/2\times 1/2$, but extension to more sampling rates is straightforward. Furthermore, we make the following contributions to fulfill the proposed scheme as presented in this paper:

• •

We design a new CNN structure for block up-sampling in the proposed scheme. To achieve higher reconstrution quality and simpler network structure, we explore a five-layer CNN for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning. Moreover, we propose to use different networks for the up-sampling of luma and chroma components, respectively. The chroma up-sampling CNN reuses the luma information to improve its performance.

• •

We investigate how to integrate the up-sampling CNN into the intra frame coding scheme. Besides allowing the encoder to choose the sampling rate for each CTU, as mentioned above, we also propose to allow the encoder to select the up-sampling method for each down-sampled CTU with selection from either CNN or fixed interpolation filters. To handle the boundary condition in block-wise up-sampling, we propose a two-stage up-sampling process where the first stage is within the block-by-block coding loop, and the second stage is out of the loop to refine the CTU boundaries. We also perform empirical study on how to decide the coding parameters of the down-sampled blocks to pursue frame-level rate-distortion optimization.

• •

We perform extensive experiments to validate the proposed coding scheme as well as each proposed technique. The proposed scheme is implemented based on the HEVC reference software, and is shown to achieve significant bits saving compared with HEVC anchor especially at low bit rates. The proposed up-sampling CNN not only performs better, but also is simpler and computationally more efficient than the state-of-the-art image SR networks.

The remainder of this paper is organized as follows. In Section II, we discuss related work on down/up-sampling-based coding and CNN-based image SR. Section III presents the framework of the proposed block down/up-sampling-based coding scheme. The CNN structures for luma and chroma up-sampling are discussed in Section IV. Coding parameters setting and the two-stage up-sampling process are elaborated in Sections V and VI, respectively. Section VII presents the experimental results, followed by conclusions in Section VIII. Table I lists the abbreviations used in this paper.

II Related Work

In this section, we review the previous work that relates to our research in two categories. The first is down/up-sampling-based image and video coding, and the second is recently emerging CNN-based image SR.

II-A Down/Up-sampling-Based Coding

Down-sampling before encoding and up-sampling after decoding is a well known strategy for image and video coding in scenarios where the transmission bandwidth is limited. Many researches on this topic have been focused on developing efficient up-sampling methods. For example, the down/up-sampling-based video coding scheme in [5] adopts the video SR method proposed in [20], which is specifically designed for compressed videos by incorporating information like motion vectors into the SR task using a Bayesian framework. The scheme proposed by Shen et al. [7] adopts another up-sampling method, which belongs to learning-based SR methods, and imposes constraints of nearest neighbor searching region and rectifies the “unreal” pixels using inter-resolution and inter-frame correlations. Another scheme proposed by Barreto et al. [6] takes into account the locally variant image characteristics, and performs region-based SR to improve the reconstruction quality. The segmentation of regions is performed at the encoder side, and the segmentation map is signaled as side information to the decoder to guide the SR process.

The abovementioned researches all perform down-sampling of the entire image/frame. However, it is noted that a uniform down-sampling rate cannot suit for all the different image regions that have variant features. Locally adaptive down-sampling rates are then proposed. In [10], the appropriate down-sampling rates have been derived through theoretical analyses. In [9], compliant with block-based coding, down-sampling rates are made adaptive for each block and selected from $1\times 1$, $1/2\times 1$, $1\times 1/2$, and $1/2\times 1/2$.

Most of the previous studies on down/up-sampling-based coding adopt fixed, hand-crafted interpolation filters for both down- and up-sampling. In this work, we propose to utilize CNN models for up-sampling to enhance the reconstruction quality. In addition, we also adopt block-level adaptive down-sampling rates with selection from $1\times 1$ or $1/2\times 1/2$, as extension to more down-sampling rates is straightforward.

II-B CNN for Image SR

Super-resolution or resolution enhancement aims at reconstructing high-resolution (HR) signal from low-resolution (LR) observation, which has been studied extensively in the literature. Existing image SR methods can be categorized into interpolation-based, reconstruction-based, and learning-based ones [11]. Recently, inspired by the success of deep learning, researchers have put more attention to learning-based SR using CNN.

Dong et al. first proposed a CNN-based method for single image SR, termed SRCNN [12], which has a simple network structure but demonstrated excellent performance. Later on, several researches have been conducted to improve upon SRCNN at several aspects. First, deeper networks have been explored to enhance the performance, such as the very deep network known as VDSR [15]. Second, it is observed that the training of SRCNN converges too slowly, and residue learning [21], i.e. learning the difference between LR and HR images rather than directly learning the HR images, is adopted to accelerate the training and also improves the reconstruction quality [15]. Third, the input to SRCNN is an interpolated version of LR image, which is to be enhanced by the network. The fixed interpolation filters before the network may not be optimal. Thus, an end-to-end learning strategy, i.e. directly learning from the LR to the HR with embedding the resolution change into the network, is observed to perform better [14].

In this paper, we explore a new five-layer CNN structure for block up-sampling. Some key ingredients in the previously studied networks, such as residue learning and resolution change embedded in network, have been borrowed into our designed network. Our network structure is greatly simplified to reduce computational complexity, but still achieves satisfactory reconstruction quality, compared to the state-of-the-arts [15, 14].

III Framework of the Proposed Scheme

It is generally agreed that natural images/videos are equipped with locally variant features, and thus different regions may require different coding methods or parameters. For example, there are 35 intra prediction modes defined in HEVC intra coding, one of which can be selected for each block [2]. A down/up-sampling-based coding scheme provides more dimensions of freedom to explore so as to suit for different regions. While previous work has studied locally adaptive down-sampling rates [9, 10], other dimensions such as adaptive down-sampling filters, adaptive coding parameters (e.g. quantization parameters), adaptive up-sampling filters, can be taken into account as well. Therefore, we propose to perform block-level down/up-sampling to embrace the flexibility, and to enable both adaptive down-sampling rates and adaptive up-sampling filters in the coding scheme. More adaptation will be considered in the future.

Fig. 1 depicts the flowchart of our proposed intra frame coding scheme. An input frame is divided into blocks while for each block the best coding mode is decided. In this paper, the block is chosen to be of the same size as CTU, i.e. consisting of 64$\times$64 luma samples (Y) and 2 channels of 32$\times$32 chroma samples (U and V, or Cb and Cr), due to the YUV 4:2:0 format. Each CTU can be either coded at its full resolution, or down-sampled and coded at low resolution. Here, the down-sampling is performed using the fixed filters presented in [22].

Next, if the CTU is down-sampled and coded, it should be up-sampled back to its original resolution so as not to disrupt the normal intra coding of the subsequent CTUs. For this up-sampling, each down-sampled CTU can choose either CNN-based up-sampling, or the fixed, discrete cosine transform based interpolation filters (DCTIF) [23]. We adopt DCTIF in addition to our proposed CNN-based up-sampling, because DCTIF is already adopted in HEVC for fractional pixel interpolation for motion compensation [2], and it is computationally simple but achieves good quality for smooth image regions. While CNN is much more complicated than DCTIF, we expect CNN to deal with complex image regions such as structures. The CNN-based up-sampling is elaborated in Section IV.

There are two mode decision steps shown in Fig. 1. The first is for each down-sampled coded CTU, one up-sampling method is decided. This is performed by comparing the up-sampled results of both methods with the original CTU, and choosing the result with less distortion, since the down-sampled coding rate is the same. The second mode decision is to choose low-resolution coding or full-resolution coding for each CTU, which is performed by comparing the rate-distortion (R-D) costs of both coding modes. The distortion values of both coding modes are calculated at full resolution for fair comparison. Due to the down-sampling, low-resolution coding may incur much higher distortion but needs much less coding rate, thus it would be beneficial to adjust the coding parameters for down-sampled coded CTUs to pursue the overall R-D optimization, as elaborated in Section V.

In addition, the block-level down/up-sampling bears a side effect of the boundary conditions during down- and up-sampling. Specifically, all the down- and up-sampling methods, including CNN-based ones, need appropriate boundary conditions. In general, such methods perform worse at image boundaries due to lack of information. We carefully address this problem. For down-sampling there are two cases: first, the original frame is entirely down-sampled to provide the down-sampled version of each CTU to be compressed; second, if a CTU chose full-resolution coding mode, the reconstructed CTU needs to be down-sampled so as to provide appropriate reference for the intra prediction of subsequent down-sampled CTUs. In both cases, we adopt the border replication method, i.e. replicating the values at the borders outwards, to provide the unavailable pixels at image boundaries or CTU boundaries. For up-sampling, we propose a two-stage method that uses different boundary conditions. The two-stage up-sampling is depicted in Fig. 1, and will be elaborated in Section VI.

IV CNN-Based Up-sampling

Image SR is a severely ill-posed problem, and the key to relax the ill-posedness is the modeling of natural image prior. Training CNN for image SR is essentially embedding the natural image prior into the network parameters. And previous work [12, 13, 14, 15, 16] has demonstrated that CNN-based SR outperforms almost all the other methods in terms of both objective and subjective reconstruction quality. Hence, we hope to develop an efficient CNN-based up-sampling method to be applied into our intra frame coding scheme.

A trend in deep learning is to use deeper and deeper networks. For example, SRCNN [12] has 3 layers, but VDSR [15] has 20 layers. Though the latter indeed achieves higher reconstruction quality, it also incurs higher computational cost. How to balance the reconstruction quality and computational complexity is an important issue to consider when designing the CNN structure, especially in video coding. In addition, note that the blocks to be up-sampled in our scheme have been compressed, and the distortion may be significant because of low bit rate coding. Thus, the CNN is expected to alleviate the distortion while at the same time to perform super-resolution. We are then motivated to explore a five-layer CNN for up-sampling, more complex than SRCNN (to deal with coding distortion) but much simpler than VDSR (to reduce computational cost). The network structures for the up-sampling of luma and chroma components are depicted in Figs. 2 and 3, and discussed in the following two subsections, respectively.

IV-A CNN for Luma Up-sampling

To achieve high reconstruction quality with a shallow network, we have borrowed some key ingredients from previous work, such as resolution change within the network, multi-scale fusion, and residue learning. The CNN for luma up-sampling (shown in Fig. 2) can be divided into four parts: multi-scale feature extraction, deconvolution, multi-scale reconstruction, and residue learning, which are discussed one by one in the following.

IV-A1 Multi-scale Feature Extraction

There are two layers designed to extract multi-scale features from the input LR block. Each layer consists of multiple convolutional kernels, each of which is followed by a rectified linear unit (ReLU) as nonlinear activation function. It is well known that, an impressive advantage of CNN is to automate the feature extraction from raw data, which eliminates the necessity of hand-crafted features. Therefore, we directly input the LR compressed block into CNN without any pre-processing.

The first layer of CNN can be expressed as

[TABLE]

where $\mathbf{W}_{1}$ and $\mathbf{B}_{1}$ represent the convolutional filters and biases of the first layer, respectively, $\mathbf{X}$ is the input LR block, $\mathbf{F}_{1}$ indicates the feature maps of the first layer, and $*$ stands for convolution.

Since the input block is already compressed, it contains compression noise especially when quantization parameter (QP) is large. The feature maps extracted by the first layer may still contain noise, and thus the second layer is inserted to suppress noise and to enhance useful features:

[TABLE]

where $(\mathbf{F}_{21},\mathbf{F}_{22})$, $(\mathbf{W}_{21},\mathbf{W}_{22})$, $(\mathbf{B}_{21},\mathbf{B}_{22})$ are the extracted feature maps, convolutional filters, and biases, respectively. Note that there are two sets of convolutional kernels that have different kernel sizes in the second layer. Different sized kernels have receptive fields at different scales, and the combination of them is capable in effectively aggregating multi-scale information, which has been widely adopted in computer vision [24, 25]. Here in the second layer, the combination of different sized kernels provides multi-scale features to be explored for super-resolution. Note that the output feature maps $\mathbf{F}_{21}$ and $\mathbf{F}_{22}$ are directly concatenated and fed into the next layer.

IV-A2 Deconvolution

In most of the previous work on image SR, either CNN-based or not, an input LR image is first up-sampled by a fixed interpolation filter (e.g. bicubic) and then enhanced. The enhancement process does not change the resolution. However, it has been pointed out that the fixed interpolation filter before enhancement may cause the loss of important information in the original LR image. An end-to-end learning, embedding the resolution change into CNN, is believed better [14]. There are two techniques in CNN for resolution upgrade: un-pooling [26] and deconvolution [27]. While the un-pooling tends to yield enlarged but sparse output, we adopt the deconvolution in our designed CNN.

As shown in Fig. 2, the third layer performs deconvolution of the multi-scale feature maps extracted by the second layer. Deconvolution changes the resolution of input by multiplying each input pixel by a filter to produce a window, and then summing over the resulting windows. A ReLU is then appended to the deconvolution, leading to

[TABLE]

where the symbol $\star$ denotes deconvolution.

The relative position of the deconvolution layer in the CNN is also an issue to consider. It can be put at the beginning, in the middle, or at the end of the entire CNN. In our designed CNN, the deconvolution layer is used to enlarge the multi-scale feature maps and the enlarged features are then used to reconstruct HR image, then it is in the middle. We have tried to put it at other positions, but empirical results show the decrease of reconstruction quality then.

IV-A3 Multi-scale Reconstruction

The reconstruction stage is composed by two convolutional layers. The fourth layer, similar to the second, performs multi-scale fusion by using two sets of convolutional kernels with different sizes,

[TABLE]

This layer takes into account both long- and short-range contextual information for reconstruction.

Then, the fifth layer performs reconstruction,

[TABLE]

Note that the fifth layer has no nonlinear unit.

IV-A4 Residue Learning

Residue learning in CNN is proposed by He et al., who introduced skip-layer connections in CNN to achieve both faster convergence in training and better performance [21]. We also adopt residue learning in our network and have observed indeed faster convergence in training. Specifically, the down-sampled block is up-sampled by a fixed interpolation filter (DCTIF in this paper for consistency) and then added to the reconstruction produced by the five-layer CNN,

[TABLE]

[TABLE]

In other words, the five-layer CNN is supposed to learn the difference between an original block and its degraded version, where the degraded version is generated by down-sampling the block, coding, and then up-sampling by DCTIF. The difference is indeed the high-frequency details in the original block that have been lost during down-sampling and coding. Learning to recover high-frequency details instead of the original image is a common strategy in image SR, with or without CNN [15, 28].

Let the original HR block be $\mathbf{Y}$, the difference between $\mathbf{Y}$ and $\hat{\mathbf{Y}}(\mathbf{X})$ calculated by mean-squared-error (MSE) drives the training of our CNN. The MSE is minimized by means of stochastic gradient descent together with standard error back-propagation algorithm.

IV-B CNN for Chroma Up-sampling

In most of the previous work on image SR, chroma components are simply interpolated by a fixed filter (e.g. bicubic) without enhancement. This is because human vision tends to be less sensitive to the change of chrominance signal, which is also the reason why the chroma components have a lower resolution in YUV 4:2:0 format. However in our coding scheme, we may further down-sample the chroma components and need to up-sample them, so we have designed a separate CNN for chroma up-sampling to achieve higher reconstruction quality. The chroma up-sampling CNN is depicted in Fig. 3, whose structure is quite similar to the luma one but augmented with two features:

IV-B1 Incorporating Luma Information

In the widely adopted YUV 4:2:0 format, luma and chroma components have been decomposed by conversion from RGB to YCbCr in advance. However, the decomposition did not fully remove the correlation among the three channels of RGB. There is still correlation between Y and Cb/Cr as can be observed from the example plots in Fig. 4. Motivated by this, predicting chroma from luma has been proposed for video coding [29, 30]. Similarly in this paper, we incorporate the luma information during the up-sampling of chroma components to improve the reconstruction quality. Moreover, the correlation between Y and Cb/Cr cannot be well described by simple linear models, as shown in Fig. 4, which inspires us to leverage the non-linear CNN models to exploit such correlation.

As shown in Fig. 3, we use all the three channels (Y, Cb, and Cr) as input to CNN. Note that for down-sampled CTUs, the luma component has 32$\times$32 elements while the chroma components have only 16$\times$16 elements each. We further down-sample the luma component to the same size as chroma to simplify the network design. Then, cross-channel features can be extracted by the first layer, and processed by the following layers sequentially.

IV-B2 Joint Training of Cb and Cr

While it is possible to train two separate networks for Cb and Cr respectively, we believe the high similarity between Cb and Cr can help reduce the amount of required models. Specifically, the CNN shown in Fig. 3 outputs reconstructed Cb and Cr simultaneously, i.e. the former four layers are exactly the same for Cb and Cr, and only the last layer is different. During training, the MSE of both Cb and Cr is used as the objective of minimization. This design leads to fewer trained models, while incurs negligible loss of reconstruction quality, as observed from our empirical results.

V Coding Parameters Setting

In this section, we would like to derive the optimal coding parameters for down-sampled CTUs so as to pursue frame-level R-D optimization. We start from the basic objective function of R-D optimization, i.e.

[TABLE]

where $J$ is the overall R-D cost, $D_{i}$ and $R_{i}$ are the distortion and rate of the $i$-th CTU, respectively, and $N$ is the total number of CTUs in the frame. $\lambda$ is the Lagrangian multiplier. In the case of intra frame coding, the compression of each CTU can be regarded as approximately independent, because of the less accurate intra prediction between CTUs [31]. Therefore, we consider the R-D cost of one CTU, and for simplicity the subscript $i$ is omitted hereafter.

In our coding scheme, the CTU can be coded at full resolution or at low resolution, but in both coding modes, the distortion $D$ shall be calculated at full resolution. However, during low-resolution coding, it is not easy to calculate the full-resolution distortion, denoted by $D_{full}$. Specifically, the down-sampled CTU (32$\times$32 in luma) is compressed by normal HEVC intra coding, during which the quadtree partition, the intra prediction modes, the quantized transform coefficients, as well as other syntax elements, need to be determined in an R-D optimized fashion. If $D_{full}$ is requested in low-resolution coding, then the down-sampled CTU needs to be up-sampled many times during the R-D optimization process of low-resolution coding. It is not only computationally expensive, but also not friendly to up-sampling that, as mentioned before, is sensitive to the lack of proper boundary conditions. Therefore, we prefer calculating the distortion directly at the low resolution, i.e. $D_{low}$, during low-resolution coding. Accordingly, in the low-resolution coding mode, $D_{full}$ is calculated only once, after the down-sampled CTU is entirely compressed and up-sampled.

Here, we take an empirical approach for investigation of the relation between $D_{full}$ and $D_{low}$. We have compressed many natural images/videos using the low resolution coding mode and different QPs, and calculated the pairs of $(D_{full},D_{low})$ in terms of sum-of-squared-difference. Some typical results are shown in Fig. 5, indicating that a linear model can be used to describe the relation, i.e.

[TABLE]

The fitted values of $\alpha$ and $\beta$ are also shown in Fig. 5. Note that different CTUs have different values. This equation seems quite intuitive, as the full-resolution distortion can be decomposed into two parts, one part incurred by the low-resolution coding, and the other part corresponding to the lost high-frequency information during down-sampling.

Given (9), the R-D cost of one CTU can be written as

[TABLE]

The R-D cost during low-resolution coding can be written as

[TABLE]

Note that the $R$ is the same in both (10) and (11). Thus, if we choose $\lambda_{low}=\frac{\lambda}{\alpha}$, then the optimization of (11) and that of (10) are equivalent. Moreover, in HEVC the quantization parameter (QP) is known to depend on the Lagrangian parameter $\lambda$, i.e.

[TABLE]

Then, the QP during low-resolution coding should be changed accordingly into

[TABLE]

This equation is also intuitively meaningful, because the low-resolution coding in general leads to less rate but more distortion, and we need to lower the QP to make both rate and distortion of low-resolution coding to be comparable to that of full-resolution coding.

However, if we adjust QP according to (13), the $\alpha$ value of each CTU is distinct, i.e. each low-resolution coded CTU has a different QP, which requires additional bits to encode. Besides, it is not easy to determine the $\alpha$ value of each CTU in practice. We are then motivated to use a predefined $\alpha$ or equivalently a fixed delta QP for low-resolution coding. To this end, we perform statistical analysis of the fitted $\alpha$ values using many natural images/videos. The empirical distribution of $\alpha$ is plotted in Fig. 6, indicating the mode of $\alpha$ is around 4. This number is reasonable as our down-sampling rate is $1/2\times 1/2$. Therefore, in our experiments, we set fixed coding parameters for low-resolution coding, i.e.

[TABLE]

[TABLE]

VI Two-Stage Up-sampling

We design a two-stage up-sampling process as shown in Fig. 1. The difference between two stages can be observed from Fig. 7. In the first stage, the CTU needs to be up-sampled for the coding of subsequent CTUs, the up-sampling at this stage can use the top and left boundaries but cannot use the bottom and right ones as they are not compressed yet. In our implementation, we fill the unavailable boundaries with zero values. However, in the second stage, the entire frame has been compressed, so the up-sampling can use all available boundaries. In essence, the second stage refines the region of each up-sampled CTU around its bottom and right boundaries. This is valid for both CNN- and DCTIF-based up-sampling.

The second stage of up-sampling is performed for only the CTUs that have chosen the low-resolution coding mode, and the up-sampling method (CNN-based or DCTIF) is already decided in the first stage. The up-sampling result of the second stage just replaces that of the first stage. The same process is performed at both encoder and decoder, then no overhead bit is required.

VII Experimental Results

We conduct extensive experiments to evaluate the performance of the proposed methods. Experimental settings are introduced, followed by the detailed experimental results and analyses in this section.

VII-A Experimental Settings

VII-A1 Implementation and Configuration

We have implemented our proposed intra frame coding scheme based on the reference software of HEVC, i.e. HM version 12.1111https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-12.1/. In HEVC intra coding, each CTU is partitioned into coding units based on a quadtree, and the luma and chroma components of one CTU must follow the same quadtree. To comply with this, the mode decision between full- and low-resolution coding is performed at CTU level combining luma and chroma, i.e. the R-D costs of luma and chroma are summed up to make decision. On the contrary, the mode decision of which up-sampling method is performed individually for Y, Cb, and Cr, i.e. if a CTU chooses low-resolution coding, three binary flags are required to indicate CNN-based or DCTIF for the channels Y, Cb, and Cr, respectively.

The CNN-based up-sampling method has been realized using Caffe [32], a popular framework for deep learning, to reuse its highly efficient implementation of convolutions.

We use the all-intra configuration suggested by HEVC common test conditions [33]. Considering down/up-sampling-based coding is a useful tool especially at low bit rates, the QP is set to {32, 37, 42, 47}. BD-rate [34] is adopted to evaluate the compression efficiency, where for the quality metric we use both PSNR and structural similarity (SSIM) [18], as the latter is believed to be more consistent with subjective quality.

VII-A2 Test Sequences

The HEVC common test sequences, including 20 video sequences of different resolutions known as Classes A, B, C, D, E [33], are used for experiments. Class F (screen content videos) is excluded as our proposed technique is designed for natural videos. In addition, to demonstrate the performance on high definition videos, we use five 4K (3840$\times$2160) sequences from the SJTU dataset [35] in experiments, as shown in Table II. For each sequence, we use only the first frame in experiments, and our empirical results indicate that the comparative results using entire sequences have similar trends.

VII-A3 CNN Training

The Caffe software is also used to train CNN models. We use the Uncompressed Colour Image Database (UCID) [19], which consists of 1338 natural images, to prepare the training data. The training data and test data (video sequences) have no overlap to demonstrate the generalization ability of CNN. The images in UCID are compressed by our scheme using different QPs, but all CTUs are forced to use the low-resolution coding mode and DCTIF for up-sampling. The reconstructed LR CTUs together with the original ones are formed into pairs of $(\mathbf{X},\mathbf{Y})$ to train the CNN as described in Section IV. It is worth noting that we have trained a different model for each QP and for Y or Cb/Cr, so in total we have 8 CNN models corresponding to the four QPs.

VII-B Results and Analyses

VII-B1 Overall Performance

The overall performance measured by BD-rate is shown in Table III. Columns under “Anchored on HEVC” are the results comparing our scheme with HM 12.1 anchor. As can be observed, our scheme improves the coding efficiency significantly, leading to on average 5.5%, 6.0%, and 2.2% BD-rate reductions on Y, U, and V, respectively, for HEVC test sequences (Classes A–E). As for UHD test sequences, our scheme achieves even higher coding gain, i.e. 9.0%, 1.6%, and 3.2% BD-rate reductions on Y, U, and V.

It is worth noting that the images used in training all have a bit-depth of 8, but there are two 10-bit sequences for test, i.e. Nebuta and SteamLocomotive (in Class A). For these two sequences, the BD-rate reduction on Y is limited but on U and V are still significant. It is possible to further improve for such sequences by including high-dynamic-range images during training.

For a few sequences, we observe the BD-rate on U and V is positive, indicating performance loss of our scheme, but for such sequences the BD-rate reduction on Y is still visible. The reason of such phenomenon is that for several CTUs, the luma component prefers low-resolution coding but the chroma components prefer full-resolution coding. However, our current implementation forces the modes (full or low) of luma and chroma to be the same to suit for HEVC intra coding. This constraint may be removed in the future to pursue better performance.

In addition, when using SSIM as quality metric, the BD-rate reductions are more significant, i.e. 8.8% and 10.5% on Y for HEVC and UHD sequences, respectively. Thus, we believe down/up-sampling-based coding is more friendly to the subjective quality at low bit rates.

We conduct another experiment to demonstrate the benefit of using CNN for up-sampling in addition to the fixed interpolation filters. In this experiment, the CNN-based up-sampling in our scheme is disabled and DCTIF is the only up-sampling method. Comparative results measured by BD-rate are presented in columns under “Anchored on HEVC+DCTIF” in Table III. As can be observed, adopting CNN-based up-sampling improves the coding efficiency of down/up-sampling-based coding by a considerable margin. The BD-rate reductions on Y, U, and V are on average 4.3%, 10.0%, and 6.0% for HEVC test sequences, and on average 5.1%, 10.5%, and 9.9% for UHD test sequences.

Some typical R-D curves achieved by different schemes are shown in Fig. 8. It can be observed that for most of the test sequences, our scheme achieves higher coding gain at lower bit rates, which is a nature of down/up-sampling-based coding. It is also visible that for different sequences, the switching bit-rates, at which the R-D curves of down/up-sampling-based coding and normal coding cross over, are quite diverse. Actually the switching bit-rate should be content dependent, which highlights the necessity of mode selection between low- and full-resolution coding.

In addition to the QPs adopted for the experiments in this paper (i.e. {32, 37, 42, 47}), we also tested the QPs 22 and 27 according to the HEVC common test conditions. Note that additional CNN models are trained for these two QPs. Table IV summarizes the BD-rate results when comparing our scheme with HM anchor at different QPs. It can be observed, as QP increases, the BD-rate reductions become more and more significant. It again demonstrates down/up-sampling-based coding is useful especially at low bit rates.

VII-B2 Mode Selection Results

Since our proposed scheme decides whether to down-sample at block level, we perform analyses of the blocks that choose low-resolution coding mode to further understand the performance. Some symbols are defined as shown in Table V, and the hitting ratios are calculated as follows,

[TABLE]

[TABLE]

where the symbol $\#$ denotes counting the amount. Table VI presents the calculated hitting ratios. $P_{Hitting}$ is on average 72.2%, 68.4%, 48.1%, 42.4%, 68.7%, 85.2% for Classes A, B, C, D, E, UHD, respectively. Taken into account the resolutions of these videos, it is obvious that the hitting ratio becomes higher as the video resolution increases. It shows the effectiveness of down/up-sampling-based coding for high definition content, and also interprets the reason that our scheme achieves higher BD-rate reduction on UHD sequences. Moreover, among the blocks choosing low-resolution coding, a majority of them choose CNN-based up-sampling method, as can be observed from the last three columns of Table VI. Meanwhile, DCTIF is also useful for certain video content and especially for chroma components.

Fig. 9 is provided for visually inspecting the blocks that choose different coding modes and different up-sampling methods. We can observe that CNN-based method is good at reconstructing structural regions, whereas DCTIF is prone to be selected for smooth and some textural regions. For example, in Fig. 9 (a), most of the CTUs containing vehicles choose CNN-based up-sampling, while most of the CTUs corresponding to road choose DCTIF. Due to different properties of the luma and chroma components, the selections of up-sampling methods are not always consistent among Y, Cb, and Cr. Note the bottom right corner in Fig. 9 (a) and (b), the CTUs mostly choose CNN-based up-sampling for Y and Cb, but choose DCTIF for Cr, since the Cr component of these CTUs is quite smooth. In addition, low-resolution coding becomes more competitive when the bit rate is lower, as can be observed by comparing the hitting ratios in Fig. 9 (a) versus (b), and (c) versus (d).

VII-B3 Generalization of CNN for Different QPs

We have trained different CNN models for different QPs in the above experiments. In practice, it may be too costly to train a different model for every QP. Thus, we investigate the generalization ability of CNN for different QPs. In the following experiments, we use the models trained at four QPs: {32, 37, 42, 47}, but the QPs during compression are set to {34, 39, 44, 49} (denoted by QP$+2$), or {30, 35, 40, 45} (denoted by QP$-2$). For each test QP, the models trained at the nearest QP are retrieved for usage. Table VII summarizes the experimental results. BD-rate reductions are still observed from these results, showing the effectiveness of the trained models when used for different QPs. Therefore, the amount of models required in practice can be much less than the number of possible QPs. Furthermore, the BD-rate reductions of QP$+2$ are usually more significant than those of QP$-2$, since higher QP corresponds to lower bit rate that prefers low-resolution coding.

VII-B4 Verification of the Designed CNN

In order to verify the performance of our designed CNN, we have compared it with the fixed interpolation filter DCTIF as well as a state-of-the-art CNN-based image SR method, i.e. VDSR [15]. VDSR is a deep network consisting of 20 layers and is shown to outperform the shallow network, SRCNN [12], by a large margin. For fair comparison, we follow the instructions in [15] to train VDSR, but using our own training data produced when QP is 32. The comparative experiments are performed as follows. The test sequences are entirely down-sampled and then compressed with QP equal to 32, and then up-sampled by each method. The comparative results of the luma component are summarized in Table VIII. It can be observed that both VDSR and our CNN-based method outperform DCTIF significantly. Our CNN-based method is better than VDSR for most of the test sequences, and achieves on average 0.16 dB gain. It is worth noting that our network is shallower and simpler than VDSR, but is very competitive due to the adopted multi-scale fusion and deconvolution, which are not used in VDSR.

We have also verified our designed chroma up-sampling CNN experimentally. In previous work on image SR, the chroma components are usually up-sampled by fixed interpolation filters. So we compare three methods: DCTIF, CNN without luma, and CNN with luma. The CNN without luma method has a similar network structure to that shown in Fig. 3 but excluding the luma information from the network input. The CNN without luma network is also trained under the same setting and using the same training data. The experimental settings are identical to those in the previous paragraph, and comparative results are shown in Table IX. It can be observed that CNN-based methods outperform DCTIF consistently, but the PSNR gain is not as much as that for luma (in Table VIII), since the chroma components of natural images are usually quite smooth and the potential improvement is limited. Moreover, the proposed CNN using luma achieves better performance than the CNN without luma, leading to on average 0.20 dB and 0.22 dB gain for Cb and Cr, respectively. Such results confirm the effectiveness of using luma information to boost the chroma up-sampling performance.

VII-B5 Verification of Two-Stage Up-sampling

We have verified the proposed two-stage up-sampling strategy by comparing with only one stage of up-sampling. Table X presents the average MSE of the reconstructed CTUs that choose low-resolution coding mode, after the first stage and after the second stage, respectively. The percentage of CTUs that benefit from the second stage (i.e. MSE decreases) is also shown in the table. Table XI further presents the BD-rate results of using the second stage. The BD-rate reductions provided by the second stage of up-sampling are on average 0.7%, 2.7%, 3.0% for HEVC test sequences, and 0.8%, 3.4%, 3.7% for UHD test sequences, on Y, U, and V, respectively. As shown, the BD-rate reductions on chroma components are higher than luma. This is due to the lower resolution of chroma (32$\times$32 for CTU) that incurs more severe influence by the lack of boundary information. Note that in our current implementation, the result of the first stage is simply replaced by that of the second stage. But as can be observed in Table X, there are a portion of blocks for which the second stage incurs worse result. We may adaptively decide whether to perform the second stage for each block, which will be studied in the future.

VII-B6 Computational Complexity

One drawback of CNN-based up-sampling methods is the high computational complexity compared to simple interpolation filters such as DCTIF. In our current implementation, the CNN is not optimized for computational speed, and thus the encoding/decoding time of our scheme is much longer than that of the highly optimized HEVC anchor. The computational time comparison is summarized in Table XII. It can be observed the increase of encoding time varies little across different videos, but the increase of decoding time varies much. In fact, most of the decoding computations are cost on the CNN up-sampling, and thus the increase of decoding time is dependent on the amount of blocks that choose CNN up-sampling.

When designing our CNN, we have tried to keep it simple while pursuing high reconstruction quality. Accordingly, we have compared the computational complexity of our CNN and VDSR, since both networks achieve comparable reconstruction quality as presented in Section VII-B4. Experimental results show that, it takes on average 0.032 and 0.123 seconds for our CNN and VDSR, respectively, to up-sample the luma component of a down-sampled and coded CTU, on a workstation with an Intel CPU at 4.0 GHz. Our CNN is almost 4 times faster than VDSR, the number being proportional to the amount of layers (our CNN has 5 layers and VDSR has 20 layers). It again demonstrates the advantage of our designed CNN.

VIII Conclusion

In this paper, we propose a CNN-based block up-sampling scheme for intra frame coding. A CTU is optionally down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution, so as to enable adaptive sampling rates for different CTUs. We carefully design the CNN structures for the up-sampling of luma and chroma, and propose to combine the CNN-based up-sampling with the fixed interpolation filter (DCTIF) to allow for adaptive up-sampling methods for the down-sampled CTUs. We also propose a two-stage up-sampling process in accordance to the block-level down/up-sampling, and study the coding parameters setting of the down-sampled CTUs for pursing frame-level R-D optimization. Our proposed methods are implemented based on HEVC reference software, and extensive experimental results demonstrate the superior performance achieved by the proposed methods than HEVC anchor. On average 5.5% and 9.0% BD-rate reductions are achieved by our scheme for HEVC common test sequences and UHD test sequences, respectively. The proposed scheme is especially useful for compressing high definition videos at low bit rates.

In the future, we plan to extend this work in two directions. First, the idea of CNN-based block-level down/up-sampling may be extended for inter frame coding in addition to intra; how to integrate down/up-sampling with motion estimation/compensation is a key issue therein. Second, though the proposed CNN structure in this paper is much simpler than the state-of-the-arts, its computational complexity is still quite high; how to accelerate the speed of CNN-based up-sampling needs to be addressed. We anticipate that new hardware/firmware based implementation of CNN is probably the solution.