Deep Neural Network-based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions

TL;DR

This survey reviews how deep learning-based neural enhancement techniques are integrated into content delivery systems for improved image and video streaming quality, discussing current architectures, challenges, and future research directions.

Contribution

It provides a comprehensive overview of neural enhancement integration in streaming systems, analyzing deployment challenges and identifying key trends and future directions.

Findings

Neural enhancement improves visual quality in streaming under variable network conditions.

Existing systems face deployment challenges related to model efficiency and real-time processing.

Future research should focus on optimizing neural models for deployment in diverse devices.

Abstract

Internet-enabled smartphones and ultra-wide displays are transforming a variety of visual apps spanning from on-demand movies and 360{\deg} videos to video-conferencing and live streaming. However, robustly delivering visual content under fluctuating networking conditions on devices of diverse capabilities remains an open problem. In recent years, advances in the field of deep learning on tasks such as super-resolution and image enhancement have led to unprecedented performance in generating high-quality images from low-quality ones, a process we refer to as neural enhancement. In this paper, we survey state-of-the-art content delivery systems that employ neural enhancement as a key component in achieving both fast response time and high visual quality. We first present the components and architecture of existing content delivery systems, highlighting their challenges and motivating the use of neural enhancement models as a countermeasure. We then cover the deployment challenges of these models and analyze existing systems and their design decisions in efficiently overcoming these technical challenges. Additionally, we underline the key trends and common approaches across systems that target diverse use-cases. Finally, we present promising future directions based on the latest insights from deep learning research to further boost the quality of experience of content delivery systems.