PatchEX: High-Quality Real-Time Temporal Supersampling through Patch-based Parallel Extrapolation

TL;DR

PatchEX is a novel real-time frame extrapolation technique that partitions tasks for parallel execution, achieving high-quality supersampling with significantly reduced latency for high-refresh-rate displays.

Contribution

It introduces a patch-based extrapolation method with parallel sub-task execution and inpainting, improving both quality and speed over existing methods.

Findings

65.29% PSNR improvement over ExtraNet

48.46% PSNR improvement over ExtraSS

6x faster than ExtraNet

Abstract

High-refresh rate displays have become very popular in recent years due to the need for superior visual quality in gaming, professional displays and specialized applications like medical imaging. However, high-refresh rate displays alone do not guarantee a superior visual experience; the GPU needs to render frames at a matching rate. Otherwise, we observe disconcerting visual artifacts such as screen tearing and stuttering. Temporal supersampling is an effective technique to increase frame rates by predicting new frames from other rendered frames. There are two methods in this space: interpolation and extrapolation. Interpolation-based methods provide good image quality at the cost of a higher latency because they also require the next rendered frame. On the other hand, extrapolation methods are much faster at the cost of quality. This paper introduces PatchEX, a novel frame extrapolation method that aims to provide the quality of interpolation at the speed of extrapolation. It smartly partitions the extrapolation task into sub-tasks and executes them in parallel to improve both quality and latency. It then uses a patch-based inpainting method and a custom shadow prediction approach to fuse the generated sub-frames. This approach significantly reduces the overall latency while maintaining the quality of the output. Our results demonstrate that PatchEX achieves a 65.29% and 48.46% improvement in PSNR over the latest extrapolation methods ExtraNet and ExtraSS, respectively, while being 6x and 2x faster, respectively.