HR-INR: Continuous Space-Time Video Super-Resolution via Event Camera

TL;DR

This paper introduces HR-INR, a novel continuous space-time video super-resolution framework that leverages event camera data and implicit neural representations to enhance resolution and frame rate, especially in dynamic scenes.

Contribution

It proposes a new INR-based C-STVSR method utilizing event cameras for better motion capture and long-term dependencies, addressing limitations of previous INR-based approaches.

Findings

Outperforms existing methods on multiple datasets

Effectively captures fast and complex motions

Demonstrates strong generalization capabilities

Abstract

Continuous space-time video super-resolution (C-STVSR) aims to simultaneously enhance video resolution and frame rate at an arbitrary scale. Recently, implicit neural representation (INR) has been applied to video restoration, representing videos as implicit fields that can be decoded at an arbitrary scale. However, existing INR-based C-STVSR methods typically rely on only two frames as input, leading to insufficient inter-frame motion information. Consequently, they struggle to capture fast, complex motion and long-term dependencies (spanning more than three frames), hindering their performance in dynamic scenes. In this paper, we propose a novel C-STVSR framework, named HR-INR, which captures both holistic dependencies and regional motions based on INR. It is assisted by an event camera -- a novel sensor renowned for its high temporal resolution and low latency. To fully utilize the rich temporal information from events, we design a feature extraction consisting of (1) a regional event feature extractor -- taking events as inputs via the proposed event temporal pyramid representation to capture the regional nonlinear motion and (2) a holistic event-frame feature extractor for long-term dependence and continuity motion. We then propose a novel INR-based decoder with spatiotemporal embeddings to capture long-term dependencies with a larger temporal perception field. We validate the effectiveness and generalization of our method on four datasets (both simulated and real data), showing the superiority of our method. The project page is available at https://github.com/yunfanLu/HR-INR