Trajectory-aware Shifted State Space Models for Online Video Super-Resolution

TL;DR

This paper introduces TS-Mamba, an online video super-resolution method that models long-term trajectories and aggregates spatio-temporal information efficiently, achieving state-of-the-art results with reduced computational complexity.

Contribution

The paper proposes a novel trajectory-aware shifted state space model (TS-Mamba) that enhances long-term temporal modeling and efficiency in online video super-resolution.

Findings

Achieves state-of-the-art performance on three VSR datasets.

Reduces computational complexity by over 22.7% in MACs.

Effectively models long-term trajectories for improved super-resolution.

Abstract

Online video super-resolution (VSR) is an important technique for many real-world video processing applications, which aims to restore the current high-resolution video frame based on temporally previous frames. Most of the existing online VSR methods solely employ one neighboring previous frame to achieve temporal alignment, which limits long-range temporal modeling of videos. Recently, state space models (SSMs) have been proposed with linear computational complexity and a global receptive field, which significantly improve computational efficiency and performance. In this context, this paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba), leveraging both long-term trajectory modeling and low-complexity Mamba to achieve efficient spatio-temporal information aggregation. Specifically, TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames. Then, a Trajectory-aware Shifted Mamba Aggregation (TSMA) module consisting of proposed shifted SSMs blocks is employed to aggregate the selected tokens. The shifted SSMs blocks are designed based on Hilbert scannings and corresponding shift operations to compensate for scanning losses and strengthen the spatial continuity of Mamba. Additionally, we propose a trajectory-aware loss function to supervise the trajectory generation, ensuring the accuracy of token selection when training our model. Extensive experiments on three widely used VSR test datasets demonstrate that compared with six online VSR benchmark models, our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% complexity reduction (in MACs).