# Two-Stream Action Recognition-Oriented Video Super-Resolution

## TL;DR

This paper introduces two specialized video super-resolution methods tailored for enhancing action recognition accuracy in low-resolution videos by emphasizing moving regions and ensuring temporal continuity.

## Contribution

It proposes two novel SR techniques, SoSR and ToSR, designed specifically for two-stream action recognition networks, addressing spatial importance and temporal consistency.

## Key findings

- Improved recognition accuracy on UCF101 and HMDB51 datasets.
- Effective emphasis on moving regions enhances spatial SR.
- Temporal continuity preservation boosts recognition performance.

## Abstract

We study the video super-resolution (SR) problem for facilitating video analytics tasks, e.g. action recognition, instead of for visual quality. The popular action recognition methods based on convolutional networks, exemplified by two-stream networks, are not directly applicable on video of low spatial resolution. This can be remedied by performing video SR prior to recognition, which motivates us to improve the SR procedure for recognition accuracy. Tailored for two-stream action recognition networks, we propose two video SR methods for the spatial and temporal streams respectively. On the one hand, we observe that regions with action are more important to recognition, and we propose an optical-flow guided weighted mean-squared-error loss for our spatial-oriented SR (SoSR) network to emphasize the reconstruction of moving objects. On the other hand, we observe that existing video SR methods incur temporal discontinuity between frames, which also worsens the recognition accuracy, and we propose a siamese network for our temporal-oriented SR (ToSR) training that emphasizes the temporal continuity between consecutive frames. We perform experiments using two state-of-the-art action recognition networks and two well-known datasets--UCF101 and HMDB51. Results demonstrate the effectiveness of our proposed SoSR and ToSR in improving recognition accuracy.