# Exploring Frame Segmentation Networks for Temporal Action Localization

## TL;DR

This paper introduces a Frame Segmentation Network (FSN) that combines spatial and temporal CNNs for precise, dense frame-level action localization in videos, outperforming existing methods.

## Contribution

The paper proposes a novel end-to-end trainable FSN that integrates spatial and temporal context for improved temporal action localization, including a weakly supervised version.

## Key findings

- FSN achieves superior accuracy in frame-level localization.
- The model effectively combines spatial semantics with temporal context.
- Weakly supervised FSN performs well with only video-level labels.

## Abstract

Temporal action localization is an important task of computer vision. Though many methods have been proposed, it still remains an open question how to predict the temporal location of action segments precisely. Most state-of-the-art works train action classifiers on video segments pre-determined by action proposal. However, recent work found that a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. In this paper, we propose a Frame Segmentation Network (FSN) that places a temporal CNN on top of the 2D spatial CNNs. Spatial CNNs are responsible for abstracting semantics in spatial dimension while temporal CNN is responsible for introducing temporal context information and performing dense predictions. The proposed FSN can make dense predictions at frame-level for a video clip using both spatial and temporal context information. FSN is trained in an end-to-end manner, so the model can be optimized in spatial and temporal domain jointly. We also adapt FSN to use it in weakly supervised scenario (WFSN), where only video level labels are provided when training. Experiment results on public dataset show that FSN achieves superior performance in both frame-level action localization and temporal action localization.