# A Temporal Sequence Learning for Action Recognition and Prediction

## TL;DR

This paper introduces a novel approach for human action recognition and prediction by representing videos as sequences of words and applying a Temporal CNN to learn their temporal order, achieving high accuracy with low latency.

## Contribution

The work presents a new method that models videos as sentences of visual words and uses a Temporal CNN to effectively predict and recognize actions with partial video sequences.

## Key findings

- Achieves 95% accuracy with half the video frames on UCF101 and HMDB51.
- Demonstrates low-latency prediction capability.
- Attains state-of-the-art performance at full sequence completion.

## Abstract

In this work\footnote {This work was supported in part by the National Science Foundation under grant IIS-1212948.}, we present a method to represent a video with a sequence of words, and learn the temporal sequencing of such words as the key information for predicting and recognizing human actions. We leverage core concepts from the Natural Language Processing (NLP) literature used in sentence classification to solve the problems of action prediction and action recognition. Each frame is converted into a word that is represented as a vector using the Bag of Visual Words (BoW) encoding method. The words are then combined into a sentence to represent the video, as a sentence. The sequence of words in different actions are learned with a simple but effective Temporal Convolutional Neural Network (T-CNN) that captures the temporal sequencing of information in a video sentence. We demonstrate that a key characteristic of the proposed method is its low-latency, i.e. its ability to predict an action accurately with a partial sequence (sentence). Experiments on two datasets, \textit{UCF101} and \textit{HMDB51} show that the method on average reaches 95\% of its accuracy within half the video frames. Results, also demonstrate that our method achieves compatible state-of-the-art performance in action recognition (i.e. at the completion of the sentence) in addition to action prediction.