Convolutional Drift Networks for Video Classification

TL;DR

This paper introduces Convolutional Drift Networks (CDN), a novel neural architecture combining CNNs and Reservoir Computing to efficiently analyze spatio-temporal video data without complex training.

Contribution

The paper presents the CDN architecture that integrates CNNs with Reservoir Computing for simplified, effective video classification without hand-crafted features.

Findings

Achieved state-of-the-art level performance on egocentric video datasets.

Only trained a single feed-forward layer in the CDN.

Demonstrated effectiveness on complex spatio-temporal tasks.

Abstract

Analyzing spatio-temporal data like video is a challenging task that requires processing visual and temporal information effectively. Convolutional Neural Networks have shown promise as baseline fixed feature extractors through transfer learning, a technique that helps minimize the training cost on visual information. Temporal information is often handled using hand-crafted features or Recurrent Neural Networks, but this can be overly specific or prohibitively complex. Building a fully trainable system that can efficiently analyze spatio-temporal data without hand-crafted features or complex training is an open challenge. We present a new neural network architecture to address this challenge, the Convolutional Drift Network (CDN). Our CDN architecture combines the visual feature extraction power of deep Convolutional Neural Networks with the intrinsically efficient temporal processing provided by Reservoir Computing. In this introductory paper on the CDN, we provide a very simple baseline implementation tested on two egocentric (first-person) video activity datasets.We achieve video-level activity classification results on-par with state-of-the art methods. Notably, performance on this complex spatio-temporal task was produced by only training a single feed-forward layer in the CDN.