Batch-Based Activity Recognition from Egocentric Photo-Streams

TL;DR

This paper introduces a batch-driven deep learning approach using LSTM units for activity recognition from low-frame-rate egocentric photo-streams, effectively capturing temporal features without relying on event boundaries.

Contribution

It proposes two novel batch-processing architectures with overlapping and non-overlapping batches to improve temporal modeling in low-frame-rate activity recognition.

Findings

Effective in capturing temporal evolution of features

Does not require event boundary annotations

Validated on a public dataset with positive results

Abstract

Activity recognition from long unstructured egocentric photo-streams has several applications in assistive technology such as health monitoring and frailty detection, just to name a few. However, one of its main technical challenges is to deal with the low frame rate of wearable photo-cameras, which causes abrupt appearance changes between consecutive frames. In consequence, important discriminatory low-level features from motion such as optical flow cannot be estimated. In this paper, we present a batch-driven approach for training a deep learning architecture that strongly rely on Long short-term units to tackle this problem. We propose two different implementations of the same approach that process a photo-stream sequence using batches of fixed size with the goal of capturing the temporal evolution of high-level features. The main difference between these implementations is that one explicitly models consecutive batches by overlapping them. Experimental results over a public dataset acquired by three users demonstrate the validity of the proposed architectures to exploit the temporal evolution of convolutional features over time without relying on event boundaries.