End-to-End Fine-Grained Action Segmentation and Recognition Using Conditional Random Field Models and Discriminative Sparse Coding

TL;DR

This paper introduces an end-to-end framework combining a temporal CRF with discriminative sparse coding for fine-grained action segmentation and recognition, demonstrating improved performance on surgical and food preparation datasets.

Contribution

It presents a novel joint learning approach for CRF weights and a shared discriminative dictionary for structured output action recognition.

Findings

Performs on par or better than state-of-the-art methods

Effective on surgical and food preparation datasets

Joint learning of CRF and sparse coding enhances accuracy

Abstract

Fine-grained action segmentation and recognition is an important yet challenging task. Given a long, untrimmed sequence of kinematic data, the task is to classify the action at each time frame and segment the time series into the correct sequence of actions. In this paper, we propose a novel framework that combines a temporal Conditional Random Field (CRF) model with a powerful frame-level representation based on discriminative sparse coding. We introduce an end-to-end algorithm for jointly learning the weights of the CRF model, which include action classification and action transition costs, as well as an overcomplete dictionary of mid-level action primitives. This results in a CRF model that is driven by sparse coding features obtained using a discriminative dictionary that is shared among different actions and adapted to the task of structured output learning. We evaluate our method on three surgical tasks using kinematic data from the JIGSAWS dataset, as well as on a food preparation task using accelerometer data from the 50 Salads dataset. Our results show that the proposed method performs on par or better than state-of-the-art methods.