Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets

TL;DR

This paper critically examines six popular human activity recognition datasets, revealing data ambiguities and recording irregularities that challenge current ML models, and proposes methods for dataset improvement.

Contribution

It provides a detailed analysis of dataset ambiguities, introduces a categorization mask for dataset patching, and highlights areas for enhancing data collection practices.

Findings

Identification of data ambiguities and irregularities

Analysis of false classification intersections

Proposals for dataset improvement

Abstract

The research of machine learning (ML) algorithms for human activity recognition (HAR) has made significant progress with publicly available datasets. However, most research prioritizes statistical metrics over examining negative sample details. While recent models like transformers have been applied to HAR datasets with limited success from the benchmark metrics, their counterparts have effectively solved problems on similar levels with near 100% accuracy. This raises questions about the limitations of current approaches. This paper aims to address these open questions by conducting a fine-grained inspection of six popular HAR benchmark datasets. We identified for some parts of the data, none of the six chosen state-of-the-art ML methods can correctly classify, denoted as the intersect of false classifications (IFC). Analysis of the IFC reveals several underlying problems, including ambiguous annotations, irregularities during recording execution, and misaligned transition periods. We contribute to the field by quantifying and characterizing annotated data ambiguities, providing a trinary categorization mask for dataset patching, and stressing potential improvements for future data collections.