Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition

TL;DR

This paper introduces PatchEchoClassifier, a reservoir-based model for human activity recognition that uses knowledge distillation from a high-capacity teacher to achieve high accuracy with significantly reduced computational cost, suitable for edge devices.

Contribution

The paper presents a novel reservoir-based classifier with a knowledge distillation framework for efficient human activity recognition from sensor data.

Findings

Achieves over 80% accuracy on HAR datasets.

Requires only about one-sixth of FLOPS compared to DeepConvLSTM.

Demonstrates suitability for real-time, energy-efficient edge computing.

Abstract

This paper aims to develop an energy-efficient classifier for time-series data by introducing PatchEchoClassifier, a novel model that leverages a reservoir-based mechanism known as the Echo State Network (ESN). The model is designed for human activity recognition (HAR) using one-dimensional sensor signals and incorporates a tokenizer to extract patch-level representations. To train the model efficiently, we propose a knowledge distillation framework that transfers knowledge from a high-capacity MLP-Mixer teacher to the lightweight reservoir-based student model. Experimental evaluations on multiple HAR datasets demonstrate that our model achieves over 80 percent accuracy while significantly reducing computational cost. Notably, PatchEchoClassifier requires only about one-sixth of the floating point operations (FLOPS) compared to DeepConvLSTM, a widely used convolutional baseline. These results suggest that PatchEchoClassifier is a promising solution for real-time and energy-efficient human activity recognition in edge computing environments.