Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion

TL;DR

This paper introduces OV-HAR, a novel open vocabulary human activity recognition framework that encodes activities into text embeddings and decodes them back, enabling recognition of unseen activities without relying on large language models.

Contribution

OV-HAR is the first approach to use text embedding inversion for open vocabulary activity recognition across multiple sensor modalities.

Findings

Robust generalization to unseen activities and modalities

Effective conversion of activities into natural language for recognition

Achieves open vocabulary recognition without large language models

Abstract

Conventional human activity recognition (HAR) relies on classifiers trained to predict discrete activity classes, inherently limiting recognition to activities explicitly present in the training set. Such classifiers would invariably fail, putting zero likelihood, when encountering unseen activities. We propose Open Vocabulary HAR (OV-HAR), a framework that overcomes this limitation by first converting each activity into natural language and breaking it into a sequence of elementary motions. This descriptive text is then encoded into a fixed-size embedding. The model is trained to regress this embedding, which is subsequently decoded back into natural language using a pre-trained embedding inversion model. Unlike other works that rely on auto-regressive large language models (LLMs) at their core, OV-HAR achieves open vocabulary recognition without the computational overhead of such models. The generated text can be transformed into a single activity class using LLM prompt engineering. We have evaluated our approach on different modalities, including vision (pose), IMU, and pressure sensors, demonstrating robust generalization across unseen activities and modalities, offering a fundamentally different paradigm from contemporary classifiers.