RAG-HAR: Retrieval Augmented Generation-based Human Activity Recognition

TL;DR

RAG-HAR is a training-free, retrieval-augmented framework using large language models for human activity recognition, achieving state-of-the-art results without dataset-specific training.

Contribution

It introduces a novel retrieval-based approach leveraging LLMs and prompt optimization for accurate HAR without training or fine-tuning.

Findings

Achieves state-of-the-art performance on six HAR benchmarks.

Operates without model training or fine-tuning.

Capable of recognizing unseen human activities.

Abstract

Human Activity Recognition (HAR) underpins applications in healthcare, rehabilitation, fitness tracking, and smart environments, yet existing deep learning approaches demand dataset-specific training, large labeled corpora, and significant computational resources.We introduce RAG-HAR, a training-free retrieval-augmented framework that leverages large language models (LLMs) for HAR. RAG-HAR computes lightweight statistical descriptors, retrieves semantically similar samples from a vector database, and uses this contextual evidence to make LLM-based activity identification. We further enhance RAG-HAR by first applying prompt optimization and introducing an LLM-based activity descriptor that generates context-enriched vector databases for delivering accurate and highly relevant contextual information. Along with these mechanisms, RAG-HAR achieves state-of-the-art performance across six diverse HAR benchmarks. Most importantly, RAG-HAR attains these improvements without requiring model training or fine-tuning, emphasizing its robustness and practical applicability. RAG-HAR moves beyond known behaviors, enabling the recognition and meaningful labelling of multiple unseen human activities.