Knowledge Distillation for LLM-Based Human Activity Recognition in Homes

TL;DR

This paper explores how knowledge distillation can effectively train smaller LLMs for human activity recognition in smart homes, achieving near-large-model performance with significantly fewer parameters.

Contribution

It introduces a novel application of knowledge distillation to optimize LLMs for HAR, demonstrating comparable accuracy with much smaller models.

Findings

Smaller LLMs can approach the performance of larger models when fine-tuned with distilled knowledge.

Recognition performance improves with larger LLM sizes.

Fine-tuned small models achieve high accuracy with 50 times fewer parameters.

Abstract

Human Activity Recognition (HAR) is a central problem for context-aware applications, especially for smart homes and assisted living. A few very recent studies have shown that Large Language Models (LLMs) can be used for HAR at home, reaching high performance and addressing key challenges. In this paper, we provide new experimental results regarding the use of LLMs for HAR, on two state-of-the-art datasets. More specifically, we show how recognition performance evolves depending on the size of the LLM used. Moreover, we experiment on the use of knowledge distillation techniques to fine-tune smaller LLMs with HAR reasoning examples generated by larger LLMs. We show that such fine-tuned models can perform almost as well as the largest LLMs, while having 50 times less parameters.