SelaFD:Seamless Adaptation of Vision Transformer Fine-tuning for Radar-based Human Activity Recognition

TL;DR

This paper introduces SelaFD, a novel fine-tuning approach for Vision Transformers using Low-Rank Adaptation and feature space adapters, significantly improving radar-based human activity recognition accuracy.

Contribution

It presents a new joint fine-tuning method with LoRA and feature adapters tailored for radar Time-Doppler signatures, outperforming existing methods.

Findings

Achieved superior HAR accuracy over state-of-the-art methods

Demonstrated effectiveness of LoRA in non-visual radar data

Enhanced feature extraction with serial-parallel adapters

Abstract

Human Activity Recognition (HAR) such as fall detection has become increasingly critical due to the aging population, necessitating effective monitoring systems to prevent serious injuries and fatalities associated with falls. This study focuses on fine-tuning the Vision Transformer (ViT) model specifically for HAR using radar-based Time-Doppler signatures. Unlike traditional image datasets, these signals present unique challenges due to their non-visual nature and the high degree of similarity among various activities. Directly fine-tuning the ViT with all parameters proves suboptimal for this application. To address this challenge, we propose a novel approach that employs Low-Rank Adaptation (LoRA) fine-tuning in the weight space to facilitate knowledge transfer from pre-trained ViT models. Additionally, to extract fine-grained features, we enhance feature representation through the integration of a serial-parallel adapter in the feature space. Our innovative joint fine-tuning method, tailored for radar-based Time-Doppler signatures, significantly improves HAR accuracy, surpassing existing state-of-the-art methodologies in this domain. Our code is released at https://github.com/wangyijunlyy/SelaFD.