HAVIT: research on vision-language gesture interaction mechanism for smart furniture

TL;DR

This paper introduces HAVIT, a new model for gesture recognition in smart furniture that works well even with limited data.

Contribution

HAVIT combines Vision Transformer and ALBEF to improve gesture recognition in data-scarce scenarios.

Findings

HAVIT achieved 91.83% accuracy and 0.92 AUC on a fully labeled dataset.

It maintained 86.89% accuracy and 0.88 AUC with 20% label deficiency.

The model shows strong robustness and potential for smart furniture applications.

Abstract

With the rapid development of smart furniture, gesture recognition has gained increasing attention as a natural and intuitive interaction method. However, in practical applications, issues such as limited data resources and insufficient semantic understanding have significantly constrained the effectiveness of gesture recognition technology. To address these challenges, this study proposes HAVIT, a hybrid deep learning model based on Vision Transformer and ALBEF, aimed at enhancing the performance of gesture recognition systems under data-scarce conditions. The model achieves efficient feature extraction and accurate recognition of gesture characteristics through the organic integration of Vision Transformer’s feature extraction capabilities and ALBEF’s semantic understanding mechanism. Experimental results demonstrate that on a fully labeled dataset, the HAVIT model achieved a classification accuracy of 91.83% and an AUC value of 0.92; under 20% label deficiency conditions, the model maintained an accuracy of 86.89% and an AUC value of 0.88, exhibiting strong robustness. The research findings provide new solutions for the development of smart furniture interaction technology and hold significant implications for advancing practical applications in this field.

Figures