(LiFT) Lightweight Fitness Transformer: A language-vision model for Remote Monitoring of Physical Training

TL;DR

This paper presents LiFT, a scalable, privacy-preserving fitness tracking system using a vision-language transformer that can detect and count a wide variety of exercises from RGB video, advancing remote exercise monitoring.

Contribution

Developed a robust multitask vision-language transformer model capable of handling hundreds of exercises for remote fitness monitoring, overcoming previous data and generalization limitations.

Findings

Achieved 76.5% exercise detection accuracy.

Achieved 85.3% off-by-one repetition counting accuracy.

Assembled the large-scale Olympia dataset with over 1,900 exercises.

Abstract

We introduce a fitness tracking system that enables remote monitoring for exercises using only a RGB smartphone camera, making fitness tracking more private, scalable, and cost effective. Although prior work explored automated exercise supervision, existing models are either too limited in exercise variety or too complex for real-world deployment. Prior approaches typically focus on a small set of exercises and fail to generalize across diverse movements. In contrast, we develop a robust, multitask motion analysis model capable of performing exercise detection and repetition counting across hundreds of exercises, a scale far beyond previous methods. We overcome previous data limitations by assembling a large-scale fitness dataset, Olympia covering more than 1,900 exercises. To our knowledge, our vision-language model is the first that can perform multiple tasks on skeletal fitness data. On Olympia, our model can detect exercises with 76.5% accuracy and count repetitions with 85.3% off-by-one accuracy, using only RGB video. By presenting a single vision-language transformer model for both exercise identification and rep counting, we take a significant step toward democratizing AI-powered fitness tracking.