SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation

TL;DR

SkillFormer is a novel, efficient multi-view video architecture that accurately estimates human skill levels by fusing egocentric and exocentric views, achieving state-of-the-art results with reduced training costs.

Contribution

It introduces a unified multi-view proficiency estimation model with a CrossViewFusion module and Low-Rank Adaptation for efficient fine-tuning, outperforming prior methods.

Findings

Achieves state-of-the-art accuracy on EgoExo4D dataset.

Uses 4.5x fewer parameters than previous models.

Requires 3.75x fewer training epochs.

Abstract

Assessing human skill levels in complex activities is a challenging problem with applications in sports, rehabilitation, and training. In this work, we present SkillFormer, a parameter-efficient architecture for unified multi-view proficiency estimation from egocentric and exocentric videos. Building on the TimeSformer backbone, SkillFormer introduces a CrossViewFusion module that fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration. We leverage Low-Rank Adaptation to fine-tune only a small subset of parameters, significantly reducing training costs. In fact, when evaluated on the EgoExo4D dataset, SkillFormer achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency, using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines. It excels in multiple structured tasks, confirming the value of multi-view integration for fine-grained skill assessment. Project page at https://edowhite.github.io/SkillFormer