Fully-attentive and interpretable: vision and video vision transformers for pain detection

TL;DR

This paper introduces a fully-attentive vision transformer-based pipeline for automated pain detection from facial expressions, achieving state-of-the-art results and providing interpretable attention maps.

Contribution

It is the first to apply fully-attentive vision transformers to pain detection, demonstrating improved performance and interpretability over previous methods.

Findings

Models outperform earlier works in pain detection accuracy.

Attention maps provide reasonable interpretations of model predictions.

Hyperparameter analysis identified key configurations for optimal performance.

Abstract

Pain is a serious and costly issue globally, but to be treated, it must first be detected. Vision transformers are a top-performing architecture in computer vision, with little research on their use for pain detection. In this paper, we propose the first fully-attentive automated pain detection pipeline that achieves state-of-the-art performance on binary pain detection from facial expressions. The model is trained on the UNBC-McMaster dataset, after faces are 3D-registered and rotated to the canonical frontal view. In our experiments we identify important areas of the hyperparameter space and their interaction with vision and video vision transformers, obtaining 3 noteworthy models. We analyse the attention maps of one of our models, finding reasonable interpretations for its predictions. We also evaluate Mixup, an augmentation technique, and Sharpness-Aware Minimization, an optimizer, with no success. Our presented models, ViT-1 (F1 score 0.55 +- 0.15), ViViT-1 (F1 score 0.55 +- 0.13), and ViViT-2 (F1 score 0.49 +- 0.04), all outperform earlier works, showing the potential of vision transformers for pain detection. Code is available at https://github.com/IPDTFE/ViT-McMaster