How Effective are Self-Supervised Models for Contact Identification in Videos

TL;DR

This paper evaluates the effectiveness of self-supervised learning models in identifying physical contact in videos, using multiple CNN-based SSL models across diverse datasets to assess their robustness and applicability.

Contribution

It systematically assesses eight CNN-based SSL models for contact detection in videos and explores their performance in downstream action recognition tasks.

Findings

SSL models can effectively identify physical contacts in videos.

Performance varies across different datasets and models.

SSL models show promise for complex dynamic visual understanding.

Abstract

The exploration of video content via Self-Supervised Learning (SSL) models has unveiled a dynamic field of study, emphasizing both the complex challenges and unique opportunities inherent in this area. Despite the growing body of research, the ability of SSL models to detect physical contacts in videos remains largely unexplored, particularly the effectiveness of methods such as downstream supervision with linear probing or full fine-tuning. This work aims to bridge this gap by employing eight different convolutional neural networks (CNNs) based video SSL models to identify instances of physical contact within video sequences specifically. The Something-Something v2 (SSv2) and Epic-Kitchen (EK-100) datasets were chosen for evaluating these approaches due to the promising results on UCF101 and HMDB51, coupled with their limited prior assessment on SSv2 and EK-100. Additionally, these datasets feature diverse environments and scenarios, essential for testing the robustness and accuracy of video-based models. This approach not only examines the effectiveness of each model in recognizing physical contacts but also explores the performance in the action recognition downstream task. By doing so, valuable insights into the adaptability of SSL models in interpreting complex, dynamic visual information are contributed.