Self-supervised visual learning from interactions with objects

TL;DR

This paper proposes a method to enhance self-supervised visual learning by incorporating object-related actions, leading to improved object category recognition through better viewpoint alignment.

Contribution

It introduces a novel loss function that aligns action and visual embeddings, leveraging embodied interactions to structure visual representations in SSL.

Findings

Outperforms previous SSL methods on category recognition tasks.

Improves viewpoint-wise alignment of objects within the same category.

Embodied actions contribute to more robust visual representations.

Abstract

Self-supervised learning (SSL) has revolutionized visual representation learning, but has not achieved the robustness of human vision. A reason for this could be that SSL does not leverage all the data available to humans during learning. When learning about an object, humans often purposefully turn or move around objects and research suggests that these interactions can substantially enhance their learning. Here we explore whether such object-related actions can boost SSL. For this, we extract the actions performed to change from one ego-centric view of an object to another in four video datasets. We then introduce a new loss function to learn visual and action embeddings by aligning the performed action with the representations of two images extracted from the same clip. This permits the performed actions to structure the latent visual representation. Our experiments show that our method consistently outperforms previous methods on downstream category recognition. In our analysis, we find that the observed improvement is associated with a better viewpoint-wise alignment of different objects from the same category. Overall, our work demonstrates that embodied interactions with objects can improve SSL of object categories.