# Learning Visual Actions Using Multiple Verb-Only Labels

## TL;DR

This paper proposes a multi-verb label approach for visual action recognition and retrieval in videos, enabling larger verb vocabularies and better handling semantic ambiguities compared to traditional single-verb labels.

## Contribution

It introduces a novel multi-verb annotation and learning method that improves action recognition and retrieval performance over conventional single-verb labeling.

## Key findings

- Multi-verb representations outperform single-verb labels in recognition tasks.
- The approach enables cross-dataset retrieval and better semantic understanding.
- Multi-label verb-only models handle contextual overlaps effectively.

## Abstract

This work introduces verb-only representations for both recognition and retrieval of visual actions, in video. Current methods neglect legitimate semantic ambiguities between verbs, instead choosing unambiguous subsets of verbs along with objects to disambiguate the actions. We instead propose multiple verb-only labels, which we learn through hard or soft assignment as a regression. This enables learning a much larger vocabulary of verbs, including contextual overlaps of these verbs. We collect multi-verb annotations for three action video datasets and evaluate the verb-only labelling representations for action recognition and cross-modal retrieval (video-to-text and text-to-video). We demonstrate that multi-label verb-only representations outperform conventional single verb labels. We also explore other benefits of a multi-verb representation including cross-dataset retrieval and verb type manner and result verb types) retrieval.