Learning Action Concept Trees and Semantic Alignment Networks from Image-Description Data

TL;DR

This paper introduces a novel approach to action classification in images using weak labels from image descriptions, leveraging an Action Concept Tree and Semantic Alignment model to improve flexibility and performance.

Contribution

The paper proposes a two-stage learning method utilizing image-description data to learn action categories with an Action Concept Tree and Semantic Alignment network, enabling vocabulary expansion.

Findings

Outperforms baseline methods significantly

Introduces a new dataset for action learning from descriptions

Demonstrates effective learning from weak labels

Abstract

Action classification in still images has been a popular research topic in computer vision. Labelling large scale datasets for action classification requires tremendous manual work, which is hard to scale up. Besides, the action categories in such datasets are pre-defined and vocabularies are fixed. However humans may describe the same action with different phrases, which leads to the difficulty of vocabulary expansion for traditional fully-supervised methods. We observe that large amounts of images with sentence descriptions are readily available on the Internet. The sentence descriptions can be regarded as weak labels for the images, which contain rich information and could be used to learn flexible expressions of action categories. We propose a method to learn an Action Concept Tree (ACT) and an Action Semantic Alignment (ASA) model for classification from image-description data via a two-stage learning process. A new dataset for the task of learning actions from descriptions is built. Experimental results show that our method outperforms several baseline methods significantly.