On the effectiveness of task granularity for transfer learning

TL;DR

This paper investigates how task granularity in training affects the quality of features learned for transfer learning in video classification and captioning, demonstrating that finer granularity improves transfer performance.

Contribution

It introduces a DNN framework trained on multiple granularity levels and shows that finer task granularity enhances transfer learning effectiveness.

Findings

Finer task granularity leads to better transfer learning features.

The model outperforms existing baselines on Something-Something classification.

Provides a new strong baseline for captioning on the dataset.

Abstract

We describe a DNN for video classification and captioning, trained end-to-end, with shared features, to solve tasks at different levels of granularity, exploring the link between granularity in a source task and the quality of learned features for transfer learning. For solving the new task domain in transfer learning, we freeze the trained encoder and fine-tune a neural net on the target domain. We train on the Something-Something dataset with over 220, 000 videos, and multiple levels of target granularity, including 50 action groups, 174 fine-grained action categories and captions. Classification and captioning with Something-Something are challenging because of the subtle differences between actions, applied to thousands of different object classes, and the diversity of captions penned by crowd actors. Our model performs better than existing classification baselines for SomethingSomething, with impressive fine-grained results. And it yields a strong baseline on the new Something-Something captioning task. Experiments reveal that training with more fine-grained tasks tends to produce better features for transfer learning.