VATEX Captioning Challenge 2019: Multi-modal Information Fusion and Multi-stage Training Strategy for Video Captioning

TL;DR

This paper presents a multi-modal video captioning system that combines appearance, motion, and audio information with a multi-stage training strategy, achieving significant improvements on the VATEX benchmark for both Chinese and English captioning.

Contribution

The work introduces a novel multi-modal fusion approach and a multi-stage training process specifically tailored for video captioning tasks.

Findings

Significant performance improvements on VATEX benchmark

Effective multi-modal information fusion strategy

Successful application to both Chinese and English captioning

Abstract

Multi-modal information is essential to describe what has happened in a video. In this work, we represent videos by various appearance, motion and audio information guided with video topic. By following multi-stage training strategy, our experiments show steady and significant improvement on the VATEX benchmark. This report presents an overview and comparative analysis of our system designed for both Chinese and English tracks on VATEX Captioning Challenge 2019.