Multi-modal Feature Fusion with Feature Attention for VATEX Captioning Challenge 2020

TL;DR

This paper presents a multi-modal feature fusion approach with feature attention for video captioning, combining diverse features and ensemble decoders to significantly improve performance on the VATEX Challenge 2020.

Contribution

The study introduces a novel feature attention module and employs ensemble decoding strategies to enhance multi-modal video captioning performance.

Findings

Achieved 76.0 CIDEr score on English test set

Ranked 2nd on VATEX Challenge 2020 leaderboard

Outperformed baseline models significantly

Abstract

This report describes our model for VATEX Captioning Challenge 2020. First, to gather information from multiple domains, we extract motion, appearance, semantic and audio features. Then we design a feature attention module to attend on different feature when decoding. We apply two types of decoders, top-down and X-LAN and ensemble these models to get the final result. The proposed method outperforms official baseline with a significant gap. We achieve 76.0 CIDEr and 50.0 CIDEr on English and Chinese private test set. We rank 2nd on both English and Chinese private test leaderboard.