Boosting Video Captioning with Dynamic Loss Network

TL;DR

This paper introduces a dynamic loss network for video captioning that directly optimizes evaluation metrics, leading to improved performance over existing methods on standard datasets.

Contribution

The paper proposes a novel dynamic loss network that provides direct feedback based on evaluation metrics, enhancing video captioning accuracy.

Findings

Outperforms previous methods on MSVD and MSRVTT datasets.

Provides more efficient optimization compared to reinforcement learning approaches.

Easily adaptable to similar vision-language tasks.

Abstract

Video captioning is one of the challenging problems at the intersection of vision and language, having many real-life applications in video retrieval, video surveillance, assisting visually challenged people, Human-machine interface, and many more. Recent deep learning based methods have shown promising results but are still on the lower side than other vision tasks (such as image classification, object detection). A significant drawback with existing video captioning methods is that they are optimized over cross-entropy loss function, which is uncorrelated to the de facto evaluation metrics (BLEU, METEOR, CIDER, ROUGE). In other words, cross-entropy is not a proper surrogate of the true loss function for video captioning. To mitigate this, methods like REINFORCE, Actor-Critic, and Minimum Risk Training (MRT) have been applied but have limitations and are not very effective. This paper proposes an alternate solution by introducing a dynamic loss network (DLN), providing an additional feedback signal that reflects the evaluation metrics directly. Our solution proves to be more efficient than other solutions and can be easily adapted to similar tasks. Our results on Microsoft Research Video Description Corpus (MSVD) and MSR-Video to Text (MSRVTT) datasets outperform previous methods.