Improving Image Captioning with Better Use of Captions
Zhan Shi, Xu Zhou, Xipeng Qiu, Xiaodan Zhu

TL;DR
This paper introduces a novel image captioning architecture that leverages caption-guided visual relationship graphs and multi-task learning to improve image understanding and caption quality, achieving state-of-the-art results on MSCOCO.
Contribution
The paper presents a new architecture that better exploits caption semantics through visual relationship graphs and multi-task learning, enhancing both image representation and caption generation.
Findings
Significant performance improvement over baselines.
Achieved state-of-the-art results on MSCOCO.
Effective use of caption-guided visual relationship graphs.
Abstract
Image captioning is a multimodal problem that has drawn extensive attention in both the natural language processing and computer vision community. In this paper, we present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. The representation is then enhanced with neighbouring and contextual nodes with their textual and visual features. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences. We perform extensive experiments on the MSCOCO dataset, showing that the proposed framework significantly outperforms the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Video Analysis and Summarization · Human Pose and Action Recognition
