From Captions to Visual Concepts and Back
Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, and Li Deng, Piotr Doll\'ar, Jianfeng Gao, Xiaodong He, Margaret, Mitchell, John C. Platt, C. Lawrence Zitnick, Geoffrey Zweig

TL;DR
This paper introduces a comprehensive system that automatically generates image descriptions by learning visual detectors, language models, and multimodal similarity models from large datasets, achieving state-of-the-art results on COCO.
Contribution
It presents a novel integrated approach combining multiple instance learning, language modeling, and multimodal re-ranking for image captioning, outperforming previous methods.
Findings
Achieved a BLEU-4 score of 29.1% on COCO benchmark.
System captions are equal or better than human captions 34% of the time.
Introduced a multimodal similarity model for improved caption ranking.
Abstract
This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
