Cycle-Consistency Learning for Captioning and Grounding

TL;DR

This paper introduces CyCo, a cyclic-consistent learning framework that unifies visual grounding and image captioning, enhancing training efficiency and performance through mutual reinforcement and semi-weak supervision.

Contribution

The paper proposes a novel cyclic-consistent framework that bridges visual grounding and captioning, enabling semi-weak supervision and improved performance in both tasks.

Findings

State-of-the-art fully supervised grounding performance

Competitive semi-weakly supervised grounding results

Impressive captioning performance on benchmark datasets

Abstract

We present that visual grounding and image captioning, which perform as two mutually inverse processes, can be bridged together for collaborative training by careful designs. By consolidating this idea, we introduce CyCo, a cyclic-consistent learning framework to ameliorate the independent training pipelines of visual grounding and image captioning. The proposed framework (1) allows the semi-weakly supervised training of visual grounding; (2) improves the performance of fully supervised visual grounding; (3) yields a general captioning model that can describe arbitrary image regions. Extensive experiments show that our fully supervised grounding model achieves state-of-the-art performance, and the semi-weakly supervised one also exhibits competitive performance compared to the fully supervised counterparts. Our image captioning model has the capability to freely describe image regions and meanwhile shows impressive performance on prevalent captioning benchmarks.