Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning

TL;DR

This paper introduces WeGO, a novel iterative learning approach that leverages high-confidence predictions in one modality to guide coherence modeling in another, effectively improving cross-modal coherence without requiring labeled coherence data.

Contribution

The paper proposes WeGO, a new method for cross-modal coherence modeling that uses weak guidance from high-confidence predictions and iterative joint optimization, bypassing the need for labeled coherence data.

Findings

Outperforms existing cross-modal coherence methods on two datasets.

Effective ablation results validate key modules of the proposed approach.

Iterative boosting enhances coherence prediction accuracy.

Abstract

Cross-modal coherence modeling is essential for intelligent systems to help them organize and structure information, thereby understanding and creating content of the physical world coherently like human-beings. Previous work on cross-modal coherence modeling attempted to leverage the order information from another modality to assist the coherence recovering of the target modality. Despite of the effectiveness, labeled associated coherency information is not always available and might be costly to acquire, making the cross-modal guidance hard to leverage. To tackle this challenge, this paper explores a new way to take advantage of cross-modal guidance without gold labels on coherency, and proposes the Weak Cross-Modal Guided Ordering (WeGO) model. More specifically, it leverages high-confidence predicted pairwise order in one modality as reference information to guide the coherence modeling in another. An iterative learning paradigm is further designed to jointly optimize the coherence modeling in two modalities with selected guidance from each other. The iterative cross-modal boosting also functions in inference to further enhance coherence prediction in each modality. Experimental results on two public datasets have demonstrated that the proposed method outperforms existing methods for cross-modal coherence modeling tasks. Major technical modules have been evaluated effective through ablation studies. Codes are available at: \url{https://github.com/scvready123/IterWeGO}.