TIP and Polish: Text-Image-Prototype Guided Multi-Modal Generation via Commonality-Discrepancy Modeling and Refinement

TL;DR

The paper introduces TIPPo, a multi-modal generation framework that explicitly models commonality and discrepancy between text and images, improving thematic coherence and style consistency through dual alignment and contrastive learning.

Contribution

It proposes TIPPo, a novel multi-modal generation method with explicit input modeling, dual alignment attention, and discrepancy refinement, enhancing semantic and stylistic coherence.

Findings

TIPPo achieves superior performance in automatic evaluations.

The framework improves style consistency and semantic accuracy.

Unsupervised contrastive learning reduces representation collapse.

Abstract

Multi-modal generation struggles to ensure thematic coherence and style consistency. Semantically, existing methods suffer from cross-modal mismatch and lack explicit modeling of commonality and discrepancy. Methods that rely on fine-grained training fail to balance semantic precision with writing style consistency. These shortcomings lead to suboptimal generation quality. To tackle these issues, we propose \textbf{\textit{TIPPo}}, a simple yet effective framework with explicit input modeling and comprehensive optimization objectives. It extracts the input text and images via multi-modal encoder and adapters, then measures the visual prototype. \textbf{T}extual, \textbf{I}mage, and \textbf{P}rototype signals are then fed to our proposed Dual Alignment Attention and Difference Operator modules before language model decoding. The proposed \textbf{Po}lishPPO reinforces the style consistency, while the unsupervised contrastive learning during SFT mitigates inter-sample representation collapse. Experimental results demonstrate the promising performance of \textbf{\textit{TIPPo}} in automatic evaluation and LLM-based criteria for creativity and semantic consistency.