Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks

TL;DR

This paper introduces VDPO, a novel framework that uses large language models to generate prompts from visual inputs, significantly improving multimodal image synthesis quality across various benchmarks.

Contribution

The paper presents a new vision-driven prompt optimization framework that integrates visual and textual modules to enhance image generation performance.

Findings

VDPO outperforms existing methods on COCO and Sketchy benchmarks.

Achieves significant improvements in FID, LPIPS, BLEU, and CIDEr scores.

Demonstrates robustness, scalability, and generalization in diverse tasks.

Abstract

Vision generation remains a challenging frontier in artificial intelligence, requiring seamless integration of visual understanding and generative capabilities. In this paper, we propose a novel framework, Vision-Driven Prompt Optimization (VDPO), that leverages Large Language Models (LLMs) to dynamically generate textual prompts from visual inputs, guiding high-fidelity image synthesis. VDPO combines a visual embedding prompt tuner, a textual instruction generator, and a vision generation module to achieve state-of-the-art performance in diverse vision generation tasks. Extensive experiments on benchmarks such as COCO and Sketchy demonstrate that VDPO consistently outperforms existing methods, achieving significant improvements in FID, LPIPS, and BLEU/CIDEr scores. Additional analyses reveal the scalability, robustness, and generalization capabilities of VDPO, making it a versatile solution for in-domain and out-of-domain tasks. Human evaluations further validate the practical superiority of VDPO in generating visually appealing and semantically coherent outputs.