Empowering Visual Creativity: A Vision-Language Assistant to Image Editing Recommendations

TL;DR

This paper introduces a new task called Image Editing Recommendation (IER) that automatically generates diverse, creative editing instructions from an image and a vague user prompt, supported by a multimodal framework called Creativity-VLA.

Contribution

The paper proposes the IER task, creates a dedicated dataset, and develops Creativity-VLA with a novel token-for-localization mechanism for improved edit-instruction generation.

Findings

Creativity-VLA effectively generates relevant, creative editing instructions.

The model outperforms baselines in relevance and diversity.

The token-for-localization mechanism enhances local editing support.

Abstract

Advances in text-based image generation and editing have revolutionized content creation, enabling users to create impressive content from imaginative text prompts. However, existing methods are not designed to work well with the oversimplified prompts that are often encountered in typical scenarios when users start their editing with only vague or abstract purposes in mind. Those scenarios demand elaborate ideation efforts from the users to bridge the gap between such vague starting points and the detailed creative ideas needed to depict the desired results. In this paper, we introduce the task of Image Editing Recommendation (IER). This task aims to automatically generate diverse creative editing instructions from an input image and a simple prompt representing the users' under-specified editing purpose. To this end, we introduce Creativity-Vision Language Assistant~(Creativity-VLA), a multimodal framework designed specifically for edit-instruction generation. We train Creativity-VLA on our edit-instruction dataset specifically curated for IER. We further enhance our model with a novel 'token-for-localization' mechanism, enabling it to support both global and local editing operations. Our experimental results demonstrate the effectiveness of \ours{} in suggesting instructions that not only contain engaging creative elements but also maintain high relevance to both the input image and the user's initial hint.