Bayesian Optimization for Controlled Image Editing via LLMs

TL;DR

BayesGenie leverages Bayesian Optimization with Large Language Models to enable precise, user-friendly image editing via natural language, without fine-tuning, outperforming existing methods in accuracy and semantic preservation.

Contribution

The paper introduces BayesGenie, a model-agnostic framework combining LLMs and Bayesian Optimization for accurate, fine-tuning-free image editing from natural language descriptions.

Findings

Outperforms existing methods in editing accuracy

Maintains semantic integrity of images

Works effectively across various LLMs

Abstract

In the rapidly evolving field of image generation, achieving precise control over generated content and maintaining semantic consistency remain significant limitations, particularly concerning grounding techniques and the necessity for model fine-tuning. To address these challenges, we propose BayesGenie, an off-the-shelf approach that integrates Large Language Models (LLMs) with Bayesian Optimization to facilitate precise and user-friendly image editing. Our method enables users to modify images through natural language descriptions without manual area marking, while preserving the original image's semantic integrity. Unlike existing techniques that require extensive pre-training or fine-tuning, our approach demonstrates remarkable adaptability across various LLMs through its model-agnostic design. BayesGenie employs an adapted Bayesian optimization strategy to automatically refine the inference process parameters, achieving high-precision image editing with minimal user intervention. Through extensive experiments across diverse scenarios, we demonstrate that our framework significantly outperforms existing methods in both editing accuracy and semantic preservation, as validated using different LLMs including Claude3 and GPT-4.