Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge

TL;DR

This paper introduces BLPO, a bi-level prompt optimization framework that enhances multimodal LLM-based evaluation of AI-generated images by converting visual data into text, overcoming context limitations and improving alignment with human judgments.

Contribution

We propose a novel bi-level prompt optimization method that jointly refines judge and image-to-text prompts for better multimodal evaluation under context constraints.

Findings

BLPO improves evaluation accuracy across multiple datasets.

The method effectively converts images into textual cues.

BLPO outperforms existing prompt optimization techniques.

Abstract

Large language models (LLMs) have become widely adopted as automated judges for evaluating AI-generated content. Despite their success, aligning LLM-based evaluations with human judgments remains challenging. While supervised fine-tuning on human-labeled data can improve alignment, it is costly and inflexible, requiring new training for each task or dataset. Recent progress in auto prompt optimization (APO) offers a more efficient alternative by automatically improving the instructions that guide LLM judges. However, existing APO methods primarily target text-only evaluations and remain underexplored in multimodal settings. In this work, we study auto prompt optimization for multimodal LLM-as-a-judge, particularly for evaluating AI-generated images. We identify a key bottleneck: multimodal models can only process a limited number of visual examples due to context window constraints, which hinders effective trial-and-error prompt refinement. To overcome this, we propose BLPO, a bi-level prompt optimization framework that converts images into textual representations while preserving evaluation-relevant visual cues. Our bi-level optimization approach jointly refines the judge prompt and the I2T prompt to maintain fidelity under limited context budgets. Experiments on four datasets and three LLM judges demonstrate the effectiveness of our method.