Banana100: Breaking NR-IQA Metrics by 100 Iterative Image Replications with Nano Banana Pro

TL;DR

This paper introduces Banana100, a dataset exposing the failure of current NR-IQA metrics to detect quality degradation in images after iterative multi-turn editing, highlighting a critical weakness in current image editing systems.

Contribution

The paper presents Banana100, a large dataset of 28,000 images degraded through 100 iterative edits, revealing the inability of existing NR-IQA metrics to detect quality loss.

Findings

Existing NR-IQA metrics fail to detect degradation in iteratively edited images.

Iterative editing leads to rapid accumulation of artifacts and noise.

Current evaluation tools may not ensure the safety and robustness of multi-modal systems.

Abstract

The multi-step, iterative image editing capabilities of multi-modal agentic systems have transformed digital content creation. Although latest image editing models faithfully follow instructions and generate high-quality images in single-turn edits, we identify a critical weakness in multi-turn editing, which is the iterative degradation of image quality. As images are repeatedly edited, minor artifacts accumulate, rapidly leading to a severe accumulation of visible noise and a failure to follow simple editing instructions. To systematically study these failures, we introduce Banana100, a comprehensive dataset of 28,000 degraded images generated through 100 iterative editing steps, including diverse textures and image content. Alarmingly, image quality evaluators fail to detect the degradation. Among 21 popular no-reference image quality assessment (NR-IQA) metrics, none of them consistently assign lower scores to heavily degraded images than to clean ones. The dual failures of generators and evaluators may threaten the stability of future model training and the safety of deployed agentic systems, if the low-quality synthetic data generated by multi-turn edits escape quality filters. We release the full code and data to facilitate the development of more robust models, helping to mitigate the fragility of multi-modal agentic systems.