Fashion Matrix: Editing Photos by Just Talking

TL;DR

Fashion Matrix is a hierarchical AI system that enables intuitive photo editing in fashion through natural language prompts, integrating large language models with semantic segmentation and visual foundation models for diverse editing tasks.

Contribution

The paper introduces Fashion Matrix, a novel AI framework that combines LLMs, semantic segmentation, and visual models for flexible, prompt-driven fashion photo editing.

Findings

Effective in garment and accessory editing tasks

Supports diverse editing operations like recoloring and removal

Demonstrates strong collaborative potential of pre-trained models

Abstract

The utilization of Large Language Models (LLMs) for the construction of AI systems has garnered significant attention across diverse fields. The extension of LLMs to the domain of fashion holds substantial commercial potential but also inherent challenges due to the intricate semantic interactions in fashion-related generation. To address this issue, we developed a hierarchical AI system called Fashion Matrix dedicated to editing photos by just talking. This system facilitates diverse prompt-driven tasks, encompassing garment or accessory replacement, recoloring, addition, and removal. Specifically, Fashion Matrix employs LLM as its foundational support and engages in iterative interactions with users. It employs a range of Semantic Segmentation Models (e.g., Grounded-SAM, MattingAnything, etc.) to delineate the specific editing masks based on user instructions. Subsequently, Visual Foundation Models (e.g., Stable Diffusion, ControlNet, etc.) are leveraged to generate edited images from text prompts and masks, thereby facilitating the automation of fashion editing processes. Experiments demonstrate the outstanding ability of Fashion Matrix to explores the collaborative potential of functionally diverse pre-trained models in the domain of fashion editing.