TalkFashion: Intelligent Virtual Try-On Assistant Based on Multimodal Large Language Model

TL;DR

TalkFashion is a versatile virtual try-on system that uses large language models to understand user instructions and perform full outfit changes or local edits automatically, improving flexibility and visual quality.

Contribution

The paper introduces TalkFashion, a novel multimodal large language model-based system that enables multifunctional, instruction-guided virtual try-on with automated local editing capabilities.

Findings

Outperforms existing methods in semantic consistency

Achieves higher visual quality in try-on tasks

Supports fully automated local editing without manual masks

Abstract

Virtual try-on has made significant progress in recent years. This paper addresses how to achieve multifunctional virtual try-on guided solely by text instructions, including full outfit change and local editing. Previous methods primarily relied on end-to-end networks to perform single try-on tasks, lacking versatility and flexibility. We propose TalkFashion, an intelligent try-on assistant that leverages the powerful comprehension capabilities of large language models to analyze user instructions and determine which task to execute, thereby activating different processing pipelines accordingly. Additionally, we introduce an instruction-based local repainting model that eliminates the need for users to manually provide masks. With the help of multi-modal models, this approach achieves fully automated local editings, enhancing the flexibility of editing tasks. The experimental results demonstrate better semantic consistency and visual quality compared to the current methods.