SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents

TL;DR

SmartAvatar introduces an AI-driven framework that uses vision-language models and an iterative verification process to generate high-quality, customizable 3D human avatars from images or text prompts, with fine control and animation capabilities.

Contribution

It presents a novel autonomous verification loop leveraging VLMs for precise, iterative refinement of avatars, enhancing control, quality, and customization over existing diffusion-based methods.

Findings

Outperforms recent systems in mesh quality and identity fidelity.

Enables fine-grained, iterative avatar customization via natural language.

Supports pose manipulation with consistent identity and appearance.

Abstract

SmartAvatar is a vision-language-agent-driven framework for generating fully rigged, animation-ready 3D human avatars from a single photo or textual prompt. While diffusion-based methods have made progress in general 3D object generation, they continue to struggle with precise control over human identity, body shape, and animation readiness. In contrast, SmartAvatar leverages the commonsense reasoning capabilities of large vision-language models (VLMs) in combination with off-the-shelf parametric human generators to deliver high-quality, customizable avatars. A key innovation is an autonomous verification loop, where the agent renders draft avatars, evaluates facial similarity, anatomical plausibility, and prompt alignment, and iteratively adjusts generation parameters for convergence. This interactive, AI-guided refinement process promotes fine-grained control over both facial and body features, enabling users to iteratively refine their avatars via natural-language conversations. Unlike diffusion models that rely on static pre-trained datasets and offer limited flexibility, SmartAvatar brings users into the modeling loop and ensures continuous improvement through an LLM-driven procedural generation and verification system. The generated avatars are fully rigged and support pose manipulation with consistent identity and appearance, making them suitable for downstream animation and interactive applications. Quantitative benchmarks and user studies demonstrate that SmartAvatar outperforms recent text- and image-driven avatar generation systems in terms of reconstructed mesh quality, identity fidelity, attribute accuracy, and animation readiness, making it a versatile tool for realistic, customizable avatar creation on consumer-grade hardware.