FaceLLM: A Multimodal Large Language Model for Face Understanding

TL;DR

FaceLLM is a specialized multimodal large language model trained on a novel face-focused dataset, significantly improving performance on facial understanding tasks by leveraging synthetic supervision from language models.

Contribution

This work introduces FaceLLM and a new dataset, FairFaceGPT, enabling domain-specific facial understanding with weakly supervised question-answer pairs.

Findings

FaceLLM achieves state-of-the-art results on face-centric tasks.

The weakly supervised pipeline effectively generates high-quality training data.

Synthetic supervision enhances domain-specific multimodal model performance.

Abstract

Multimodal large language models (MLLMs) have shown remarkable performance in vision-language tasks. However, existing MLLMs are primarily trained on generic datasets, limiting their ability to reason on domain-specific visual cues such as those in facial images. In particular, tasks that require detailed understanding of facial structure, expression, emotion, and demographic features remain underexplored by MLLMs due to the lack of large-scale annotated face image-text datasets. In this work, we introduce FaceLLM, a multimodal large language model trained specifically for facial image understanding. To construct the training data, we propose a novel weakly supervised pipeline that uses ChatGPT with attribute-aware prompts to generate high-quality question-answer pairs based on images from the FairFace dataset. The resulting corpus, called FairFaceGPT, covers a diverse set of attributes including expression, pose, skin texture, and forensic information. Our experiments demonstrate that FaceLLM improves the performance of MLLMs on various face-centric tasks and achieves state-of-the-art performance. This work highlights the potential of synthetic supervision via language models for building domain-specialized MLLMs, and sets a precedent for trustworthy, human-centric multimodal AI systems. FairFaceGPT dataset and pretrained FaceLLM models are publicly available in the project page.