Facial Attractiveness Prediction in Live Streaming: A New Benchmark and Multi-modal Method

TL;DR

This paper introduces LiveBeauty, the largest live streaming facial attractiveness dataset, and proposes a multi-modal method that combines holistic and semantic features for improved attractiveness prediction in live scenarios.

Contribution

The paper presents the first large-scale live-specific FAP dataset and a novel multi-modal approach that enhances prediction accuracy in live streaming contexts.

Findings

Achieved state-of-the-art performance on LiveBeauty and other datasets.

Demonstrated the effectiveness of multi-modal feature integration.

Provided a new benchmark for live streaming facial attractiveness prediction.

Abstract

Facial attractiveness prediction (FAP) has long been an important computer vision task, which could be widely applied in live streaming for facial retouching, content recommendation, etc. However, previous FAP datasets are either small, closed-source, or lack diversity. Moreover, the corresponding FAP models exhibit limited generalization and adaptation ability. To overcome these limitations, in this paper we present LiveBeauty, the first large-scale live-specific FAP dataset, in a more challenging application scenario, i.e., live streaming. 10,000 face images are collected from a live streaming platform directly, with 200,000 corresponding attractiveness annotations obtained from a well-devised subjective experiment, making LiveBeauty the largest open-access FAP dataset in the challenging live scenario. Furthermore, a multi-modal FAP method is proposed to measure the facial attractiveness in live streaming. Specifically, we first extract holistic facial prior knowledge and multi-modal aesthetic semantic features via a Personalized Attractiveness Prior Module (PAPM) and a Multi-modal Attractiveness Encoder Module (MAEM), respectively, then integrate the extracted features through a Cross-Modal Fusion Module (CMFM). Extensive experiments conducted on both LiveBeauty and other open-source FAP datasets demonstrate that our proposed method achieves state-of-the-art performance. Dataset will be available soon.