Learning an Ensemble Token from Task-driven Priors in Facial Analysis

TL;DR

This paper introduces KT-Adapter, a method that learns a knowledge token within self-attention to efficiently unify high-fidelity features for improved facial analysis performance.

Contribution

The paper proposes a novel knowledge token learning method that integrates high-quality features efficiently using self-attention, enhancing facial analysis tasks.

Findings

Improved facial analysis performance with statistically significant results.

Knowledge token approach achieves high efficiency with negligible computational cost.

Enhanced feature representations across multiple facial analysis benchmarks.

Abstract

Facial analysis exhibits task-specific feature variations. While Convolutional Neural Networks (CNNs) have enabled the fine-grained representation of spatial information, Vision Transformers (ViTs) have facilitated the representation of semantic information at the patch level. While advances in backbone architectures have improved over the past decade, combining high-fidelity models often incurs computational costs on feature representation perspective. In this work, we introduce KT-Adapter, a novel methodology for learning knowledge token which enables the integration of high-fidelity feature representation in computationally efficient manner. Specifically, we propose a robust prior unification learning method that generates a knowledge token within a self-attention mechanism, sharing the mutual information across the pre-trained encoders. This knowledge token approach offers high efficiency with negligible computational cost. Our results show improved performance across facial analysis, with statistically significant enhancements observed in the feature representations.