Face, Whole-Person, and Object Classification in a Unified Space Via The Interleaved Multi-Domain Identity Curriculum

TL;DR

This paper introduces the Interleaved Multi-Domain Identity Curriculum (IMIC), a training method enabling foundation models to perform object, face, and person recognition tasks simultaneously in a shared embedding space without catastrophic forgetting.

Contribution

The paper presents IMIC, a novel interleaved training schedule that allows fine-tuning foundation models on multiple recognition tasks concurrently, maintaining generalization and outperforming prior methods.

Findings

IMIC enables multi-task recognition in a single embedding space.

EVA-02 and CLIP models achieved human-level multi-task performance.

The approach preserves out-of-distribution generalization.

Abstract

Vision foundation models can perform generalized object classification in zero-shot mode, and face/person recognition when they are fine-tuned. However, fine-tuned models suffer from catastrophic forgetting. We create models that perform four tasks (object recognition, face recognition from high- and low-quality images, and person recognition from whole-body images) in a single embedding space -- without incurring substantial catastrophic forgetting. To accomplish this, we introduce two variants of the Interleaved Multi-Domain Identity Curriculum (IMIC): a gradient-coupled, interleaving training schedule that fine-tunes a foundation backbone simultaneously on all four tasks. The IMIC method proved effective with three foundation model bases: DINOv3, CLIP, and EVA-02. Two of these (EVA-02 and CLIP) performed comparably with domain experts on all four tasks concurrently and were more accurate than humans at multi-tasking across face, body, and object datasets. Further, we demonstrate that our approach does not substantially harm out-of-distribution generalization, thus maintaining a key property of foundation models. Analysis of the most accurate model variants (EVA-02 + IMIC A and B) showed linearly separable representations of the four tasks in the unified embedding space, but with substantial sharing of features across tasks. Fewer than 100 PCs calculated from any one task could perform all other tasks with nearly zero performance degradation.