DirMixE: Harnessing Test Agnostic Long-tail Recognition with Hierarchical Label Vartiations

TL;DR

This paper introduces DirMixE, a hierarchical mixture-of-experts approach for test-agnostic long-tail recognition, effectively capturing global and local label distribution variations to improve model robustness and performance.

Contribution

We propose DirMixE, a novel MoE strategy using Dirichlet meta-distributions to model local and global variations, along with a Latent Skill Finetuning framework for efficient foundation model adaptation.

Findings

DirMixE outperforms existing methods on long-tail datasets.

The hierarchical Dirichlet approach captures diverse test distributions.

Theoretical bounds support the effectiveness of variance regularization.

Abstract

This paper explores test-agnostic long-tail recognition, a challenging long-tail task where the test label distributions are unknown and arbitrarily imbalanced. We argue that the variation in these distributions can be broken down hierarchically into global and local levels. The global ones reflect a broad range of diversity, while the local ones typically arise from milder changes, often focused on a particular neighbor. Traditional methods predominantly use a Mixture-of-Expert (MoE) approach, targeting a few fixed test label distributions that exhibit substantial global variations. However, the local variations are left unconsidered. To address this issue, we propose a new MoE strategy, DirMixE, which assigns experts to different Dirichlet meta-distributions of the label distribution, each targeting a specific aspect of local variations. Additionally, the diversity among these Dirichlet meta-distributions inherently captures global variations. This dual-level approach also leads to a more stable objective function, allowing us to sample different test distributions better to quantify the mean and variance of performance outcomes. Building on this idea, we develop a general Latent Skill Finetuning (LSF) framework for parameter-efficient finetuning of foundation models. We provide implementations based on LoRA and Adapter. Theoretically, we derive upper bounds on the generalization error for both standard learning and PEFT. Under mild assumptions, we show that the variance-based regularization helps tighten these bounds. Furthermore, we prove that the covering number of the PEFT hypothesis class scales with the number of trainable parameters. Finally, extensive experiments on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist validate the effectiveness of DirMixE.