Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition

TL;DR

This paper introduces DQRoute, a modular framework for long-tailed recognition that combines difficulty-aware training with dynamic expert collaboration, improving performance on rare and hard classes.

Contribution

It proposes a novel difficulty-aware optimization method with a mixture-of-experts architecture and input-adaptive expert routing, trained end-to-end.

Findings

Significant performance gains on long-tailed benchmarks.

Improved accuracy on rare and difficult classes.

Effective input-adaptive expert routing without a centralized router.

Abstract

Long-tailed visual recognition is challenging not only due to class imbalance but also because of varying classification difficulty across categories. Simply reweighting classes by frequency often overlooks those that are intrinsically hard to learn. To address this, we propose \textbf{DQRoute}, a modular framework that combines difficulty-aware optimization with dynamic expert collaboration. DQRoute first estimates class-wise difficulty based on prediction uncertainty and historical performance, and uses this signal to guide training with adaptive loss weighting. On the architectural side, DQRoute employs a mixture-of-experts design, where each expert specializes in a different region of the class distribution. At inference time, expert predictions are weighted by confidence scores derived from expert-specific OOD detectors, enabling input-adaptive routing without the need for a centralized router. All components are trained jointly in an end-to-end manner. Experiments on standard long-tailed benchmarks demonstrate that DQRoute significantly improves performance, particularly on rare and difficult classes, highlighting the benefit of integrating difficulty modeling with decentralized expert routing.