Dual-Imbalance Continual Learning for Real-World Food Recognition

TL;DR

This paper introduces DIME, a novel continual learning framework for real-world food recognition that addresses dual imbalance in data and class increments, improving accuracy and efficiency.

Contribution

DIME employs parameter-efficient fine-tuning and a class-count guided spectral merging strategy to handle dual imbalance in continual food recognition tasks.

Findings

DIME improves accuracy by over 3% compared to existing methods.

The framework effectively manages long-tailed class distributions.

Efficient deployment is achieved with a single merged adapter.

Abstract

Visual food recognition in real-world dietary logging scenarios naturally exhibits severe data imbalance, where a small number of food categories appear frequently while many others occur rarely, resulting in long-tailed class distributions. In practice, food recognition systems often operate in a continual learning setting, where new categories are introduced sequentially over time. However, existing studies typically assume that each incremental step introduces a similar number of new food classes, which rarely happens in real world where the number of newly observed categories can vary significantly across steps, leading to highly uneven learning dynamics. As a result, continual food recognition exhibits a dual imbalance: imbalanced samples within each food class and imbalanced numbers of new food classes to learn at each incremental learning step. In this work, we introduce DIME, a Dual-Imbalance-aware Adapter Merging framework for continual food recognition. DIME learns lightweight adapters for each task using parameter-efficient fine-tuning and progressively integrates them through a class-count guided spectral merging strategy. A rank-wise threshold modulation mechanism further stabilizes the merging process by preserving dominant knowledge while allowing adaptive updates. The resulting model maintains a single merged adapter for inference, enabling efficient deployment without accumulating task-specific modules. Experiments on realistic long-tailed food benchmarks under our step-imbalanced setup show that the proposed method consistently improves by more than 3% over the strongest existing continual learning baselines. Code is available at https://github.com/xiaoyanzhang1/DIME.