Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning

TL;DR

This paper introduces a novel continual learning framework for multimodal food analysis that effectively mitigates catastrophic forgetting using Dual-LoRA adapters and a quality-enhanced pseudo replay strategy, enabling models to learn new food tasks without retraining from scratch.

Contribution

The paper presents a new continual learning approach combining Dual-LoRA architecture with quality-enhanced pseudo replay for multimodal food analysis, addressing catastrophic forgetting in large models.

Findings

Superior performance in mitigating forgetting on Uni-Food dataset

First effective continual learning method for complex food tasks

Enhanced reliability of generated replay samples

Abstract

Food analysis has become increasingly critical for health-related tasks such as personalized nutrition and chronic disease prevention. However, existing large multimodal models (LMMs) in food analysis suffer from catastrophic forgetting when learning new tasks, requiring costly retraining from scratch. To address this, we propose a novel continual learning framework for multimodal food learning, integrating a Dual-LoRA architecture with Quality-Enhanced Pseudo Replay. We introduce two complementary low-rank adapters for each task: a specialized LoRA that learns task-specific knowledge with orthogonal constraints to previous tasks' subspaces, and a cooperative LoRA that consolidates shared knowledge across tasks via pseudo replay. To improve the reliability of replay data, our Quality-Enhanced Pseudo Replay strategy leverages self-consistency and semantic similarity to reduce hallucinations in generated samples. Experiments on the comprehensive Uni-Food dataset show superior performance in mitigating forgetting, representing the first effective continual learning approach for complex food tasks.