MOST: MR reconstruction Optimization for multiple downStream Tasks via continual learning

TL;DR

MOST introduces a continual learning framework to optimize MR reconstruction for multiple downstream tasks simultaneously, reducing error propagation and domain gaps, and outperforming traditional methods.

Contribution

This work extends downstream task-oriented MR reconstruction optimization to multiple tasks using continual learning, addressing catastrophic forgetting and improving multi-task performance.

Findings

MOST outperforms baseline reconstruction networks without finetuning.

MOST surpasses naive finetuning and conventional continual learning methods.

The approach effectively mitigates catastrophic forgetting in multi-task MR reconstruction.

Abstract

Deep learning-based Magnetic Resonance (MR) reconstruction methods have focused on generating high-quality images but often overlook the impact on downstream tasks (e.g., segmentation) that utilize the reconstructed images. Cascading separately trained reconstruction network and downstream task network has been shown to introduce performance degradation due to error propagation and domain gaps between training datasets. To mitigate this issue, downstream task-oriented reconstruction optimization has been proposed for a single downstream task. Expanding this optimization to multi-task scenarios is not straightforward. In this work, we extended this optimization to sequentially introduced multiple downstream tasks and demonstrated that a single MR reconstruction network can be optimized for multiple downstream tasks by deploying continual learning (MOST). MOST integrated techniques from replay-based continual learning and image-guided loss to overcome catastrophic forgetting. Comparative experiments demonstrated that MOST outperformed a reconstruction network without finetuning, a reconstruction network with na\"ive finetuning, and conventional continual learning methods. The source code is available at: https://github.com/SNU-LIST/MOST.