Knowledge Distillation for Continual Learning of Biomedical Neural Fields

TL;DR

This paper explores how neural fields used in biomedical imaging suffer from catastrophic forgetting and demonstrates that knowledge distillation can effectively enable continual learning in these models.

Contribution

It introduces a strategy using knowledge distillation to mitigate catastrophic forgetting in neural fields during incremental data learning.

Findings

Knowledge distillation reduces forgetting in neural fields.

The extent of forgetting varies with the neural field model.

Distillation enables continual learning in neural fields.

Abstract

Neural fields are increasingly used as a light-weight, continuous, and differentiable signal representation in (bio)medical imaging. However, unlike discrete signal representations such as voxel grids, neural fields cannot be easily extended. As neural fields are, in essence, neural networks, prior signals represented in a neural field will degrade when the model is presented with new data due to catastrophic forgetting. This work examines the extent to which different neural field approaches suffer from catastrophic forgetting and proposes a strategy to mitigate this issue. We consider the scenario in which data becomes available incrementally, with only the most recent data available for neural field fitting. In a series of experiments on cardiac cine MRI data, we demonstrate how knowledge distillation mitigates catastrophic forgetting when the spatiotemporal domain is enlarged or the dimensionality of the represented signal is increased. We find that the amount of catastrophic forgetting depends, to a large extent, on the neural fields model used, and that distillation could enable continual learning in neural fields.