The Surprising Positive Knowledge Transfer in Continual 3D Object Shape Reconstruction

TL;DR

This paper demonstrates that continual learning in 3D shape reconstruction can surprisingly lead to positive knowledge transfer, improving performance across tasks without special heuristics, and introduces a new analysis method for understanding this transfer.

Contribution

It introduces continual 3D shape reconstruction tasks showing positive transfer, and provides a novel analysis of knowledge transfer via output distribution shift.

Findings

Positive backward and forward transfer observed during training.

Representation learning improves performance on both learned and new categories.

Potential for proxy tasks in continual classification.

Abstract

Continual learning has been extensively studied for classification tasks with methods developed to primarily avoid catastrophic forgetting, a phenomenon where earlier learned concepts are forgotten at the expense of more recent samples. In this work, we present a set of continual 3D object shape reconstruction tasks, including complete 3D shape reconstruction from different input modalities, as well as visible surface (2.5D) reconstruction which, surprisingly demonstrate positive knowledge (backward and forward) transfer when training with solely standard SGD and without additional heuristics. We provide evidence that continuously updated representation learning of single-view 3D shape reconstruction improves the performance on learned and novel categories over time. We provide a novel analysis of knowledge transfer ability by looking at the output distribution shift across sequential learning tasks. Finally, we show that the robustness of these tasks leads to the potential of having a proxy representation learning task for continual classification. The codebase, dataset and pre-trained models released with this article can be found at https://github.com/rehg-lab/CLRec