A Second-Order Perspective on Model Compositionality and Incremental Learning

TL;DR

This paper provides a theoretical framework for understanding compositionality in deep networks using second-order Taylor approximations, and introduces incremental training algorithms that enhance multi-task learning and modularity.

Contribution

It offers a novel second-order theoretical perspective on model compositionality and proposes dual incremental training algorithms for improved multi-task and modular learning.

Findings

Within pre-training basin, modules are more composable.

Incremental algorithms enable effective multi-task learning.

Modules support unlearning and task specialization.

Abstract

The fine-tuning of deep pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into a single, multi-task model. However, identifying the conditions that promote compositionality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify compositionality in standard non-linear networks through the second-order Taylor approximation of the loss function. The proposed formulation highlights the importance of staying within the pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as a whole. We probe their application in incremental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks. Code available at https://github.com/aimagelab/mammoth.