Modularity in Deep Learning: A Survey

TL;DR

This survey explores the concept of modularity in deep learning, examining data, task, and model aspects, and discusses its benefits, implementations, and future research directions.

Contribution

It provides a comprehensive overview of modularity in deep learning, categorizing its applications and highlighting its advantages across various sub-fields.

Findings

Modularity improves interpretability and scalability of deep learning models.

Different types of modularity are applied across data, task, and model levels.

Future research directions include formal definitions and new modular architectures.

Abstract

Modularity is a general principle present in many fields. It offers attractive advantages, including, among others, ease of conceptualization, interpretability, scalability, module combinability, and module reusability. The deep learning community has long sought to take inspiration from the modularity principle, either implicitly or explicitly. This interest has been increasing over recent years. We review the notion of modularity in deep learning around three axes: data, task, and model, which characterize the life cycle of deep learning. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules. We describe different instantiations of the modularity principle, and we contextualize their advantages in different deep learning sub-fields. Finally, we conclude the paper with a discussion of the definition of modularity and directions for future research.