Model Complexity of Deep Learning: A Survey

TL;DR

This survey systematically reviews the latest research on deep learning model complexity, focusing on expressive capacity and effective complexity, and discusses their implications for generalization, optimization, and model design.

Contribution

It provides a comprehensive categorization and analysis of recent studies on deep learning model complexity, highlighting key factors and future research directions.

Findings

Deep learning model complexity involves expressive capacity and effective complexity.

Model framework, size, optimization, and data influence complexity.

Understanding complexity aids in generalization and model optimization.

Abstract

Model complexity is a fundamental problem in deep learning. In this paper we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.