Neural Complexity Measures

TL;DR

This paper introduces Neural Complexity (NC), a meta-learning framework that learns a scalar complexity measure to predict and improve the generalization of deep neural networks across various tasks.

Contribution

It presents a data-driven approach to learn a neural complexity measure that can be integrated into training to enhance generalization, outperforming existing manual measures.

Findings

NC effectively predicts generalization across tasks

Incorporating NC improves model regularization and performance

NC outperforms existing complexity measures in experiments

Abstract

While various complexity measures for deep neural networks exist, specifying an appropriate measure capable of predicting and explaining generalization in deep networks has proven challenging. We propose Neural Complexity (NC), a meta-learning framework for predicting generalization. Our model learns a scalar complexity measure through interactions with many heterogeneous tasks in a data-driven way. The trained NC model can be added to the standard training loss to regularize any task learner in a standard supervised learning scenario. We contrast NC's approach against existing manually-designed complexity measures and other meta-learning models, and we validate NC's performance on multiple regression and classification tasks