Deep Learning: Generalization Requires Deep Compositional Feature Space Design

TL;DR

This paper argues that the design of the feature space using deep compositional functions significantly influences the generalization ability of deep models, with experiments showing improvements through compositional design and learning rate decay.

Contribution

It introduces the importance of deep compositional feature space design for better generalization and demonstrates how it can mitigate information loss in convolutional networks.

Findings

Compositional feature space design improves generalization.

Learning rate decay acts as an implicit regularizer.

Marginalizing information loss enhances model performance.

Abstract

Generalization error defines the discriminability and the representation power of a deep model. In this work, we claim that feature space design using deep compositional function plays a significant role in generalization along with explicit and implicit regularizations. Our claims are being established with several image classification experiments. We show that the information loss due to convolution and max pooling can be marginalized with the compositional design, improving generalization performance. Also, we will show that learning rate decay acts as an implicit regularizer in deep model training.