Combinatorial Designs for Deep Learning

TL;DR

This paper introduces a combinatorial design approach to dropout in deep learning, aiming to balance edge frequency and reduce overfitting, by analyzing and constructing dropout node patterns inspired by experimental design.

Contribution

It proposes a novel combinatorial design method for dropout in neural networks, enhancing regularization by balancing edge sampling frequencies.

Findings

Designs improve dropout regularization effectiveness

Balanced dropout reduces overfitting

Constructed combinatorial patterns demonstrate theoretical and practical benefits

Abstract

Deep learning is a machine learning methodology using multi-layer neural network. A multi-layer neural network can be regarded as a chain of complete bipartite graphs. The nodes of the first partita is the input layer and the last is the output layer. The edges of a bipartite graph function as weights which are represented as a matrix. The values of i-th partita are computed by multiplication of the weight matrix and values of (i-1)-th partita. Using mass training and teacher data, the weight parameters are estimated little by little. Overfitting (or Overlearning) refers to a model that models the `training data` too well. It then becomes difficult for the model to generalize to new data which were not in the training set. The most popular method to avoid overfitting is called dropout. Dropout deletes a random sample of activations (nodes) to zero during the training process. A random sample of nodes causes more irregular frequency of dropout edges. There is a similar sampling concept in the area of design of experiments. We propose a combinatorial design on dropout nodes from each partita which balances the frequency of edges. We analyze and construct such designs in this paper.