Improving learnability of neural networks: adding supplementary axes to disentangle data representation

TL;DR

This paper investigates how adding supplementary axes to neural networks enhances learnability and disentangles data representations, leading to more robust training across various models and datasets.

Contribution

It demonstrates that concatenating supplementary information improves neural network training and disentanglement, validated on simple and complex models with different data types.

Findings

Concatenation of supplementary axes improves training robustness and accuracy.

The approach is effective for both simple MLPs and complex CNNs.

Findings generalize to ultrasound images and cGANs.

Abstract

Over-parameterized deep neural networks have proven to be able to learn an arbitrary dataset with 100$\%$ training accuracy. Because of a risk of overfitting and computational cost issues, we cannot afford to increase the number of network nodes if we want achieve better training results for medical images. Previous deep learning research shows that the training ability of a neural network improves dramatically (for the same epoch of training) when a few nodes with supplementary information are added to the network. These few informative nodes allow the network to learn features that are otherwise difficult to learn by generating a disentangled data representation. This paper analyzes how concatenation of additional information as supplementary axes affects the training of the neural networks. This analysis was conducted for a simple multilayer perceptron (MLP) classification model with a rectified linear unit (ReLU) on two-dimensional training data. We compared the networks with and without concatenation of supplementary information to support our analysis. The model with concatenation showed more robust and accurate training results compared to the model without concatenation. We also confirmed that our findings are valid for deeper convolutional neural networks (CNN) using ultrasound images and for a conditional generative adversarial network (cGAN) using the MNIST data.