Boundary Optimizing Network (BON)

TL;DR

The paper introduces Boundary Optimizing Network (BON), a method that enhances neural network generalization by generating synthetic data around original points to prevent overfitting, with improvements demonstrated on CIFAR-10 and Iris datasets.

Contribution

It proposes a novel collaborative generative approach for data augmentation to improve neural network generalization, and introduces BON++ to address catastrophic forgetting in the generator.

Findings

BON improves convergence on CIFAR-10 with Densenet.

BON++ effectively mitigates catastrophic forgetting in the generator.

The approach has potential for creating better decision boundaries in high-dimensional spaces.

Abstract

Despite all the success that deep neural networks have seen in classifying certain datasets, the challenge of finding optimal solutions that generalize still remains. In this paper, we propose the Boundary Optimizing Network (BON), a new approach to generalization for deep neural networks when used for supervised learning. Given a classification network, we propose to use a collaborative generative network that produces new synthetic data points in the form of perturbations of original data points. In this way, we create a data support around each original data point which prevents decision boundaries from passing too close to the original data points, i.e. prevents overfitting. We show that BON improves convergence on CIFAR-10 using the state-of-the-art Densenet. We do however observe that the generative network suffers from catastrophic forgetting during training, and we therefore propose to use a variation of Memory Aware Synapses to optimize the generative network (called BON++). On the Iris dataset, we visualize the effect of BON++ when the generator does not suffer from catastrophic forgetting and conclude that the approach has the potential to create better boundaries in a higher dimensional space.