Game-Theoretic Gradient Control for Robust Neural Network Training

TL;DR

This paper introduces a game-theoretic approach to improve neural network robustness against input noise by modifying backpropagation with gradient dropout and target noising, demonstrating significant benefits in regression tasks.

Contribution

It proposes a novel gradient dropout method framed within compositional game theory, enhancing neural network robustness through controlled neuron gradient nullification and target variable perturbation.

Findings

Gradient dropout improves robustness on regression tasks.

Target noising with stable distributions enhances stability.

Results depend on dataset and hyperparameter tuning.

Abstract

Feed-forward neural networks (FFNNs) are vulnerable to input noise, reducing prediction performance. Existing regularization methods like dropout often alter network architecture or overlook neuron interactions. This study aims to enhance FFNN noise robustness by modifying backpropagation, interpreted as a multi-agent game, and exploring controlled target variable noising. Our "gradient dropout" selectively nullifies hidden layer neuron gradients with probability 1 - p during backpropagation, while keeping forward passes active. This is framed within compositional game theory. Additionally, target variables were perturbed with white noise or stable distributions. Experiments on ten diverse tabular datasets show varying impacts: improvement or diminishing of robustness and accuracy, depending on dataset and hyperparameters. Notably, on regression tasks, gradient dropout (p = 0.9) combined with stable distribution target noising significantly increased input noise robustness, evidenced by flatter MSE curves and more stable SMAPE values. These results highlight the method's potential, underscore the critical role of adaptive parameter tuning, and open new avenues for analyzing neural networks as complex adaptive systems exhibiting emergent behavior within a game-theoretic framework.