Generalizing Orthogonalization for Models with Non-Linearities

TL;DR

This paper introduces a method to extend orthogonalization techniques to non-linear neural networks, improving bias mitigation and data normalization in complex models with activations like ReLU.

Contribution

It presents a novel approach for orthogonalization in non-linear models, broadening the applicability of bias correction methods beyond linear frameworks.

Findings

Effective bias mitigation in neural networks with ReLU activations

Normalization of CNNs for metadata attributes

Rectification of embeddings for undesired information

Abstract

The complexity of black-box algorithms can lead to various challenges, including the introduction of biases. These biases present immediate risks in the algorithms' application. It was, for instance, shown that neural networks can deduce racial information solely from a patient's X-ray scan, a task beyond the capability of medical experts. If this fact is not known to the medical expert, automatic decision-making based on this algorithm could lead to prescribing a treatment (purely) based on racial information. While current methodologies allow for the "orthogonalization" or "normalization" of neural networks with respect to such information, existing approaches are grounded in linear models. Our paper advances the discourse by introducing corrections for non-linearities such as ReLU activations. Our approach also encompasses scalar and tensor-valued predictions, facilitating its integration into neural network architectures. Through extensive experiments, we validate our method's effectiveness in safeguarding sensitive data in generalized linear models, normalizing convolutional neural networks for metadata, and rectifying pre-existing embeddings for undesired attributes.