Exploring Linear Feature Disentanglement For Neural Networks

TL;DR

This paper investigates whether certain features in neural networks become linearly separable earlier and can be detached, proposing a learnable mask to identify and prune such features with minimal performance impact.

Contribution

It introduces a learnable mask module to distinguish linear from non-linear features, enabling partial feature detachment and a pruning strategy in neural networks.

Findings

Some features reach linear separability earlier than others.

Partial feature detachment minimally affects model performance.

The method is validated on four datasets with promising results.

Abstract

Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs). Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs, i.e., whether there exists a part of features arriving at the linear separable feature space in the intermediate layers, that does not require further non-linear variation but an affine transformation instead. To validate the above hypothesis, we explore the problem of linear feature disentanglement for neural networks in this paper. Specifically, we devise a learnable mask module to distinguish between linear and non-linear features. Through our designed experiments we found that some features reach the linearly separable space earlier than the others and can be detached partly from the NNs. The explored method also provides a readily feasible pruning strategy which barely affects the performance of the original model. We conduct our experiments on four datasets and present promising results.