Neural Random Subspace

TL;DR

Neural Random Subspace (NRS) is a novel deep learning method that integrates the random subspace technique into neural networks, enabling efficient, accurate, and end-to-end learnable non-linear feature representations for various tasks.

Contribution

This paper introduces Neural Random Subspace, the first deep learning-based random subspace method that combines end-to-end training with improved efficiency and accuracy over traditional ensemble methods.

Findings

NRS outperforms random forests and GBDTs on 35 datasets.

NRS improves CNN performance on 2D and 3D recognition tasks.

NRS achieves these with minimal additional computational cost.

Abstract

The random subspace method, known as the pillar of random forests, is good at making precise and robust predictions. However, there is not a straightforward way yet to combine it with deep learning. In this paper, we therefore propose Neural Random Subspace (NRS), a novel deep learning based random subspace method. In contrast to previous forest methods, NRS enjoys the benefits of end-to-end, data-driven representation learning, as well as pervasive support from deep learning software and hardware platforms, hence achieving faster inference speed and higher accuracy. Furthermore, as a non-linear component to be encoded into Convolutional Neural Networks (CNNs), NRS learns non-linear feature representations in CNNs more efficiently than previous higher-order pooling methods, producing good results with negligible increase in parameters, floating point operations (FLOPs) and real running time. Compared with random subspaces, random forests and gradient boosting decision trees (GBDTs), NRS achieves superior performance on 35 machine learning datasets. Moreover, on both 2D image and 3D point cloud recognition tasks, integration of NRS with CNN architectures achieves consistent improvements with minor extra cost. Code is available at https://github.com/CupidJay/NRS_pytorch.