Learning Neural Networks on SVD Boosted Latent Spaces for Semantic Classification

TL;DR

This paper introduces a method using singular value decomposition to reduce the dimensionality of text representations for neural networks, maintaining or improving performance while significantly lowering computational costs.

Contribution

It proposes a novel approach of applying SVD to create lower-dimensional latent spaces for neural network training in text classification tasks.

Findings

Reduced computational complexity with maintained or improved accuracy

Neural networks trained on SVD-processed spaces outperform traditional methods

Effective scaling to large vocabularies in text classification

Abstract

The availability of large amounts of data and compelling computation power have made deep learning models much popular for text classification and sentiment analysis. Deep neural networks have achieved competitive performance on the above tasks when trained on naive text representations such as word count, term frequency, and binary matrix embeddings. However, many of the above representations result in the input space having a dimension of the order of the vocabulary size, which is enormous. This leads to a blow-up in the number of parameters to be learned, and the computational cost becomes infeasible when scaling to domains that require retaining a colossal vocabulary. This work proposes using singular value decomposition to transform the high dimensional input space to a lower-dimensional latent space. We show that neural networks trained on this lower-dimensional space are not only able to retain performance while savoring significant reduction in the computational complexity but, in many situations, also outperforms the classical neural networks trained on the native input space.