Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization

TL;DR

This paper investigates the use of Group Lasso regularization to automatically select salient nodes in deep neural networks, improving model efficiency while maintaining high classification accuracy in speech recognition tasks.

Contribution

It introduces a method applying Group Lasso regularization to DNNs for node selection, demonstrating its effectiveness in speech recognition.

Findings

gLasso effectively selects necessary nodes for high performance

Comparison shows gLasso outperforms L2 regularization in node selection

Selected nodes maintain high classification accuracy

Abstract

We examine the effect of the Group Lasso (gLasso) regularizer in selecting the salient nodes of Deep Neural Network (DNN) hidden layers by applying a DNN-HMM hybrid speech recognizer to TED Talks speech data. We test two types of gLasso regularization, one for outgoing weight vectors and another for incoming weight vectors, as well as two sizes of DNNs: 2048 hidden layer nodes and 4096 nodes. Furthermore, we compare gLasso and L2 regularizers. Our experiment results demonstrate that our DNN training, in which the gLasso regularizer was embedded, successfully selected the hidden layer nodes that are necessary and sufficient for achieving high classification power.