Investigation of Large-Margin Softmax in Neural Language Modeling

TL;DR

This paper explores applying large-margin softmax techniques, successful in face recognition, to neural language models to improve discriminative power and analyze effects on speech recognition performance.

Contribution

It introduces large-margin softmax into neural language modeling, compares margin types and norm-scaling strategies, and evaluates their impact on speech recognition metrics.

Findings

Perplexity slightly worsened with large-margin softmax

Word error rate remained comparable to standard softmax

Semantic and syntactic relationships are preserved in word vectors

Abstract

To encourage intra-class compactness and inter-class separability among trainable feature vectors, large-margin softmax methods are developed and widely applied in the face recognition community. The introduction of the large-margin concept into the softmax is reported to have good properties such as enhanced discriminative power, less overfitting and well-defined geometric intuitions. Nowadays, language modeling is commonly approached with neural networks using softmax and cross entropy. In this work, we are curious to see if introducing large-margins to neural language models would improve the perplexity and consequently word error rate in automatic speech recognition. Specifically, we first implement and test various types of conventional margins following the previous works in face recognition. To address the distribution of natural language data, we then compare different strategies for word vector norm-scaling. After that, we apply the best norm-scaling setup in combination with various margins and conduct neural language models rescoring experiments in automatic speech recognition. We find that although perplexity is slightly deteriorated, neural language models with large-margin softmax can yield word error rate similar to that of the standard softmax baseline. Finally, expected margins are analyzed through visualization of word vectors, showing that the syntactic and semantic relationships are also preserved.