Is deeper always better? Replacing linear mappings with deep learning networks in the Discriminative Lexicon Model

TL;DR

This paper investigates whether deep neural networks improve the modeling of language comprehension and production over linear methods within the Discriminative Lexicon Model, with mixed results across datasets and tasks.

Contribution

It introduces replacing linear mappings with deep neural networks in the Discriminative Lexicon Model and compares their effectiveness across datasets and tasks.

Findings

Deep learning improves mapping accuracy for large, diverse datasets.

Deep models outperform linear ones for words with pseudo-morphological structure.

Frequency-informed deep learning significantly outperforms linear frequency-based mappings.

Abstract

Recently, deep learning models have increasingly been used in cognitive modelling of language. This study asks whether deep learning can help us to better understand the learning problem that needs to be solved by speakers, above and beyond linear methods. We utilise the Discriminative Lexicon Model introduced by Baayen and colleagues, which models comprehension and production with mappings between numeric form and meaning vectors. While so far, these mappings have been linear (Linear Discriminative Learning, LDL), in the present study we replace them with deep dense neural networks (Deep Discriminative Learning, DDL). We find that DDL affords more accurate mappings for large and diverse datasets from English and Dutch, but not necessarily for Estonian and Taiwan Mandarin. DDL outperforms LDL in particular for words with pseudo-morphological structure such as chol+er. Applied to average reaction times, we find that DDL is outperformed by frequency-informed linear mappings (FIL). However, DDL trained in a frequency-informed way ('frequency-informed' deep learning, FIDDL) substantially outperforms FIL. Finally, while linear mappings can very effectively be updated from trial-to-trial to model incremental lexical learning, deep mappings cannot do so as effectively. At present, both linear and deep mappings are informative for understanding language.