# Putting words in context: LSTM language models and lexical ambiguity

## TL;DR

This paper investigates how LSTM language models handle lexical ambiguity by probing their hidden states, revealing they encode both lexical and contextual information but with room for improvement in context representation.

## Contribution

It introduces a method to analyze LSTM hidden states for lexical and contextual information, highlighting the model's strengths and limitations in handling ambiguity.

## Key findings

- LSTM models encode significant lexical and contextual information
- Contextual information in hidden states can be improved
- Both lexical and contextual info are represented to a large extent

## Abstract

In neural network models of language, words are commonly represented using context-invariant representations (word embeddings) which are then put in context in the hidden layers. Since words are often ambiguous, representing the contextually relevant information is not trivial. We investigate how an LSTM language model deals with lexical ambiguity in English, designing a method to probe its hidden representations for lexical and contextual information about words. We find that both types of information are represented to a large extent, but also that there is room for improvement for contextual information.