Pointing the Unknown Words
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, Yoshua, Bengio

TL;DR
This paper introduces a novel attention-based neural network approach that adaptively predicts unknown words by combining source context pointing and vocabulary prediction, improving NLP task performance.
Contribution
The paper presents a new model with dual softmax layers and an adaptive decision mechanism to better handle rare and unknown words in neural language models.
Findings
Improved translation quality on Europarl English-French corpus.
Enhanced summarization results on Gigaword dataset.
Adaptive softmax decision mechanism outperforms traditional methods.
Abstract
The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Text Readability and Simplification · Natural Language Processing Techniques
MethodsSoftmax
