TL;DR
This paper examines how optimizing for task success in visual dialogue models can hinder language richness, and explores the potential for improved language quality to enhance task performance across different dialogue games.
Contribution
It reveals that task success-focused training limits linguistic development and demonstrates that better language grounding can improve accuracy in visual dialogue tasks.
Findings
Task success optimization can restrict language richness in models.
Models improve accuracy when learning to handle less frequent words.
Discrepancy between language proficiency and task success is consistent across models and tasks.
Abstract
When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success. We show that in the popular end-to-end approach, this choice prevents the model from learning to generate linguistically richer dialogues, since the acquisition of language proficiency takes longer than learning the guessing task. By comparing models playing different games (GuessWhat, GuessWhich, and Mutual Friends), we show that this discrepancy is model- and task-agnostic. We investigate whether and when better language quality could lead to higher task success. We show that in GuessWhat, models could increase their accuracy if they learn to ground, encode, and decode also words that do not occur frequently in the training set.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
