Dataset vs Reality: Understanding Model Performance from the Perspective of Information Need

TL;DR

This paper emphasizes the importance of understanding the specific information need behind datasets, arguing that models trained on datasets with different underlying information needs may not perform well on real-world problems, and advocates for considering this perspective in dataset creation and model evaluation.

Contribution

The paper introduces the concept of information need as a key factor in dataset design and model performance, using QA and IC tasks as case studies to highlight differences in datasets and their implications.

Findings

Datasets for different tasks reflect distinct information needs.

Differences in dataset creation processes impact model performance.

Considering information need can improve dataset relevance and model applicability.

Abstract

Deep learning technologies have brought us many models that outperform human beings on a few benchmarks. An interesting question is: can these models well solve real-world problems with similar settings (e.g., identical input/output) to the benchmark datasets? We argue that a model is trained to answer the same information need for which the training dataset is created. Although some datasets may share high structural similarities, e.g., question-answer pairs for the question answering (QA) task and image-caption pairs for the image captioning (IC) task, they may represent different research tasks aiming for answering different information needs. To support our argument, we use the QA task and IC task as two case studies and compare their widely used benchmark datasets. From the perspective of information need in the context of information retrieval, we show the differences in the dataset creation processes, and the differences in morphosyntactic properties between datasets. The differences in these datasets can be attributed to the different information needs of the specific research tasks. We encourage all researchers to consider the information need the perspective of a research task before utilizing a dataset to train a model. Likewise, while creating a dataset, researchers may also incorporate the information need perspective as a factor to determine the degree to which the dataset accurately reflects the research task they intend to tackle.