Hard to Cheat: A Turing Test based on Answering Questions about Images

TL;DR

This paper proposes using question answering about images as a robust Turing Test to evaluate machine intelligence, emphasizing its potential to address holistic understanding challenges in AI.

Contribution

It introduces a novel approach to assess AI through image-based question answering as a more robust Turing Test compared to existing tasks.

Findings

Highlights the robustness of image question answering as a Turing Test

Contrasts this approach with grounding and description generation tasks

Discusses tools for measuring progress in holistic AI understanding

Abstract

Progress in language and image understanding by machines has sparkled the interest of the research community in more open-ended, holistic tasks, and refueled an old AI dream of building intelligent machines. We discuss a few prominent challenges that characterize such holistic tasks and argue for "question answering about images" as a particular appealing instance of such a holistic task. In particular, we point out that it is a version of a Turing Test that is likely to be more robust to over-interpretations and contrast it with tasks like grounding and generation of descriptions. Finally, we discuss tools to measure progress in this field.