Reading Comprehension Ability Test-A Turing Test for Reading Comprehension

TL;DR

This paper introduces the Reading Comprehension Ability Test (CAT), a Turing-test-like framework to evaluate and compare machine and human reading comprehension across multiple ability levels.

Contribution

It proposes a novel assessment method, CAT, to measure and differentiate machine and human reading comprehension abilities at various levels.

Findings

CAT can distinguish between different levels of comprehension ability.

It provides a benchmark for evaluating AI reading understanding.

The test aligns with human comprehension levels.

Abstract

Reading comprehension is an important ability of human intelligence. Literacy and numeracy are two most essential foundation for people to succeed at study, at work and in life. Reading comprehension ability is a core component of literacy. In most of the education systems, developing reading comprehension ability is compulsory in the curriculum from year one to year 12. It is an indispensable ability in the dissemination of knowledge. With the emerging artificial intelligence, computers start to be able to read and understand like people in some context. They can even read better than human beings for some tasks, but have little clue in other tasks. It will be very beneficial if we can identify the levels of machine comprehension ability, which will direct us on the further improvement. Turing test is a well-known test of the difference between computer intelligence and human intelligence. In order to be able to compare the difference between people reading and machines reading, we proposed a test called (reading) Comprehension Ability Test (CAT).CAT is similar to Turing test, passing of which means we cannot differentiate people from algorithms in term of their comprehension ability. CAT has multiple levels showing the different abilities in reading comprehension, from identifying basic facts, performing inference, to understanding the intent and sentiment.