A note about the generalisation of the C-tests

TL;DR

This paper explores a new way to measure task performance by weighting tasks according to the complexity of their solutions, offering a natural extension of C-tests.

Contribution

It introduces a novel task performance measure based on solution complexity, generalizing C-tests without relying on universal task distributions.

Findings

Proposes a complexity-based weighting scheme for tasks

Reinterprets C-tests as an interactive generalization

Provides a framework for aggregating and decomposing task performance

Abstract

In this exploratory note we ask the question of what a measure of performance for all tasks is like if we use a weighting of tasks based on a difficulty function. This difficulty function depends on the complexity of the (acceptable) solution for the task (instead of a universal distribution over tasks or an adaptive test). The resulting aggregations and decompositions are (now retrospectively) seen as the natural (and trivial) interactive generalisation of the C-tests.