Analysis of Learner Independent Variables for Estimating Assessment Items Difficulty Level

TL;DR

This paper develops an ordinal regression model utilizing learner independent and generic variables to predict assessment item difficulty levels in engineering courses, achieving around 80% accuracy.

Contribution

It introduces a novel predictive model based on specific variables for estimating assessment difficulty, enhancing assessment quality in engineering education.

Findings

Achieved approximately 80% classification accuracy.

Identified key variables influencing assessment difficulty.

Validated the model across three engineering courses.

Abstract

The quality of assessment determines the quality of learning, and is characterized by validity, reliability and difficulty. Mastery of learning is generally represented by the difficulty levels of assessment items. A very large number of variables are identified in the literature to measure the difficulty level. These variables, which are not completely independent of one another, are categorized into learner dependent, learner independent, generic, non-generic and score based. This research proposes a model for predicting the difficulty level of assessment items in engineering courses using learner independent and generic variables. An ordinal regression model is developed for predicting the difficulty level, and uses six variables including three stimuli variables (item presentation, usage of technical notations and number of resources), two content related variables (number of concepts and procedures) and one task variable (number of conditions). Experimental results from three engineering courses provide around 80% accuracy in classification of items using the proposed model.