Difficulty-level Modeling of Ontology-based Factual Questions

TL;DR

This paper introduces a novel methodology for modeling the difficulty levels of ontology-based factual questions by leveraging Item Response Theory and training logistic regression models for different learner categories.

Contribution

It proposes an ontology-based feature set and a new approach using IRT to predict question difficulty for various learner proficiency levels.

Findings

Effective difficulty prediction for different learner categories.

Use of ontology-based metrics for difficulty assessment.

Improved accuracy over simple existing methods.

Abstract

Semantics based knowledge representations such as ontologies are found to be very useful in automatically generating meaningful factual questions. Determining the difficulty level of these system generated questions is helpful to effectively utilize them in various educational and professional applications. The existing approaches for finding the difficulty level of factual questions are very simple and are limited to a few basic principles. We propose a new methodology for this problem by considering an educational theory called Item Response Theory (IRT). In the IRT, knowledge proficiency of end users (learners) are considered for assigning difficulty levels, because of the assumptions that a given question is perceived differently by learners of various proficiencies. We have done a detailed study on the features (factors) of a question statement which could possibly determine its difficulty level for three learner categories (experts, intermediates and beginners). We formulate ontology based metrics for the same. We then train three logistic regression models to predict the difficulty level corresponding to the three learner categories.