Automatic Identification of Ineffective Online Student Questions in Computing Education

TL;DR

This paper develops machine learning methods to automatically identify ineffective questions in online computer science education, enabling large-scale automated feedback and question improvement.

Contribution

It introduces a classification framework for assessing question effectiveness and evaluates multiple algorithms, demonstrating feasibility for automated educational support.

Findings

Support Vector Machines achieved high classification accuracy.

Manual classification reliability was high with Cohen's Kappa of .88.

Automated methods can effectively identify ineffective questions.

Abstract

This Research Full Paper explores automatic identification of ineffective learning questions in the context of large-scale computer science classes. The immediate and accurate identification of ineffective learning questions opens the door to possible automated facilitation on a large scale, such as alerting learners to revise questions and providing adaptive question revision suggestions. To achieve this, 983 questions were collected from a question & answer platform implemented by an introductory programming course over three semesters in a large research university in the Southeastern United States. Questions were firstly manually classified into three hierarchical categories: 1) learning-irrelevant questions, 2) effective learning-relevant questions, 3) ineffective learningrelevant questions. The inter-rater reliability of the manual classification (Cohen's Kappa) was .88. Four different machine learning algorithms were then used to automatically classify the questions, including Naive Bayes Multinomial, Logistic Regression, Support Vector Machines, and Boosted Decision Tree. Both flat and single path strategies were explored, and the most effective algorithms under both strategies were identified and discussed. This study contributes to the automatic determination of learning question quality in computer science, and provides evidence for the feasibility of automated facilitation of online question & answer in large scale computer science classes.