Research on the Application of Large Language Models in Automatic Question Generation: A Case Study of ChatGLM in the Context of High School Information Technology Curriculum

TL;DR

This paper evaluates ChatGLM's effectiveness in automatically generating high school IT exam questions, showing it can produce clearer questions and increase teachers' willingness to use them, thus improving educational assessment efficiency.

Contribution

It demonstrates the successful application of prompt engineering with ChatGLM for diverse question generation and provides insights into its advantages over human questions.

Findings

ChatGLM outperforms humans in clarity and willingness to use.

No significant difference in hit rate and fit between ChatGLM and human questions.

Potential for ChatGLM to enhance question generation efficiency in education.

Abstract

This study investigates the application effectiveness of the Large Language Model (LLMs) ChatGLM in the automated generation of high school information technology exam questions. Through meticulously designed prompt engineering strategies, the model is guided to generate diverse questions, which are then comprehensively evaluated by domain experts. The evaluation dimensions include the Hitting(the degree of alignment with teaching content), Fitting (the degree of embodiment of core competencies), Clarity (the explicitness of question descriptions), and Willing to use (the teacher's willingness to use the question in teaching). The results indicate that ChatGLM outperforms human-generated questions in terms of clarity and teachers' willingness to use, although there is no significant difference in hit rate and fit. This finding suggests that ChatGLM has the potential to enhance the efficiency of question generation and alleviate the burden on teachers, providing a new perspective for the future development of educational assessment systems. Future research could explore further optimizations to the ChatGLM model to maintain high fit and hit rates while improving the clarity of questions and teachers' willingness to use them.