Prompt-Engineering and Transformer-based Question Generation and Evaluation

TL;DR

This paper explores transformer-based question generation using prompt engineering, finetuning models like distilBERT and LLaMA to improve question quality for educational applications.

Contribution

It introduces a combined approach of finetuning a distilBERT model and applying prompt engineering with LLaMA for effective question generation from text.

Findings

Prompt engineering achieved over 60% similarity with baseline questions.

30% of generated questions had a similarity score above 70%.

Finetuning distilBERT improved question relevance.

Abstract

Question generation has numerous applications in the educational context. Question generation can prove helpful for students when reviewing content and testing themselves. Furthermore, a question generation model can aid teachers by lessening the burden of creating assessments and other practice material. This paper aims to find the best method to generate questions from textual data through a transformer model and prompt engineering. In this research, we finetuned a pretrained distilBERT model on the SQuAD question answering dataset to generate questions. In addition to training a transformer model, prompt engineering was applied to generate questions effectively using the LLaMA model. The generated questions were compared against the baseline questions in the SQuAD dataset to evaluate the effectiveness of four different prompts. All four prompts demonstrated over 60% similarity on average. Of the prompt-generated questions, 30% achieved a high similarity score greater than 70%.