# Exploratory Evaluation of Learning Behaviors Using a ChatGPT-Based Question-Generating Bot Among Occupational Therapy Students Preparing for the National Licensure Exam: A One-Month Pre-post Study With Cluster Analysis

## TL;DR

This study explored how occupational therapy students used a ChatGPT-based bot to prepare for exams, finding that time spent studying was more important than the number of questions generated.

## Contribution

The study introduces a novel use of a ChatGPT-based bot for exam preparation and identifies distinct learning behavior patterns among students.

## Key findings

- Exam scores significantly improved after one month of using the ChatGPT-based bot.
- Total usage time was more strongly associated with score improvements than the number of questions generated.
- Cluster analysis revealed three distinct learning behavior patterns among students.

## Abstract

Introduction: This exploratory study examined occupational therapy (OT) students’ learning behaviors using a custom-built ChatGPT version 4.0 (OpenAI, Inc., San Francisco, United States) based question-generating bot. Particularly, this study examined changes in mock examination scores before and after one month of use, correlations between usage logs and score improvements, and learning behavior clusters to identify distinct engagement patterns.

Methods: This study comprised 29 OT students, using a one-group pre-post design. The intervention comprised one month of access to a ChatGPT-based bot, developed to generate multiple-choice questions in anatomy, kinesiology, and physiology. Participants were instructed to use the bot as part of their self-study and submit weekly reports via Google Forms (Google LLC, Mountain View, California, United States), documenting the number of questions generated, the number of sessions, and total usage time. Performance was assessed using two 50-item mock exams administered before (Exam I) and after (Exam II) the intervention. Paired t-tests were used to compare the exam scores. Correlations were calculated between the score gains (Exams II-I) and the learning logs. Ward’s hierarchical cluster analysis was performed to explore the distinct patterns of learning behavior.

Results: In total, 28 datasets were analyzed. Exam scores significantly improved between Exam I (17.1 ± 4.4) and Exam II (22.1 ± 7.6), with a mean gain of +5.0 ± 6.0 (t-statistic=4.40, degrees of freedom=27, p<0.001, Cohen’s d=0.83). Correlation analysis indicated weak-to-moderate associations between Exam II scores and the number of questions (r=0.39, p<0.05) and sessions (r=0.36, p<0.10), but no significant association with total usage time. Cluster analysis identified three groups: Cluster 1 (n=14) with low activity and minimal gains; Cluster 2 (n=9) with more questions but little improvement; and Cluster 3 (n=6) with moderate activity but high total usage time, achieving the greatest improvements.

Conclusions: This exploratory study examined OT students’ learning behaviors when using a ChatGPT-based question-generating bot. The findings suggest that increased total time investment may be more relevant to performance gains than the generated question quantity. Future studies should incorporate qualitative analyses of the learning processes and controlled comparisons to further validate these patterns.