Evaluation of GPT-based large language generative AI models as study aids for the national licensure examination for registered dietitians in Japan

TL;DR

This study evaluates GPT-based AI models as study aids for Japan's dietitian licensure exam, finding limited accuracy and consistency, with some models marginally surpassing passing thresholds but overall requiring further improvements.

Contribution

It provides a comprehensive assessment of current LLM-based AI models' performance in nutritional exam preparation, highlighting their strengths and limitations.

Findings

Bing-Precise and Bing-Creative exceeded the 60% passing threshold.

Models showed inconsistent answers across repeated attempts.

Prompt engineering had limited impact on improving accuracy.

Abstract

Generative artificial intelligence (AI) based on large language models (LLMs), such as ChatGPT, has demonstrated remarkable progress across various professional fields, including medicine and education. However, their performance in nutritional education, especially in Japanese national licensure examination for registered dietitians, remains underexplored. This study aimed to evaluate the potential of current LLM-based generative AI models as study aids for nutrition students. Questions from the Japanese national examination for registered dietitians were used as prompts for ChatGPT and three Bing models (Precise, Creative, Balanced), based on GPT-3.5 and GPT-4. Each question was entered into independent sessions, and model responses were analyzed for accuracy, consistency, and response time. Additional prompt engineering, including role assignment, was tested to assess potential performance improvements. Bing-Precise (66.2%) and Bing-Creative (61.4%) surpassed the passing threshold (60%), while Bing-Balanced (43.3%) and ChatGPT (42.8%) did not. Bing-Precise and Bing-Creative generally outperformed others across subject fields except Nutrition Education, where all models underperformed. None of the models consistently provided the same correct responses across repeated attempts, highlighting limitations in answer stability. ChatGPT showed greater consistency in response patterns but lower accuracy. Prompt engineering had minimal effect, except for modest improvement when correct answers and explanations were explicitly provided. While some generative AI models marginally exceeded the passing threshold, overall accuracy and answer consistency remained suboptimal. Moreover, all the models demonstrated notable limitations in answer consistency and robustness. Further advancements are needed to ensure reliable and stable AI-based study aids for dietitian licensure preparation.