ChatGPT Performance on Standardized Testing Exam -- A Proposed Strategy for Learners

TL;DR

This study evaluates ChatGPT's ability to answer GRE quantitative questions, demonstrating that prompt modifications significantly improve its accuracy, which has implications for test preparation strategies.

Contribution

It introduces a systematic approach to enhancing ChatGPT's performance on standardized test questions through prompt modifications and provides empirical evidence of accuracy improvements.

Findings

ChatGPT achieved 84% accuracy with prompt modifications.

Prompt modifications significantly improved accuracy from 69% to 84%.

The study offers insights into effective prompt strategies for test preparation.

Abstract

This study explores the problem solving capabilities of ChatGPT and its prospective applications in standardized test preparation, focusing on the GRE quantitative exam. Prior research has shown great potential for the utilization of ChatGPT for academic purposes in revolutionizing the approach to studying across various disciplines. We investigate how ChatGPT performs across various question types in the GRE quantitative domain, and how modifying question prompts impacts its accuracy. More specifically this study addressed two research questions: 1. How does ChatGPT perform in answering GRE-based quantitative questions across various content areas? 2. How does the accuracy of ChatGPT vary with modifying the question prompts? The dataset consisting of 100 randomly selected GRE quantitative questions was collected from the ETS official guide to GRE test preparation. We used quantitative evaluation to answer our first research question, and t-test to examine the statistical association between prompt modification and ChatGPT's accuracy. Results show a statistical improvement in the ChatGPT's accuracy after applying instruction priming and contextual prompts to the original questions. ChatGPT showed 84% accuracy with the modified prompts compared to 69% with the original data. The study discusses the areas where ChatGPT struggled with certain questions and how modifications can be helpful for preparing for standardized tests like GRE and provides future directions for prompt modifications.