Exploring the Effectiveness of GPT Models in Test-Taking: A Case Study of the Driver's License Knowledge Test

TL;DR

This study demonstrates how GPT models can be enhanced with external context to improve test-taking accuracy, achieving a 96% score on a driver's license exam, but still facing limitations in certain questions.

Contribution

The paper introduces a method for integrating external contextual information into GPT models to improve their question-answering performance on specific knowledge tests.

Findings

GPT-3 achieved 96% accuracy with context, 82% without.

Providing context improves answer accuracy but does not eliminate errors.

Prompt length and format influence model performance.

Abstract

Large language models such as Open AI's Generative Pre-trained Transformer (GPT) models are proficient at answering questions, but their knowledge is confined to the information present in their training data. This limitation renders them ineffective when confronted with questions about recent developments or non-public documents. Our research proposes a method that enables GPT models to answer questions by employing context from an information source not previously included in their training data. The methodology includes preprocessing of contextual information, the embedding of contexts and queries, constructing prompt through the integration of context embeddings, and generating answers using GPT models. We applied this method in a controlled test scenario using the California Driver's Handbook as the information source. The GPT-3 model achieved a 96% passing score on a set of 50 sample driving knowledge test questions. In contrast, without context, the model's passing score fell to 82%. However, the model still fails to answer some questions correctly even with providing library of context, highlighting room for improvement. The research also examined the impact of prompt length and context format, on the model's performance. Overall, the study provides insights into the limitations and potential improvements for GPT models in question-answering tasks.