Prompt-Based Cost-Effective Evaluation and Operation of ChatGPT as a Computer Programming Teaching Assistant
Marc Ballestero-Rib\'o, Daniel Ortiz-Mart\'inez

TL;DR
This paper evaluates GPT models for providing programming feedback, proposes prompt techniques to automate evaluation, and discusses strategies for deploying LLMs as teaching assistants in programming education.
Contribution
It introduces a prompt-based method to assess LLM feedback quality and outlines a practical strategy for using LLMs as programming teaching assistants.
Findings
GPT-4T outperforms GPT-3.5T but still generates some incorrect feedback.
Proposed prompts enable automated evaluation and diagnostic analysis.
A strategy for deploying LLMs as educational tools is outlined.
Abstract
The dream of achieving a student-teacher ratio of 1:1 is closer than ever thanks to the emergence of large language models (LLMs). One potential application of these models in the educational field would be to provide feedback to students in university introductory programming courses, so that a student struggling to solve a basic implementation problem could seek help from an LLM available 24/7. This article focuses on studying three aspects related to such an application. First, the performance of two well-known models, GPT-3.5T and GPT-4T, in providing feedback to students is evaluated. The empirical results showed that GPT-4T performs much better than GPT-3.5T, however, it is not yet ready for use in a real-world scenario. This is due to the possibility of generating incorrect information that potential users may not always be able to detect. Second, the article proposes a carefully…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Healthcare and Education · Online Learning and Analytics
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · 15 Ways to Contact How can i speak to someone at Delta Airlines · Attention Is All You Need · Linear Layer · Weight Decay · Multi-Head Attention · {Dispute@FaQ-s}How to file a dispute with Expedia? · Layer Normalization · Cosine Annealing · Dense Connections
