An Eye for an AI: Evaluating GPT-4o's Visual Perception Skills and Geometric Reasoning Skills Using Computer Graphics Questions

TL;DR

This paper evaluates GPT-4o's abilities in visual perception and geometric reasoning through computer graphics questions, revealing its potential and limitations, and offers strategies for educators to integrate GenAI into teaching.

Contribution

It introduces two new datasets for assessing LMMs on CG questions and evaluates GPT-4o's performance, highlighting its strengths and areas for improvement in educational contexts.

Findings

GPT-4o can independently solve visual CG questions but with accuracy limitations.

Major challenges remain in the quality of generated results.

Proposes novel teaching strategies to incorporate GenAI despite current limitations.

Abstract

CG (Computer Graphics) is a popular field of CS (Computer Science), but many students find this topic difficult due to it requiring a large number of skills, such as mathematics, programming, geometric reasoning, and creativity. Over the past few years, researchers have investigated ways to harness the power of GenAI (Generative Artificial Intelligence) to improve teaching. In CS, much of the research has focused on introductory computing. A recent study evaluating the performance of an LLM (Large Language Model), GPT-4 (text-only), on CG questions, indicated poor performance and reliance on detailed descriptions of image content, which often required considerable insight from the user to return reasonable results. So far, no studies have investigated the abilities of LMMs (Large Multimodal Models), or multimodal LLMs, to solve CG questions and how these abilities can be used to improve teaching. In this study, we construct two datasets of CG questions requiring varying degrees of visual perception skills and geometric reasoning skills, and evaluate the current state-of-the-art LMM, GPT-4o, on the two datasets. We find that although GPT-4o exhibits great potential in solving questions with visual information independently, major limitations still exist to the accuracy and quality of the generated results. We propose several novel approaches for CG educators to incorporate GenAI into CG teaching despite these limitations. We hope that our guidelines further encourage learning and engagement in CG classrooms.