Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch

TL;DR

This paper proposes a new method for generating educational diagrams using in-context examples with large language models, aiming to reduce hallucinations and improve diagram quality, but faces challenges due to model stochasticity and complex contexts.

Contribution

Introduces a novel RST-based in-context diagram generation method and evaluates its effectiveness in reducing hallucinations and enhancing diagram faithfulness.

Findings

Method decreases factual hallucination rates

Improves diagram alignment with context

Higher complexity increases hallucination risk

Abstract

Generative artificial intelligence (AI) has found a widespread use in computing education; at the same time, quality of generated materials raises concerns among educators and students. This study addresses this issue by introducing a novel method for diagram code generation with in-context examples based on the Rhetorical Structure Theory (RST), which aims to improve diagram generation by aligning models' output with user expectations. Our approach is evaluated by computer science educators, who assessed 150 diagrams generated with large language models (LLMs) for logical organization, connectivity, layout aesthetic, and AI hallucination. The assessment dataset is additionally investigated for its utility in automated diagram evaluation. The preliminary results suggest that our method decreases the rate of factual hallucination and improves diagram faithfulness to provided context; however, due to LLMs' stochasticity, the quality of the generated diagrams varies. Additionally, we present an in-depth analysis and discussion on the connection between AI hallucination and the quality of generated diagrams, which reveals that text contexts of higher complexity lead to higher rates of hallucination and LLMs often fail to detect mistakes in their output.