Teaching Astronomy with Large Language Models

TL;DR

This study explores integrating large language models into astronomy education, demonstrating how structured guidance enhances AI literacy, improves assessment methods, and fosters critical evaluation skills among students.

Contribution

The paper introduces AstroTutor, a domain-specific LLM-based tutoring system, and evaluates its effectiveness in developing AI literacy and assessment strategies in astronomy education.

Findings

LLM-assisted grading correlates well with human evaluation and offers detailed feedback.

Students' reliance on LLMs decreased over time, indicating skill development.

LLM-facilitated interviews show promise as scalable, individualized assessments.

Abstract

We present a study of LLM integration in final-year undergraduate astronomy education, examining how students develop AI literacy through structured guidance and documentation requirements. We developed AstroTutor, a domain-specific astronomy tutoring system enhanced with curated arXiv content, and deployed it alongside general-purpose LLMs in the course. Students documented their AI usage through homework reflections and post-course surveys. We analyzed student evolution in AI interaction strategies and conducted experimental comparisons of LLM-assisted versus traditional grading methods. LLM grading showed strong correlation with human evaluation while providing more detailed and consistent feedback. We also piloted LLM-facilitated interview-based examinations as a scalable alternative to traditional assessments, demonstrating potential for individualized evaluation that addresses common testing limitations. Students experienced decreased rather than increased reliance on LLMs over the semester, developing critical evaluation skills and strategic tool selection. They evolved from basic assistance-seeking to verification workflows, with documentation requirements fostering metacognitive awareness. Students developed effective prompting strategies, contextual enrichment techniques, and cross-verification practices. Our findings suggest that structured LLM integration with transparency requirements and domain-specific tools can enhance astronomy education while building essential AI literacy skills. We provide implementation guidelines for educators and make our AstroTutor repository freely available.