Can LLMs Produce Original Astronomy Research in a Semester? A Graduate Class Experiment

TL;DR

This study explores whether large language models can assist graduate students in conducting original astronomy research within a semester, highlighting both potential benefits and current limitations.

Contribution

It provides empirical insights into LLMs' effectiveness and challenges in supporting complex scientific research tasks in astronomy education.

Findings

LLMs helped students complete research drafts within a semester.

Models often provided false citations and struggled with complex code and data retrieval.

Students' perceptions of LLM usefulness varied, with concerns about creativity and accuracy.

Abstract

We discuss the results of using large language models (LLMs) to conduct original scientific research in an unfamiliar subject area during the Fall 2025 semester. Students in a graduate astronomy and astrophysics course were asked to test whether LLMs could help them complete research tasks faster and at a level of detail and accuracy required for scientific publication. Most students employed LLMs for a total of 5-10 hours. While all students completed a draft paper on an unsolved problem related to galaxies by semester's end, their impressions of the models' value varied. About half thought that the models saved them time. Many noted that LLMs failed to provide appropriately detailed insights or steps to addressing open, niche questions over a several-month timeframe. The LLMs also frequently (about 20% of the time) returned false citations, links, or summaries of papers. The models struggled with generating complex functional code, accessing online packages or Application Programming Interfaces (APIs), and retrieving astronomical datasets from existing archives. In writing code and in chats, the LLMs made implicit, overly simplifying assumptions and often doubled down even after being corrected. Given the rapid pace of LLM development, new models may soon address at least some of these issues and thus significantly enhance research productivity. Yet students expressed concerns about how LLM use might dampen creativity and reflection during the research process. To improve learning experiences in future semesters, the class will first discuss LLM best practices and limitations. Students will be encouraged to explore free online resources for tips for generative model applications and will decide for themselves whether to use LLMs for their research project. This white paper was not written using LLMs.