Data Science Students Perspectives on Learning Analytics: An Application of Human-Led and LLM Content Analysis

TL;DR

This study combines human-led content analysis with advanced LLM techniques to explore postgraduate students' perspectives on learning analytics, revealing their understanding and interests through collaborative data analysis.

Contribution

It introduces a novel integration of RAG and LLM methods with human analysis to examine student perspectives on learning analytics.

Findings

Students adopted structured data science methodologies.

Seven thematic categories of student questions were identified.

Students in data science showed deeper understanding of analytics.

Abstract

Objective This study is part of a series of initiatives at a UK university designed to cultivate a deep understanding of students' perspectives on analytics that resonate with their unique learning needs. It explores collaborative data processing undertaken by postgraduate students who examined an Open University Learning Analytics Dataset (OULAD). Methods A qualitative approach was adopted, integrating a Retrieval-Augmented Generation (RAG) and a Large Language Model (LLM) technique with human-led content analysis to gather information about students' perspectives based on their submitted work. The study involved 72 postgraduate students in 12 groups. Findings The analysis of group work revealed diverse insights into essential learning analytics from the students' perspectives. All groups adopted a structured data science methodology. The questions formulated by the groups were categorised into seven themes, reflecting their specific areas of interest. While there was variation in the selected variables to interpret correlations, a consensus was found regarding the general results. Conclusion A significant outcome of this study is that students specialising in data science exhibited a deeper understanding of learning analytics, effectively articulating their interests through inferences drawn from their analyses. While human-led content analysis provided a general understanding of students' perspectives, the LLM offered nuanced insights.