Methodological Variation in Studying Staff and Student Perceptions of AI

TL;DR

This study compares various methodological approaches to assess AI perceptions among staff and students, revealing that different analysis methods can yield varying results and highlighting the importance of methodological choices.

Contribution

It systematically evaluates how different qualitative and quantitative analysis methods influence perceptions of AI, emphasizing the impact of methodological variation.

Findings

Different analyses produce varying results for the same data source.

Content-based analyses reveal nuances missed by overall sentiment measures.

Methodological choices significantly affect perception assessments.

Abstract

In this paper, we compare methodological approaches for comparing student and staff perceptions, and ask: how much do these measures vary across different approaches? We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure, or with a mixed methods approach that compares two distinct data sources - e.g. a quantitative questionnaire with qualitative comments. To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups. We conduct two analyses for each data source: with a sentiment and stance analysis, we measure overall negativity/positivity of the comments and focus group conversations, respectively. Meanwhile, word clouds from the comments and a thematic analysis of the focus groups provide further detail on the content of this qualitative data - particularly the thematic analysis, which includes both similarities and differences between students and staff. We show that different analyses can produce different results - for a single data source. This variation stems from the construct being evaluated - an overall measure of positivity/negativity can produce a different picture from more detailed content-based analyses. We discuss the implications of this variation for institutional contexts, and for the comparisons from previous studies.