Dear admission committee…: Which moves in application essays predict student master grades?

TL;DR

This study uses text analysis to show that specific content in application essays can predict student success in master's programs.

Contribution

The study introduces a novel method using LDA to objectively analyze application essays and predict student grades.

Findings

Five out of seven essay 'moves' significantly affect student grades.

Moves like 'master specific' and 'interest to learn' positively correlate with success.

Moves like 'research skills' and 'societal impact' negatively correlate with success.

Abstract

Application essays are a commonly used admission instrument for students entering higher education. The quality of the essay is usually scored, but this score is often subjective and has poor interrater reliability due to the unstructured format of the essays. This results in mixed findings on the validity of application essays as an admission instrument. We propose a more objective method of using application essays, using Latent Dirichlet Allocation (LDA), a text mining method, to distinguish seven moves occurring in application essays written by students who apply to a master degree program. We use the probability that these moves occur in the essay to predict study success in the master. Thereby we answer the following research question: What is the effect of discussing different moves in students’ application essays on the student grades in a master program? From the seven different moves (functional unit of text) we distinguished, five of which have a significant effect on student grades. The moves we labeled as ‘master specific’ and ‘interest to learn’ have a positive effect on student grades, and the moves we labeled as ‘research skills’, ‘societal impact’ and ‘city and university’ have a negative effect. Our interpretation of this finding is that topics related to intrinsic motivation and specific knowledge, as opposed to generic knowledge, are positively related with study success. We thereby demonstrate that application essays can be a valid predictor of study success. This contributes to justifying their use as admission instruments.

Figures