Streamlining Admission with LOR Insights: AI-Based Leadership Assessment in Online Master's Program

TL;DR

This paper presents LORI, an AI tool using NLP and large language models to efficiently assess leadership qualities in recommendation letters for online master's admissions, improving accuracy and reducing review time.

Contribution

Introduction of LORI, a novel AI-based system utilizing RoBERTa and LLAMA models to automatically evaluate leadership skills in recommendation letters for graduate admissions.

Findings

RoBERTa model achieves 91.6% F1 score

High precision and recall in leadership attribute detection

Streamlines the admissions review process

Abstract

Letters of recommendation (LORs) provide valuable insights into candidates' capabilities and experiences beyond standardized test scores. However, reviewing these text-heavy materials is time-consuming and labor-intensive. To address this challenge and support the admission committee in providing feedback for students' professional growth, our study introduces LORI: LOR Insights, a novel AI-based detection tool for assessing leadership skills in LORs submitted by online master's program applicants. By employing natural language processing and leveraging large language models using RoBERTa and LLAMA, we seek to identify leadership attributes such as teamwork, communication, and innovation. Our latest RoBERTa model achieves a weighted F1 score of 91.6%, a precision of 92.4%, and a recall of 91.6%, showing a strong level of consistency in our test data. With the growing importance of leadership skills in the STEM sector, integrating LORI into the graduate admissions process is crucial for accurately assessing applicants' leadership capabilities. This approach not only streamlines the admissions process but also automates and ensures a more comprehensive evaluation of candidates' capabilities.