PEAK SHIFT ESTIMATION A novel method to estimate ranking of selectively omitted examination data

TL;DR

This paper introduces Peak Shift Estimation, a novel method for ranking examination difficulty levels using selectively omitted data, demonstrating robustness in simulations and real-world application with promising correlation results.

Contribution

The paper presents the first method to estimate examination difficulty rankings from selectively omitted data, advancing analysis of such unique datasets.

Findings

Peak Shift Estimation accurately ranks university entrance exams in simulations.

The method achieves a correlation coefficient of 0.844 with true rankings in real data.

80% of universities are ranked within 25 positions using this method.

Abstract

In this paper, we focus on examination results when examinees selectively skip examinations, to compare the difficulty levels of these examinations. We call the resultant data 'selectively omitted examination data' Examples of this type of examination are university entrance examinations, certification examinations, and the outcome of students' job-hunting activities. We can learn the number of students accepted for each examination and organization but not the examinees' identity. No research has focused on this type of data. When we know the difficulty level of these examinations, we can obtain a new index to assess organization ability, how many students pass, and the difficulty of the examinations. This index would reflect the outcomes of their education corresponding to perspectives on examinations. Therefore, we propose a novel method, Peak Shift Estimation, to estimate the difficulty level of an examination based on selectively omitted examination data. First, we apply Peak Shift Estimation to the simulation data and demonstrate that Peak Shift Estimation estimates the rank order of the difficulty level of university entrance examinations very robustly. Peak Shift Estimation is also suitable for estimating a multi-level scale for universities, that is, A, B, C, and D rank university entrance examinations. We apply Peak Shift Estimation to real data of the Tokyo metropolitan area and demonstrate that the rank correlation coefficient between difficulty level ranking and true ranking is 0.844 and that the difference between 80 percent of universities is within 25 ranks. The accuracy of Peak Shift Estimation is thus low and must be improved; however, this is the first study to focus on ranking selectively omitted examination data, and therefore, one of our contributions is to shed light on this method.