Educational improvement through machine learning: Strategic models for better PISA scores

TL;DR

This study uses machine learning to identify key factors influencing student success in PISA exams, offering insights for better education policies.

Contribution

The study introduces new prediction models using variables like access to IT and metacognition to improve PISA scores.

Findings

Access to information technology significantly impacts student success in PISA exams.

Metacognition and weekly instructional hours are key predictors of student achievement.

Proposed models help policymakers design effective education strategies for better PISA outcomes.

Abstract

In this study, in addition to traditional variables such as economic wealth or the number of books read, on which many studies have already been conducted, variables that are thought to influence student achievement and better predict success are identified. Random Forest algorithm was used to identify important variables based on the PISA 2018 data, covering all three domains of science, mathematics and reading. The study found that the main factors influencing the success of students in countries that perform well in the PISA exam are essentially access to information technology, weekly hours of instruction in the subject, economic-social and cultural status, parents’ occupation, level of metacognition, awareness of PISA, sense of competition and attitudes towards reading. New prediction models based on these variables were proposed. The proposed models will give a significant advantage to policy makers who want to improve their country’s PISA score and implement appropriate education policies.

Figures

Most Important Variables in Reading(a), Science(b) and Math(c) Section.

Measurement areas and important variables in PISA 2018.

Equations

Full text

Introduction

Today, various measurement and evaluation tools are used worldwide to analyze and compare both the quality of educational practices and student achievement [1] The best known and most effective comparative assessment program among them is the Program for International Student Assessment (PISA) [2]. OECD countries benefit greatly from the results of this program when evaluating and improving their education systems. PISA, which is conducted every three years by the OECD, assesses the skills of 15-year-old students in reading, mathematics and science and provides a valuable basis for international comparisons [3]. In contrast to traditional measurement and assessment methods, PISA focuses on assessing students’ critical thinking and practical problem-solving skills. In this sense, it aims to measure not only students’ access to knowledge, but also how they process and apply it in real-world contexts. As the need to raise educational standards in a globalized world becomes ever greater, PISA has grown in importance [4]. The results of the program provide an objective basis for understanding the position and performance of a country’s education system in an international comparison and promote a climate of competition [5]. In addition, this data allows countries to assess structural differences, policy orientations and implementation outcomes in their education systems. PISA also facilitates the investigation of the causes of individual success or failure and provides important insights for the development of strategies to improve student performance [6]. In order to gain relevant insights, the PISA data has been used in a variety of publications/studies for various analyzes and comparisons. A close examination of the purpose and content of these studies reveals that most focus on comparing countries’ performance in science, reading and maths and attempt to identify the determinants of success or failure in order to inform educational policy makers [7–12]. Studies that aim to improve the quality of learning processes and provide meaningful insights have also gained attention [10,13–15]. However, much of this literature relies on a limited number of variables and rarely applies advanced or innovative analytical techniques, so it has limited explanatory power. In general, student achievement is explained by commonly accepted variables such as socioeconomic status or the number of books in the household, which are well known and often do not require robust empirical testing. This underscores the need for researchers and policy makers to explore additional factors- that are as important or even more effective than socioeconomic status and that contribute to student achievement.

In this context, the rich, multivariate and culturally diverse structure of the PISA data requires not only traditional statistical analysis, but also the application of artificial intelligence-based algorithms. While many studies limit themselves to regression or correlation analyzes [16,17], they often overlook non-linear relationships, interactions between variables and contextual dependencies. In recent years, the increasing use of machine learning algorithms in educational research has shown that they have great potential to explain complex phenomena and make accurate predictions [18,19].

On the other hand, recent studies have shown that methods such as the Random Forest Algorithm (RFA) offer unique advantages in this area. In particular, its ability to detect nonlinear interactions and assess the relative importance of variables in large training datasets is remarkable [20–22]. While traditional approaches often ignore complex dependencies and interaction effects, RFA can uncover complex patterns that classical statistical models cannot detect. This property has made it a particularly valuable tool for comparative educational research. However, despite all this potential, the use of RFA in PISA-based studies is still limited, pointing to an important methodological gap that our study attempts to fill [23].

The main objective of this study is to identify the factors that influence student achievement in OECD countries with above-average performance in reading, maths and science, beyond traditional variables such as economic wealth or the number of books read, and to develop a predictive model based on these variables that can provide functional insights to policy makers in countries seeking to improve their PISA results. The following research questions guide the study:

RQ1) What are the most important variables influencing OECD countries’ PISA results?

RQ2) Can a predictive model based on these critical variables serve as an early warning system for countries seeking to improve their PISA performance?

With this in mind, to answer the relevant research questions in the study, an application of the Random Forest Algorithm was conducted with 24 categorical variables based on PISA (2018) results in science, reading and mathematics and student information for 37 OECD countries. RFA as an ensemble learning method consisting of multiple decision trees has been shown to be effective in both classification tasks and in determining the significance of variables [24]. Especially in complex social systems such as education, where traditional statistical models may not be able to capture nonlinear and high-dimensional relationships, RFA offers significant advantages. Its ability to work with large data sets, tolerate missing values and deal with multicollinearity has led to its increasing use in the field of educational data mining [25]. In this study, RFA was used not only to create powerful predictive models, but also to provide decision makers with evidence-based prioritization of influential variables. As a result of the analysis conducted using the R software, six variables for reading, six for mathematics and eight for science were identified as the most important factors influencing student performance. These variables included access to information technology, weekly hours of instruction in the respective subject area. Economic-social and cultural status, parents’ occupation and level of metacognition were found to be important and common variables for all three areas. Other variables that have an independent effect only on the respective domain are weekly hours of instruction in a foreign language for the reading domain; a sense of competition for the math domain; for the science domain, awareness of PISA was found to be the perception of difficulty and the emotional state related to the act of reading. Consequently, the determination of these variables is very important, especially when it comes to helping teachers and education authorities in the countries concerned to make correct and effective decisions in policy-making processes. In this context, it is clear that the study has a supportive and guiding character.

To support this aim and ensure a coherent flow of research, the structure of the article has been designed to reflect a logical sequence from context to conclusion. The Literature Review section discusses the contribution of PISA data to educational policy, identifies the key variables influencing student achievement and explores the limitations of traditional methods of analysis, providing a sound theoretical and empirical basis for the study. In this context, the importance of machine learning-based approaches --in particular the Random Forest Algorithm (RFA)-- is emphasized and the reasons for its selection are explained. The “Materials and methods” section provides a detailed description of the PISA 2018 dataset used in the study, covering aspects such as the selection of the sample, the coding of the variables and the data pre-processing procedures. It also outlines the reasons for the choice of the RFA and explains how it was implemented in the analysis. The “Results and findings” section presents the variables that have the greatest impact on student achievement in reading, math, and science. The relative importance of these variables and their common or domain-specific effects are interpreted and illustrated using visual representations. Finally, the “discussion and conclusions” section situates the findings within the wider literature and evaluates the methodological contributions of the study. Practical recommendations for policy makers are proposed, and directions for future research are suggested.

Background on PISA

The effects of recent technological developments can be clearly felt in educational research. Especially with the help of current technological methods and techniques such as data mining and machine learning, there are a large number of studies conducted with the aim of evaluating the quality of the education system, accelerating the learning and teaching process, investigating the factors that influence the quality of learning activity or directly affect learning success, and contributing to policy development processes in this direction. Among these studies, those based on the PISA data are particularly noteworthy because their scope and impact are international. Table 1 lists the studies that have been conducted based on PISA data to identify the factors that affect student’ learning outcomes and to suggest how education policies should be designed in this regard.

Table 1: PISA-based academic studies.

If we examine the studies by purpose and year, we find that most of the studies published in the 2010s were conducted with the aim of demonstrating the contributions of PISA so that they can be used as an international basis and target for guiding education policy. It becomes clear that the benefits of the study, such as the improvement of the country’s education policies and activities and the possibility of international comparison, were taken for granted and the studies were conducted with the aim of making direct functional use of the PISA data and investigating the factors that influence student performance.

When the existing literature is examined in terms of the methodology employed, it was observed that previous studies applying the Random Forest Algorithm (RFA) to PISA data [12,20–22] generally focused on a single domain or were conducted using a limited set of variables. In contrast, the present study offered a notable methodological and practical contribution by simultaneously modelling student performance in three domains (reading, mathematics, and science), utilizing a broader variable set, and presenting the variable importance results in a comparative format. The choice of the Random Forest algorithm was not only aligned with the scope and structure of the study but also supported by its strong presence in the literature regarding its high accuracy and compatibility with categorical data. The Random Forest Algorithm (RFA) has outperformed logistic regression in analyzes with categorical data [26]. In a comparative study with 115 binary data sets and 14 different algorithms, it achieved the highest accuracy of all methods tested [27]. It also provided the most successful results compared to CART, C5.0, Naïve Bayes, Linear Discriminant Analysis and K-Nearest Neighbour in similar comparative analyzes [28]. In an analysis with PISA 2018 data, it provided higher accuracy than Hierarchical Linear Modeling [29]. It even outperformed Artificial Neural Networks and Support Vector Machines in predicting student satisfaction [30]. A review of 72 education-related studies conducted between 2015 and 2023 emphasized the strong correspondence with educational datasets and the ability to provide interpretable and meaningful predictions [11]. In light of these findings, the RFA was employed in the present study as a functional tool for identifying variable importance, constructing predictive models, and generating threshold values. Its methodological flexibility and analytical precision played a key role in achieving the core objectives of the study.

Materials and methods

In this section of the study, which was carried out to identify the main variables affecting the successful performance of countries with a score above the OECD average in reading, mathematics, and science, and to create a prediction model for other countries aiming to increase their PISA achievement through these variables, detailed information on the research data and methodological procedure is provided.

Obtaining data

The dataset of the study consists of PISA (2018) scores of 37 OECD countries in science, reading and mathematics and other variables including information about PISA participating students. When the data set was examined, it was seen that there was no missing data. .Subsequently, the non-categorical variables that provide information about the students were categorized. In Table 2, abbreviations and explanations related to the 24 categorical variables obtained within the scope of the PISA exam and to be used within the scope of the study are given.

Table 2: Information on the variables whose effect on PISA scores.

Preparing data for analysis

The data set was subjected to a comprehensive pre-processing process before being analysed. Firstly, the continuous variables were converted into meaningful categories and made usable for the analysis. Distribution characteristics and literature applications were taken into account when categorising the variables. The variables that were considered necessary were coded and insignificant variables were removed. In addition, the names of the variables were standardised according to the abbreviations. Countries’ scores in science, reading and maths were assessed on the basis of the OECD average. 487 points in reading, 489 points in maths and 489 points in science were accepted as the OECD average. Countries that scored above these thresholds were categorised as high-performing countries, while countries that scored below were categorised as low-performing countries. Thus, the performance levels of the countries were categorised into two categories and a suitable structure for the classification algorithms was created.

Random forest application

When analysing the data, the random forest algorithm was preferred in order to determine the significance of the variables and create a prediction model. This algorithm is a machine learning method that works on the basis of decision trees and shows high performance in both classification and regression problems [24]. Random Forest can harmonise with high-imensional data structures, minimize multicollinearity problems and objectively represent the ranking of variable importance [25].

The structure of the categorical data used in the study further emphasizes the advantages of the random forest algorithm. The algorithm in question was favoured not only in terms of predictive success, but also in terms of its ability to define complex interactions between variables. In this context, recent studies such as [20,22,23] emphasise that the RFA method is increasingly used in educational data mining.

In the analysis, the data set was transferred to the programme R (version 4.3) and the Random Forest algorithm was applied via the Boruta package. The Boruta is a wrapping method based on Random Forest and calculates the significance levels of the variables using Z-scores [31]. Thanks to this package, the predictive power of all variables in the data is tested and only the significant variables are included in the model. During the analysis, a “shadow copy” was created for each variable and its Z-scores were compared; only variables with unique information value were included in the model. This increased the accuracy of the model and the degree of explainability.

Visualization of analysis results

In order to better understand the results of the analysis, the importance levels of the variables and their classification successes are shown graphically in Figs 1 and 2. The prominent variables for each domain (reading, mathematics and science) are visualized separately; this facilitates the comparison of the determining factors in different domains. These visualizations support both scientific interpretation and a more effective presentation of conclusions to policy makers.

Most Important Variables in Reading(a), Science(b) and Math(c) Section.

Measurement areas and important variables in PISA 2018.

Finding and results

In this part of the study, findings and results of the data analyses conducted to answer the research questions are presented.

RQ1: The most important variables affecting students’ PISA success

In line with the first research objective of determining the most important variables affecting students’ PISA achievement; standardised PISA scores (science, mathematics, reading) were obtained for OECD countries and a classification variable was created according to the success or failure of the countries. After the class variable was determined, the first findings regarding the results of the analysis carried out to determine which of the variables categorised in the previous stage, which were thought to have an effect on the achievement of the countries, had the most significant effect on achievement are presented in Fig 1 (a–c).

When the box plot, in which green, yellow and red coloured boxes represent important, temporarily important and unimportant variables respectively, is examined in detail, it is seen that 9 variables are important in total. ICTRES (ICT resources), CPERFORLAN (The typically required to attend: number of [class periods] per week in foreign language), CPERWEEK ( in a normal, full week at school, how many [class periods] are you required to attend in total?), METASPAM (Meta-cognition: assess credibility), ESCS (Index of economic, social and cultural status), HISEI (Index highest parental occupational status) for reading; COMPETE (Competitiveness), HISEI, METASPAM, ESCS, ICTRES, CPERWEEK for mathematics; ICTRES, CPERWEEK, PISADIFF (Perception of difficulty of the PISA test), METASPAM, COMPETE, JOYREAD (Joy/Like reading), ESCS, HISEI for science. On the other hand, it is seen that some of the variables have a direct effect on achievement in more than one field, even at different levels. In this respect, the variables whose area of influence and intersections are revealed according to the results of the analyses are presented in the Fig 2 in a more understandable way.

When the Fig 2 is analysed in detail, it is seen that ICTRES, CPERWEEK, ESCS, METASPAM and HISEI variables are determined as the most important variables for science, mathematics and reading sections, despite their different degrees of importance. On the other hand, another remarkable result is that CPERFORLAN variable is important for the reading section, PISADIFF and JOYREAD variables are important for the science section, and COMPETE variable is important for both maths and science sections. In the following, the average importance coefficients of the variables presented in Table 3 regarding the degree of importance in the related field are explicitly included.

Table 3: Coefficient table of determined variables (reading, maths and science).

When Table 3 is analysed in detail; 6 variables are determined as the most important variables for the Reading section. It is seen that the variable with the highest average importance coefficient is CPERWEEK (7.266618) and the variable with the lowest importance coefficient is METASPAM (3.666450). When the coefficient estimates obtained from the average importance coefficient are analysed, the most important variable belonging to the Reading section is CPERWEEK (.844). The variables with the lowest prediction coefficient among the most important variables were METASPAM (.124) and ICTRES (.124).

For the mathematics section, 6 variables were determined as the most important variables. It is seen that the variable with the highest average importance coefficient is COMPETE (7.060205) and the variable with the lowest average importance coefficient is CPERWEEK (4.434677). When the coefficient estimates obtained in this section are analysed, it is seen that the highest coefficient of estimation is COMPETE (.628) and the lowest coefficient of estimation is CPERWEEK (.073).

In the science department, 8 variables were determined as the most important variable. It is seen that the variable with the highest average importance coefficient is METASPAM (8.875383) and the variable with the lowest importance coefficient is COMPETE (3.629179). When the coefficient estimates obtained in this section are analysed, the variable with the highest coefficient of estimation is METASPAM (.550) and the variable with the lowest coefficient of estimation is COMPETE (.029).

RQ2: Proposal for PISA score prediction models

The equations obtained from the coefficient estimates obtained through the model are presented in Model 1 for Reading, Model 2 for Maths and Model 3 for Science.

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

When the total coefficient values to be obtained over the models are analysed, it is predicted that a country with a score of greater than 3,219 from the model obtained from the Reading section can be successful. It is predicted that a country with a score of greater than 2.396 in Maths and greater than 2.732 in Science will be successful. Based on these results, it is predicted that countries can use the results obtained from the models as an early warning system. Thus, the answer to the second research question was obtained.

Discussion and conclusions

This study was conducted using data from PISA 2018 to identify the main factors influencing student performance in high-performing countries among OECD countries and to develop a predictive model based on these variables. Thanks to the RFA applied, not only non-linear relationships but also the interactions and rank order of importance of the variables were analysed. In this respect, the RFA went beyond classical statistical methods and offered more contextual and explanatory analysis possibilities; it recognised patterns that were not visible with traditional models [22,23]. One of the main reasons why the RFA was preferred is that it allows the simultaneous modelling of a large number of independent variables in high-dimensional data sets. This algorithm offers a great advantage, especially in terms of calculating the relative importance of the variables and eliminating multicollinearity problems [25]. However, the RFA algorithm does not directly show causal relationships, but only classifies the predictive power between the variables. Therefore, caution is required when interpreting the results obtained. It is recommended to test possible cause-effect relationships with supporting analyses.

The results of the analysis show that the most important variables influencing students’ PISA success are the following: Access to information technology in all subjects, weekly hours of instruction in each subject, metacognitive awareness, socioeconomic and cultural status, and parents’ occupation. In addition, weekly instruction in a foreign language was found to be significant for reading, competition in maths and science, perception of test difficulty, awareness and students’ emotional engagement in reading.

When examining the variables that influence PISA success, the variable of access to information technology should be mentioned first, which was identified as the most important variable for all areas. Tools and applications based on information technologies have been found to provide students with quick access to relevant sources of information, thereby facilitating their engagement in active learning environments and accelerating their learning processes [11,32–34]. On the other hand, the level of awareness of the use of technological tools and the use of relevant tools for educational purposes are very important for students’ academic success. Among the variables that affect PISA success, economic, social and cultural status and parents’ occupation are usually mentioned together. Socioeconomic status, family cultural structure, parents’ occupation, and their attitudes toward education have been shown to directly influence students’ academic success in numerous studies [35–38]. As a study by [39] shows, Vietnam, despite being one of the poorest countries in the world, performed very well in maths and reading in the 2012, 2015 and 2018 PISA exams compared to many other countries with high economic power. In this regard, it can be said that the economic level or socio-cultural disadvantages of students can be partially overcome with a good coaching and mentoring system. On the other hand [40], have clearly shown in their studies the contribution of parental support to students’ success. There are also various studies in the literature that support the idea that parental support is directly related to the parenting skills of the parents in question [41].

When examining the variable of weekly instructional hours, which is one of the most important variables influencing PISA results, it is generally found that in all three subjects, the length of weekly instructional hours devoted to learning in the subject in question, i.e., the amount of time devoted to learning, has a direct impact on students’ academic performance [42]. state in their study that time management is a variable that should be considered as part of the implementation of a correct educational policy. Similarly, this also applies to the variable of weekly lessons in a foreign language, which was identified as an important variable based on the reading section. The amount and continuity of lessons play a crucial role in improving students’ grammar, vocabulary and reading strategies [43].

Among the main variables that influence PISA success, another variable that attracts attention is metacognition. Metacognition refers to a person’s ability to understand, control and regulate their own thought processes. The relationship between academic success and metacognition is related to students’ ability to manage learning processes more effectively. As [44] emphasise, students with metacognitive skills are more likely to be able to set learning goals, develop strategies, monitor their learning progress and make adjustments when necessary. As [45] suggests, the use of metacognitive processes increases students’ deep understanding, problem-solving skills and critical thinking. This helps them to be more successful in exams, projects and overall academic performance. Metacognition is often positively associated with academic success because it allows students to be more intentional about their learning processes. Metacognition can therefore have a positive impact on students’ academic success as it enables them to utilise learning strategies effectively.

The variable that has a direct impact on PISA success in maths is the variable competition. Competition refers to the rivalry between individuals or groups to achieve certain goals. This competition is often an incentive for better performance, success or the achievement of a specific goal. Maths lessons, where what is right or wrong can be clearly defined, provide a more suitable basis for performance comparisons, which encourages competition. However, in subjects such as psysical education, science and reading, competition is not so obvious as the subject is often based on subjective judgement and collaboration [46]. In these subjects, a variety of skills are often emphasised and the focus is on collaboration and understanding rather than competition.

Other unique variables that directly impact PISA success in science include two variables: emotional state related to the act of reading and perception of the difficulty of the PISA test. The enjoyment of reading supports students’ ability to understand and evaluate scientific texts. Positive reading motivation promotes deeper learning by increasing students’ interest in scientific topics. On the other hand, the perception of the difficulty of the test, i.e., the perception of the quality of the content of the texts read, influences the students’ attitude and their success in the exams. A high perception of difficulty creates stress and anxiety in students. This can lead to poor performance in science classes. Therefore, it is important that students are helped to develop a positive attitude towards the exams. However, the effects of these two variables can vary from student to student. As suggested by [47], students’ individual characteristics, learning styles and motivation levels interact with these factors. It is important for teachers to understand how students respond to these factors and support their individual learning needs in order to increase learning success in science courses.

Strategic recommendations for practitioners

The results obtained provide concrete data that can be utilised to shape educational policy. In this context, improving digital accessibility should not be limited to the provision of hardware, but should also be supported by the development of digital pedagogical content for teachers and students. The integration of metacognitive skills into the curriculum, especially through self-directed learning strategies, will help students to manage their learning processes more consciously and effectively. It is important to increase the number of lessons only quantitatively and to focus this process on learning outcomes by enriching it with skilled content. The competitive environment should be constructively encouraged, especially in areas that require objective assessment, such as maths. However, supporting co-operative learning activities in subjects such as science and reading can lead to more effective outcomes. In addition, counselling services for economically disadvantaged students and support systems that involve the family should be strengthened, which will help to reduce inequalities in education [39].

Limitations and future research

This study has some methodological limitations. First of all, the analysis covers only 37 OECD member countries. A direct generalisation of the results to countries outside the OECD is therefore only possible to a limited extent and should be interpreted taking contextual differences into account. Furthermore, the study was limited to data at student level; variables at teacher, school or system level were excluded from this model. This restricts the scope of the model and limits the impact analysis to a specific level. In future studies, the use of statistical models at multiple levels, the combination of different data sources and the preference for mixed methods supported by qualitative interviews will help to obtain results that are both generalisable and explanatory.

General evaluation

This study has made the multidimensional structures behind the PISA success more visible by utilising the powerful classification and determination of the degree of importance of variables by the RFA, providing a remarkable framework for evidence-based policy making in education. The flexibility that RFA offers in explaining complex interactions that are often ignored by traditional methods is a significant advantage, especially in multivariate systems such as education [20,22]. The insights gained not only provide directional information about the factors that influence student success, but also about which components of the education system to focus on.

Indeed, recent OECD reports [48,49] clearly emphasise the influence of students’ access to digital resources, levels of metacognitive awareness and time spent learning on long-term success. The results of this research also support and strengthen these findings through a machine learning-based approach to analysis. RFA can therefore be evaluated not only as a classification tool, but also as a strategic method that can contribute to the development of more equitable,student-centred and effective educational policies.

In future studies, the application of similar methods in different groups of countries, educational systems or disciplines will both prepare the ground for comparative analyses and allow stronger conclusions to be drawn about the generalisability of this model.

Supporting information

S1 DataDataset containing the variables influencing PISA scores.(XLSX)