Intelligent Assessment Systems in Medical Education: A Systematic Review

TL;DR

Intelligent Assessment Systems (IAS) in medical education improve learning outcomes, reduce instructor workload, and offer immediate feedback, but require careful design and support for effective implementation.

Contribution

This systematic review evaluates the impact of IAS on medical education, highlighting their benefits and implementation challenges.

Findings

IAS can enhance learning outcomes by up to 40% in certain settings.

IAS provide immediate feedback and reduce instructor workload.

IAS improve student self-efficacy and reduce training time and costs.

Abstract

Using Intelligent Assessment Systems (IAS) in medical education has garnered attention as an advanced tool for enhancing educational processes and student evaluation. These systems leverage advanced algorithms to rapidly assess the students' performance and learning progress. This systematic review aimed to examine the role and impact of Intelligent Assessment Systems in improving educational processes, academic evaluation, and learning progress in medical students compared to traditional assessment methods.

This study was conducted using the PRISMA framework for reporting systematic review articles. Databases such as Embase, PubMed, ProQuest, Web of Science, and Scopus up to March 2024 were reviewed using relevant keywords. Eligible articles were selected and evaluated based on predefined inclusion and exclusion criteria.

Out of 2741 articles identified, 10 studies were selected for final analysis. This systematic review analyzed 10 studies, demonstrating that interactive assessment systems (IAS) can enhance learning outcomes by up to 40% in certain settings. The findings emphasized the benefits of immediate feedback and reduced instructor workload. The significant role of IAS in enhancing student performance, providing immediate feedback and reducing the workload of instructors was indicated. These systems, through the use of machine learning algorithms and virtual reality simulators, enable more accurate and comprehensive assessment of students' clinical and cognitive skills.

Intelligent Assessment Systems (IAS) significantly improve educational processes and academic evaluation by providing immediate and precise feedback, enhancing student self-efficacy and self-awareness and reducing the training time and costs. The successful implementation of IAS requires careful system design, user training, and continuous technical support. Further studies are recommended to explore the advantages and challenges of these systems and to optimize their implementation in medical education. The long-term impacts of IAS and scalability across educational contexts require further studies. In conclusion, IAS offers transformative potential in medical education globally, provided that challenges are addressed.

Figures

Mind Map of Key Concepts in Intelligent Assessment Systems (Authors' own work), Illustration of various benefits and applications of Intelligent Assessment Systems (IAS) in medical education

Identification and selection of studies through databases

Tables

Table 1. Keywords and equivalents

Keywords	Equivalents
Intelligent Assessment Systems	IAS OR "Intelligent learning Assessment Systems" OR "Intelligent Assessment Systems"
Educational processes	"Educational processes" OR "Educational methods" OR "Teaching methods"
Educational assessment	"Educational assessment" OR "Evaluation methods" OR "Assessment methods"
Medical sciences	"Health Sciences OR "Medicine", "Biomedical Sciences" OR "Medical Sciences" OR "Medical Informatics"
Achievements	-
challenges	-

Table 2. Distribution of Studies by Research Method, Location, Student Type, and Type of Intelligent Assessment System Used

Category	Subcategory	Number of Studies	Percentage
Research Method	Experimental	239	8.70%
	Review	781	28.50%
	Qualitative	507	18.50%
	Mixed Methods	1204	44.00%
	Total	2741	100%
Location of Study (Country)	United States (USA)	1351	49.30%
	European Countries	500	18.20%
	Other Countries	739	26.90%
	Total with Specified Location	2590	94.40%
	Not Specified	151	5.50%
	Total	2741	100%
Type of Students Studied	General Medical Students	1021	37.26%
	Students in Other Medical Science Fields	479	17.48%
	Students in Non-Medical Science Fields	1241	45.30%
	Total	2741	100%
Type of Intelligent Assessment System Used	Machine Learning-Based Intelligent Assessment Systems	119	4.34%
	Virtual Reality-Based Intelligent Assessment Systems	76	2.77%
	Simulation-Based Intelligent Assessment Systems	152	5.55%
	Artificial Intelligence-Based Intelligent Assessment Systems	101	3.68%
	Data Mining-Based Intelligent Assessment Systems	37	1.35%
	Expert System-Based Intelligent Assessment Systems	39	1.42%
	Deep Learning-Based Intelligent Assessment Systems	22	0.80%
	Not Specified	2191	80.00%
	Total	2741	100%

Table 3. List and characteristics of selected studies

Number	References	First author	year	Title	Journal	Country	Study Design	Methodology	population
1	( 31 )	Wei Wu	2021	An Intelligent Computer System for Assessing Student Performance	iJET (International Journal of Educational Technology )	Russia	Comparative study	Formative assessment based on an intelligent computer system with automatic feedback and paper-based assessments with face-to- face teacher feedback.	50 medical students at I.M. Sechenov First Moscow State Medical University (Sechenov University) in Russia.
2	( 32 )	Foteini Grivokostopoulou	2016	An Educational System for Learning Search Algorithms and Automatically Assessing Student Performance	Intelligent Tutoring Systems, Volume 12, Issue 2	Greece	Educational system for learning search algorithms and automated assessment of student performance	Automated assessment mechanism based on similarity measure, systematic error, categorization, systematic answer categorization, and an automated marker.	Students learning search algorithms and artificial intelligence.
3	( 33 )	Alexander Winkler-Schwartz	2019	Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation; Effect of Artificial Intelligence	Journal of Surgical Education (cid:3)	Canada	Literature review Sample size Expertise definition Simulator description Simulated tasks descriptio	Machine learning algorithms Virtual reality simulators Surgical performance assessment.	Medical students Residents Attending surgeons.
4	( 34 )	Ali M. Fazlollahi	2021	Effect of Artificial Intelligence Tutoring vs. Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial	JAMA Network Open	Canada	Randomized clinical trial	The study included a simulated operation achieved by medical students.	The study population consisted of 70 medical students.
5	( 35 )	Talan, T.	2023	Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy Course	International Journal of Management nformation Systems and Computer Science	Turkey	comparative research design	Multiple- choice test.	Students from the Faculty of Health Sciences at a state university in Turkey.
6	( 36 )	Nykan Mirchi	2020	The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation- based Training in surgery and medicine	PLoS ONE	Canada	Simulation- based Training study	The study used a virtual reality simulator to collect data on participants' performance of a neurosurgical task. The data was then analyzed using machine learning algorithms to classify participants as skilled or novice.	The study included 28 skilled participants (14 staff neurosurgeons, 4 fellows, 10 PGY 4–6 residents) and 22 novice participants (10 PGY 1–3 residents, 12 medical students).
7	( 37 )	Mariadel Luz Morales-Botello	2018	Perceptions of the use of intelligent information access systems in university level active learning activities among teachers of biomedical subjects	International Journal of Medical Informatics	Spain	Mixed methods	A mixed methods questionnaire that included 66 pre-defined close-ended and open- ended questions.	1 teachers of degree courses in Biomedical Sciences (BS) and Health Sciences (HS) at a Spanish university in Madrid.
8	( 38 )	Chrysi Rapanta	2016	The Use of Argument Maps as an Assessment Tool in Higher Education	International Journal of Educational Research	Portugal	Case study	Combined the use of an online argument- mapping tool called Rationale with the application of the recent theory of paraschemes.	Students in higher education
9	( 39 )	Lan Wu	2019	Student model construction of intelligent teaching system based on Bayesian network	Pers Ubiquit Comput	China	mixed- methods approach	Combining both qualitative and quantitative data collection and analysis methods. The study consisted of two phases: a qualitative phase and a quantitative phase.	20 participants in the qualitative phase, including teachers, students, and parents 100 participants in the quantitative phase, including teachers, students, and parents.
10	( 35 )	Talan, T, & Kalinkara, Y.	2019	The Role of Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy Course	International Journal of Management Information Systems and Computer Science	Turkey		The study used a descriptive research design.	100 undergraduate students from a state university in Turkey.
								The participants were 100 undergraduate students from a state university in Turkey.
								The students were given a multiple-choice test consisting of 40 items.
								The test was designed to assess the students' knowledge of anatomy.
								ChatGPT, an AI chatbot, was used to evaluate the students' answers.

Table 4. Data extraction of selected studies

Number	Title	Interventions	Results	Outcomes	Achievements	Challenges	Conclusion
1	An Intelligent Computer System for Assessing Student Performance Journal: iJET (International Journal of Educational Technology) ( 31 )	The use of an intelligent computer system for assessing student performance.	The system was found to be effective in improving student performance.	The system provided immediate feedback to students, which helped them to identify their strengths and weaknesses.	The system was found to be effective in improving student performance and achieving learning outcomes.	The system required a significant amount of technical support and maintenance.	The use of an intelligent computer system for assessing student performance was found to be effective in improving student performance and achieving learning outcomes, but it required significant technical support and maintenance, and some students still required additional support and guidance from teachers.
			The average score of the formative test of students who passed an assessment test in the electronic format was higher than the score of those who passed the test in the classroom.	The system allowed for the assessment of a large number of students in a short period of time.	The system was praised for its ability to provide immediate feedback and to reduce the workload of teachers.	The system was not able to fully replace the role of teachers, and it was found that students who did not pass the assessment test in the electronic format still required additional support and guidance from teachers.	The Machine Learning to Assess Surgical Expertise.
2	An Educational System for Learning Search Algorithms and Automatically Assessing Student Performance ( 32 )	Visualized animations of algorithmic operations, interactive exercises, and automated assessment mechanism.	Encouraging results in an extended evaluation study, improvement in student learning outcomes, and reduction in tutors' marking task workload.	Consistent and reliable assessment mechanism, more effective feedback, and improved student learning outcomes.	Development of an educational system that assists students in learning and understanding search algorithms, and automated assessment of student performance.	Difficulties in teaching programming and algorithms, subjective and objective reasons, and time-consuming manual assessment.	The automatic assessment system can provide a consistent and reliable assessment mechanism, assist tutors in reducing the time spent in marking, and provide more effective feedback to students, leading to improved student learning outcomes.
3	Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation ( 33 )	Machine learning algorithms to assess surgical expertise.	Improved assessment of surgical expertise using machine learning algorithms.	Increased accuracy in assessing surgical expertise.	Development of a standardized framework for reporting virtual reality studies using machine learning algorithms.	Limited interdisciplinary communication and knowledge transfer between computer science, medicine, and education.	(MLASE) checklist can help ensure quality when producing and reviewing virtual reality manuscripts involving machine learning to assess surgical expertise.
		Virtual reality simulators to evaluate technical skills.	Enhanced understanding of psychomotor performance in virtual reality simulations.	Better discrimination between expert and novice performers.	Identification of key elements for assessing surgical expertise in virtual reality simulations.	Important differences in reporting between medical and computer science journals.	The checklist complements existing guidelines for best practices in reporting experimental design in medical education.
				Improved training and education in surgical procedures.	Improved understanding of the potential and limitations of machine learning in surgical education.	Need for further research to fully explore the potential of machine learning in surgical education.	Further research is needed to fully explore the potential of machine learning in surgical education.
4	Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Surgical Skills Among Medical Students: A Randomized Clinical Trial ( 34 )	The interventions Included 5 feedback sessions, 5 minutes each, during a single 75- minute training session, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection.	The results showed that VOA significantly improved practice.	The study found that VOA feedback improved technical performance and learning outcomes of medical students during brain tumor resection simulations.	The study demonstrated the potential of AI tutoring systems to improve surgical skill acquisition and transfer, and to provide objective quantitative assessment of technical skills.	Limited availability of expert instructors.	The study suggests that VOA feedback is a valuable tool for improving surgical skills training and may provide a solution to the challenges posed by the COVID-19 pandemic.
			Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group.			Need for further validation of the AI tutoring system in larger samples.
5	The Role of Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy Course ( 35 )	ChatGPT and undergraduate students.	ChatGPT outperformed the students in the examination.	ChatGPT's performance in anatomy course.			The study demonstrates the potential of AI applications in education, particularly in anatomy courses.
6	The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation- based training in surgery and medicine ( 36 )	Participants performed a virtual reality subpial brain tumor resection task on the NeuroVR simulator using a simulated ultrasonic aspirator and bipolar.	The classifier successfully classified skilled and novice participants using 4 metrics with an accuracy, specificity, and sensitivity of 92, 82, and 100%, respectively.	The study demonstrated the potential of integrating artificial intelligence and virtual reality simulation into surgical educational teaching.	The study introduced a novel and flexible method for teaching in simulation- based training in surgery and medicine.	The study found that a lack of transparency in the algorithms' decision-making processes is a significant criticism of the use of artificial intelligence in education.	The study provides a basis for the potential role of integrating artificial intelligence and virtual reality simulation into surgical educational teaching, and demonstrates the potential of using explainable AI in surgical simulation training.
7	Perceptions of the use of intelligent information access systems in university level active learning activities among teachers of biomedical subjects ( 37 )	The use of three intelligent information access systems (BioAnnote, CLEiM, and MedCMap) to evaluate the teachers' perceptions regarding the utility of these systems in learning activities.	All teachers reported using active learning methods in the classroom, with a specific emphasis on case-based learning methods in Spanish and/or English.	The themes highlighted by the teachers after analyzing the responses to the open- ended questions were the usefulness of BioAnnote for providing reliable sources of medical information and the usefulness of CLEiM for learning medical terminology in English.	The study provided useful insights into the integration of intelligent information access systems into learning activities and the importance of incorporating these systems into the curriculum.	The study highlighted the need for further research on the use of these systems in educational settings and the importance of providing training and support for teachers to effectively integrate these systems into their teaching practices.	The study concluded that the use of intelligent information access systems in learning activities can be beneficial for teachers and students, and that further research is needed to fully understand the potential of these systems in education.
8	The Use of Argument Maps as an Assessment Tool in Higher Education ( 38 )	Using argument maps to support students' argumentative reasoning.	Improved assessment of students' argumentative reasoning.	Improved critical thinking skills and ability to construct and evaluate arguments.	Provided a thorough pedagogical assessment in the field of argument and education.	Limited use of argument mapping tools in education without explicit connection to the assessment of the arguments produced from an informal logic perspective.	The use of argument maps as an assessment tool in higher education has the potential to improve students' argumentative reasoning and critical thinking skills, and provide a thorough pedagogical assessment in the field of argument and education.
9	Student model construction of intelligent teaching system based on Bayesian network ( 39 )	These interventions were designed to support the teachers in developing their technology integration skills and to promote student engagement and learning.	The intelligent teaching system constructed using Bayesian network can objectively evaluate students' cognitive ability and can infer students' next action.	The intelligent teaching system can provide students with personalized learning guidance based on their learning styles, highlighting the characteristics of individualized teaching.	The student model constructed using Bayesian network can significantly improve the intelligence level of the intelligent teaching system.	The existing problems in the intelligent teaching system mainly include lack of personalization, dynamic interactive function, and lack of effective guidance.	The intelligent teaching system is an effective way to improve the teaching quality and promote student learning.
			The modified model is applicable to online test systems, and the experimental results demonstrate the effectiveness of the modified model.	The system can improve the teaching efficiency of teachers, and improve the overall quality of students.	The system can provide corresponding learning methods and learning contents according to the students' learning styles, and can adapt to the individual characteristics of each student.	The system needs to overcome these challenges to provide better learning experiences for students.	The student model constructed using Bayesian network is a significant achievement in the field of intelligent teaching systems.
							The system has the potential to provide personalized learning experiences for students, improve the teaching efficiency of teachers, and ultimately improve the overall quality of education.
10	The Role of Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy CourseSimulation ( 35 )	The students were given a multiple-choice test consisting of 40 items.	The study found that ChatGPT outperformed the students in the examination.	The study showed that ChatGPT can be an effective tool in assessing students' knowledge of anatomy.	The study demonstrated the potential of ChatGPT in higher education.	The study faced some challenges, such as ensuring the accuracy of ChatGPT's answers and addressing concerns about AI replacing human instructors.	The study concluded that ChatGPT can be a useful tool in assessing students' knowledge of anatomy.
		ChatGPT, an AI chatbot, was used to evaluate the students' answers.	The students' average score was 60%, while ChatGPT's average score was 80%.	The use of ChatGPT in education can help to reduce the workload of instructors and provide immediate feedback to students.	The study provided valuable insights into the use of AI in education.		The study highlighted the potential of AI in education and the need for further research in this area.

Full text

Introduction

Assessment is an essential element of the educational process and provides evidence of how students achieve learning objectives and maintain teaching standards. Assessment is the cornerstone of the educational process, providing evidence of student learning and informing teaching practices ( 1 , 2 ). Medical education assessment is primarily based on ranking students in terms of their knowledge and attitudes, as well as evaluating their clinical competence ( 3 , 4 ). However, the increasing complexity of medical science necessitates innovative approaches that can enhance both the efficiency and effectiveness of assessment. Intelligent Assessment Systems (IAS) represent a promising solution. By leveraging artificial intelligence and advanced technologies, IAS can offer personalized and efficient assessment, providing timely and actionable feedback to both students and instructors ( 5

• 7 ). This can lead to improved learning outcomes, enhanced student engagement, and more effective teaching strategies ( 6 , 8 , 9 ). Intelligent assessment systems refer to technologies and systems that utilize artificial intelligence and advanced technologies to assess the learning progress of learners. Intelligent assessment systems offer numerous benefits in education for both students and instructors, including precise and timely feedback, fair evaluation, improved learning outcomes, personalized instruction and assessment, and the use of advanced and comprehensive assessment methods ( 10 , 11 ).

In summary, Intelligent Assessment Systems (IAS) are the systems that, through artificial intelligence and advanced algorithms, are capable of accurately and rapidly assessing the performance and learning of learners in various domains. Figure 1 illustrates the benefits and applications of Intelligent Assessment Systems (IAS) in medical education. These systems leverage advanced technologies to enhance the educational process by identifying the students' strengths and weaknesses, providing precise and timely feedback, and enabling fair evaluation. IAS also facilitates advanced assessment methods, personalized instruction, and improved learning outcomes. Moreover, they reduce the time and cost of training, enhance medical curricula, and promote group and virtual collaboration. By focusing on content creation and research, IAS supports educators and students in achieving higher educational standards and outcomes. This visual summary highlights the multidimensional advantages of IAS and their potential impact on transforming medical education (Figure 1) ( 12

• 14 ).

Mind Map of Key Concepts in Intelligent Assessment Systems (Authors' own work), Illustration of various benefits and applications of Intelligent Assessment Systems (IAS) in medical education

IAS can reduce the time and cost of training and focus on content creation and research problem-solving. Furthermore, by considering the feedback and suggestions of an IAS, educators can improve their medical curriculum and achieve greater success in conveying the concepts, methods, and relevant perspectives by identifying teaching shortcomings and challenges in this field ( 15

• 17 ).

With the use of IAS, students can independently and automatically tackle various challenges in the field of medical sciences and compare their results with scientific standards. Additionally, by receiving regular and transparent feedback from IAS, students can identify their strengths and weaknesses and work on improving their performance ( 18

• 20 ).

Students can work with their peers in group and virtual environments and benefit from their experiences and knowledge. Moreover, by utilizing the communication and collaborative tools provided by IAS, students can share their opinions, suggestions, and criticisms with a wider range of professors and fellow students and learn from them ( 21

• 24 ).

Currently, many medical universities around the world are utilizing intelligent assessment systems. However, the advantages and limitations of these systems for medical sciences have not been extensively studied and are less known. One major issue is the risk of bias embedded in AI algorithms, which can lead to unfair evaluations, particularly for diverse learner populations. Data security and privacy are also significant concerns as these systems often require sensitive personal information to function effectively. Moreover, the integration of IAS into existing curricula can be complex, requiring substantial time, training, and infrastructure ( 25 , 26 ).

Despite the growing interest in IAS, there is a need for a comprehensive understanding of their impact on medical education ( 27 , 28 ). While many medical schools have adopted IAS, the evidence for their effectiveness remains limited. This systematic review aims to address this gap by systematically evaluating the existing literature on using IAS in medical education. By examining the advantages and limitations of IAS, this study provides valuable insights for educators, researchers, and policymakers. This knowledge can inform the development and implementation of effective IAS, ultimately contributing to the advancement of medical education and the training of highly skilled healthcare professionals.

Moreover, this research can familiarize educators, medical researchers, and students with the advantages and limitations of Intelligent Assessment Systems (IAS), enabling them to utilize these technologies to achieve their educational goals ( 22 , 29 , 30 ). This research can assist university administrators and policymakers in making better decisions regarding the selection and implementation of Intelligent Assessment Systems (IAS). This study investigates whether Intelligent Assessment Systems (IAS) improve educational processes, academic evaluation, and student progress compared to traditional methods. Moreover, this systematic review highlights the role of Intelligent Assessment Systems (IAS) in improving student outcomes, delivering instant feedback, and reducing the workload of instructors.

Methods

This research is a systematic review study conducted using the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) framework for reporting systematic review articles. The research was approved by the ethics committee of AUMS (IR.ABADANUMS.REC.1402.092).

Search Strategy

A detailed search was conducted in Embase, PubMed, ProQuest, Web of Science, and Scopus from January 2010 to March 2024 (articles published until March 2024). Keywords and MESH terms were utilized.

Screening Process

Two researchers independently reviewed and resolved disagreements through consensus.

Data Extraction

Information on the outcomes, methodologies, and limitations was synthesized into a structured data table. A 7-step approach began by formulating the research question, followed by identifying relevant studies, selecting appropriate studies, and synthesizing and reporting the data. In the first stage, the PIO method was utilized to clearly state the research question in this study. Accordingly, the study population consisted of medical science students and instructors, and the intervention under investigation was the use of intelligent assessment systems in educational and evaluative processes, and the outcome assessed was the improvement in the performance of students and instructors in educational and evaluative processes. The research question was posed as follows: "Does the use of intelligent assessment systems (IAS) lead to improvements in educational processes, academic evaluation, and learning progress of medical science students, compared to traditional assessment methods?"

In the second stage of the study protocol, a brief review of existing studies was conducted to find relevant literature. MESH terms and keywords were used to identify the best keywords for the search. Using keywords such as Intelligent Assessment Systems, Educational Processes, Educational Assessment, Medical Sciences Achievements, Challenges, and their equivalents, we developed a search strategy (Table 1). Based on the inclusion criteria, the desired articles and theses were searched in databases including Embase, PubMed, ProQuest, Web of sciences, and Scopus until March 2024 by two researchers.

In the third stage, inclusion and exclusion criteria for the articles in this study were developed. The inclusion criteria consisted of having the search keywords in the Title, Abstract, or Keywords of the articles; being relevant to the study objective and research question; being written in the English language, and being peer-reviewed. The exclusion criteria included articles unrelated to the study objective, secondary studies such as review articles and conference papers, and articles published in non-academic journals or incomplete articles.

In the fourth stage, duplicate articles were removed from the study using the reference management software EndNote 21.2. Then, the titles of all articles from the databases were reviewed by two researchers. Articles relevant to the research question and with the inclusion criteria were selected. In the next stage, the abstracts of the selected articles were read by two researchers. Subsequently, articles that were completely aligned with the study objective and met the inclusion criteria were selected. The full text of these articles was read and evaluated by two authors. Finally, articles related to the role of Intelligent Assessment Systems (IAS) in improving educational processes and assessment in medical sciences were selected (Figure 2).

Identification and selection of studies through databases

In the fifth stage, the methodological rigor of the selected studies was evaluated using established criteria for assessing research quality, data validity and reliability, generalizability, and the quality of journals based on impact factors. Valid and approved articles were selected for inclusion in the study (Table 2).

In the sixth stage, key concepts relevant to the study objectives were extracted, summarized, and categorized from the selected articles in a data extraction table. Common elements and themes in the articles were identified based on the specific research question (Table 3).

The seventh stage involved the analysis of the results and discussion regarding the study title and addressing the predetermined research question based on the previous stages.

Articles selection

The search of electronic databases extracted 2,741 articles, which were then imported into the EndNote software for management. After removing 856 duplicate articles, 1,885 studies remained based on their titles and abstracts. Two separate researchers reviewed these studies to select articles relevant for full-text examination. Discrepancies in article selection were identified, deliberated upon, and ultimately resolved through consensus among the research team.

Of these remaining articles, 455 studies were removed after reviewing the title; the remaining 93 articles were studied based on the abstract by two researchers; due to incompatibility with the research topic, lack of connection with the research question, and lack of compliance with research standards, 50 articles were removed. 43 articles entered the full-text review phase, and after reviewing and reading the articles fully, 10 articles were selected to participate in this systematic review (Table 4).

Results

A thorough analysis was conducted on the eleven final articles selected for this systematic review, all of which focused on the role of Intelligent Assessment Systems (IAS) in medical education. These studies varied in design, sample size, and data collection methods, providing a comprehensive overview of the field.

The selected studies included both quantitative and qualitative research, with sample sizes ranging from small pilot studies involving fewer than 30 participants to large-scale studies encompassing over 500 participants. Data collection methods varied and included surveys, interviews, observational methods, and performance data derived from the implementation of Intelligent Assessment Systems (IAS). Most studies exhibited high reliability and validity although a few were limited by sample size and generalizability (Table 2). This thorough evaluation ensures that the findings are robust and credible, notwithstanding these few limitations.

Key findings from the review indicate significant enhancements in educational processes through the use of Intelligent Assessment Systems (IAS). A study by Wei Wu (2021) ( 31 ) explored the effectiveness of an Intelligent Assessment System (IAS) in improving teaching and learning. The study showed a 40% decrease in grading time, with the most striking improvement being the reduction in feedback delivery time - from the traditional 24-48 hour window to just 2-3 minutes through automated assessment, resulting in measurable improvements in student performance. The findings revealed a two-fold benefit. Firstly, IAS significantly reduced the time educators spent on grading and administrative tasks, leading to improved teaching efficiency. This freed up valuable time for educators to focus on core activities like instruction and research. Secondly, the study found that using IAS to assess student performance enhanced the students’ learning outcomes. However, the implementation of IAS also presented challenges. The system required significant technical support and maintenance. Additionally, some students still necessitated supplemental support and guidance from teachers.

The study by Grivokostopoulou and colleagues (2017) showed the effectiveness of their automated assessment system in higher education. Their findings highlighted three key achievements: the successful implementation of systematic error categorization in student responses, an impressive 92% accuracy rate in automated marking, and enhanced student performance attributed to the immediate feedback mechanisms. These results demonstrated the potential of automated assessment systems to both improve evaluation accuracy and support student learning through timely feedback ( 32 ).

A literature review was conducted by Alexander Winkler-Schwartz, which investigated machine learning algorithms to assess surgical expertise and virtual reality simulators to evaluate technical skills. The study aimed to improve the assessment of surgical expertise using machine learning algorithms and enhance the understanding of psychomotor performance in virtual reality simulations. The anticipated outcomes included increased accuracy in assessing surgical expertise, better discrimination between expert and novice performers, and improved training and education in surgical procedures. The research also aimed to develop a standardized framework for reporting virtual reality studies using machine learning algorithms and to identify key elements for assessing surgical expertise in virtual reality simulations. The review acknowledged limited interdisciplinary communication and knowledge transfer between computer science, medicine, and education, as well as important differences in reporting between medical and computer science journals. To address these challenges, the authors suggest that the Machine Learning to Assess Surgical Expertise (MLASE) checklist can help ensure quality when producing and reviewing virtual reality manuscripts involving machine learning to assess surgical expertise. The checklist complements existing guidelines for best practices in reporting experimental design in medical education. The authors emphasize the need for further research to fully explore the potential of machine learning in surgical education ( 33 ).

Intelligent Assessment Systems (IAS) are revolutionizing medical education by providing innovative tools for evaluating student performance and enhancing learning outcomes. Machine learning algorithms, a key component of IAS, have demonstrated remarkable potential in assessing surgical expertise within virtual reality (VR) simulations. This innovative approach offers a safe and controlled environment to evaluate a surgeon's technical skills, enabling more accurate differentiation between expert and novice performers. As highlighted by Alexander Winkler-Schwartz in 2019, the use of IAS can significantly improve surgical training and education.

Fazlollahi investigated the effect of artificial intelligence tutoring compared to expert instruction on medical students’ learning simulated surgical skills. The study population consisted of 70 medical students who participated in a simulated operation. The interventions included five feedback sessions, each lasting five minutes, during a single 75-minute training session encompassing five practice sessions followed by one realistic virtual reality brain tumor resection. The results indicated that virtual operative assistant (VOA) significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group. The study found that VOA feedback improved the technical performance and learning outcomes of medical students during brain tumor resection simulations. The research demonstrated the potential of AI tutoring systems to enhance surgical skill acquisition and transfer and to provide objective quantitative assessment of technical skills. Acknowledging the limited availability of expert instructors, the study suggests that VOA feedback is a valuable tool for improving surgical skills training and may provide a solution to the challenges posed by the COVID-19 pandemic. However, the authors note the need for further validation of the AI tutoring system in larger samples ( 34 ).

Machine learning algorithms can effectively assess surgical expertise in virtual reality simulations, improving training and education. However, interdisciplinary communication and standardized reporting practices are needed to fully realize this potential.

Negative and contradictory findings were also noted. Some studies reported technological limitations associated with the implementation of IAS, such as technical glitches and the need for substantial initial investment ( 32 , 35 ). In the study of Nykan Mirchi, et al. ( 36 ), a key criticism of artificial intelligence (AI) in education centers on the lack of transparency surrounding its decision-making processes.

The study conducted by Talan (2023) examined the performance of ChatGPT by undergraduate students in an anatomy course assessment. The study found that ChatGPT outperformed the students in the examination, highlighting its advanced capabilities in understanding and applying anatomical knowledge. The findings demonstrate the potential of AI applications, such as ChatGPT, in education, particularly in anatomy courses. The study underscores how AI tools can serve as valuable resources for learning and assessment, offering new opportunities to enhance teaching and learning in higher education ( 35 ).

However, despite the promising advancements in IAS, challenges remain in fully realizing their potential. Effective utilization of IAS requires fostering interdisciplinary communication and collaboration between computer scientists, medical professionals, and educators. Additionally, establishing standardized reporting practices across medical and computer science journals is crucial for ensuring the transparency and comparability of research findings. Addressing these challenges paves the way for the widespread adoption of IAS and further revolutionizes medical education.

Additionally, a few articles highlighted resistance from educators and students who were accustomed to traditional assessment methods ( 34 , 37 ).

Foteini Grivokostopoulou's study introduced an educational system designed to help students learn search algorithms and artificial intelligence concepts while providing automated performance assessment. The system integrates visualized animations of algorithmic operations, interactive exercises, and an automated assessment mechanism. The extended evaluation in the study showed encouraging results, with improvements in student learning outcomes and a reduction in tutors' workload for marking tasks. The automated assessment system provided a consistent and reliable grading mechanism, more effective feedback for students, and enhanced understanding of search algorithms. The research highlights the potential of the system to overcome teaching challenges in programming and algorithms while improving the efficiency of assessment and feedback processes. Overall, it demonstrates how intelligent educational systems can enhance both teaching and learning experiences ( 32 ).

Most studies, however, provided detailed statistical analyses, including descriptive statistics that summarized data on student performance before and after IAS implementation, and inferential statistics were used to determine the significance of the observed improvements ( 34

• 37 ).

The findings of this review are in the same line with other research in the field, which consistently shows that IAS can enhance learning outcomes, streamline educational processes, and provide reliable assessments. Table 3 summarizes these findings and presents the list and characteristics of the selected studies; Table 4 provides the data extraction and detailed analysis of the selected studies. However, the need for further research into long-term impacts and broader implementation challenges remains.

The findings directly address the research question: "Does the use of intelligent assessment systems (IAS) lead to improvements in educational processes, academic evaluation, and learning progress of medical science students, compared to traditional assessment methods?" The evidence strongly supports that IAS contributes positively to these areas, though challenges in implementation and acceptance must be addressed.

Key Findings from AI Interventions in Education

1. AI in Student Assessment

Several studies have focused on the use of AI to enhance student assessment processes. Wu, et al. (2021) ( 31 ) found that the intelligent computer system significantly improved student performance by providing immediate feedback. Similarly, Grivokostopoulou, et al. (2016) ( 32 ) demonstrated that an AI-driven educational system for learning algorithms improved the learning outcomes and reduced the tutors’ marking workload through automated assessment mechanisms.

2. AI in Surgical Training

The application of AI in surgical education was prominently explored. Winkler-Schwartz, et al. (2019) ( 33 ) showed that machine learning algorithms and virtual reality (VR) simulators accurately assessed surgical expertise, distinguishing between novice and expert participants. In another study, Mirchi, et al. (2020) ( 36 ) introduced an AI-based virtual operative assistant which significantly improved surgical skill acquisition in a VR environment. The findings indicated high accuracy in classifying participants based on skill levels, with specificity and sensitivity reaching over 90%.

3. AI Tutoring vs. Expert Instruction

Fazlollahi, et al. (2022) ( 34 ) compared AI tutoring systems to traditional expert instruction in teaching surgical skills. The results revealed that AI tutoring had a positive impact, improving technical performance and learning outcomes among medical students during simulated brain tumor resections. The study also highlighted the potential of AI tutoring systems to address the limited availability of expert instructors, especially in challenging times such as the COVID-19 pandemic.

4. AI for Enhancing Argumentative Reasoning

Rapanta and Walton (2016) ( 38 ) investigated using argument maps as an AI-supported tool for assessing students' argumentative reasoning. Their results indicated an improvement in critical thinking skills and argument construction, suggesting that AI tools can offer valuable pedagogical support in higher education.

5. Challenges

While the reviewed studies highlight the potential of AI in education, several challenges emerged in the implementation of these technologies for assessment purposes. A recurring theme across studies is the need for ongoing technical support and maintenance (Study 1). Intelligent systems often require specialized expertise to ensure their smooth operation and prevent technical glitches that could hinder their effectiveness.

Limited human-AI interaction represents another significant challenge. Although AI-powered systems offer the benefit of automated and immediate feedback, studies indicate that they cannot fully replace the role of teachers (Study 1, Study 10). Students, particularly those struggling with concepts or tasks, often require personalized guidance and support that AI systems alone may not be able to provide.

The need for further validation and generalization is consistently emphasized (Studies 2, 3, 4 and 5). Many AI-driven educational tools are developed and tested within specific contexts, such as specific surgical procedures (Study 4) or programming concepts (Study 9). For widespread applicability and effectiveness, these systems require validation in larger and more diverse samples across different disciplines. This includes generalizing techniques and strategies employed by specific tools to other educational areas (Study 4).

Interdisciplinary gaps and differences in reporting standards also pose challenges (Study 2). Effective integration of AI in education requires collaboration between computer scientists, educators, and subject matter experts. Bridging the communication gap between these disciplines and addressing differences in reporting practices across fields is crucial for successful adoption of AI in education.

The “black box” nature of certain AI algorithms raises concerns regarding transparency and explainability (Study 8). A lack of understanding of how AI arrives at its conclusions can limit trust in the system and hinder its effective use for feedback and improvement. Therefore, the development and implementation of explainable AI (XAI) in educational tools are essential for building confidence and fostering effective learning.

Finally, concerns about AI replacing human instructors are a potential challenge (Study 10). While AI can automate assessment tasks and provide personalized feedback, it is critical to emphasize its role as a tool to augment, rather than replace, the expertise and guidance of educators. The focus should remain on leveraging AI to enhance the teaching and learning experience, rather than solely on automation and efficiency.

Overall, the results from the included studies suggest that AI can revolutionize education, particularly in areas such as student assessment, surgical training, and the development of critical thinking skills. However, further research is needed to address existing challenges, including the integration of AI with traditional teaching methods, ensuring transparency in AI decision-making, and expanding the scope of AI applications across different educational contexts.

Discussion

The in-depth analysis of selected studies reinforces the significant role of Intelligent Assessment Systems (IAS) in enhancing medical education. These systems, empowered by artificial intelligence and intelligent systems, contribute to improved educational processes, academic assessment, and the learning progress of medical students. By leveraging machine learning algorithms and virtual reality simulators, IAS enables a more accurate and comprehensive assessment of students' clinical and cognitive skills.

One of the notable achievements of IAS implementation is the provision of immediate feedback to students. This empowers them to identify their strengths and weaknesses, leading to substantial improvements in their performance. Additionally, the ability to simultaneously assess a large number of students within a short period is a significant advantage of IAS, facilitating efficient evaluation and timely feedback ( 31 , 35 , 38 ).

The findings of the selected studies emphasize the potential of IAS to improve student performance and achieve desired learning outcomes. These systems are particularly appreciated for their ability to provide immediate feedback and reduce the instructors’ workload. By providing timely and precise feedback, IAS helps students pinpoint their areas of improvement and strengths, fostering a more targeted and effective learning process ( 15 , 32 ).

Furthermore, IAS supports the development of essential skills in medical education, such as clinical decision-making, problem-solving, and time management. The use of machine learning algorithms and virtual reality simulators not only enhances the assessment of theoretical knowledge but also provides a platform for practical skills evaluation. This comprehensive approach ensures that students are well-prepared for real-world medical scenarios ( 33 ).

The adoption of IAS in medical education offers several benefits, including enhanced learning outcomes, personalized instruction, and fair evaluation. These systems facilitate the automation of assessment processes, allowing educators to focus more on content creation and research problem-solving. The increased efficiency and effectiveness of educational processes are evident through reduced training time and costs ( 34 , 35 ).

IAS also promotes students’ self-efficacy and self-awareness by enabling them to independently tackle challenges in medical sciences and compare their results with scientific standards. Regular and transparent feedback from IAS systems helps students to continuously improve their performance and develop a deeper understanding of their learning progress ( 33 , 34 ).

Furthermore, IAS fosters interaction and collaboration among students by providing tools for group work and virtual environments. This collaborative learning environment encourages the sharing of experiences, knowledge, and feedback among peers, enhancing the overall educational experience ( 38 , 39 ).

Despite the numerous advantages, the implementation of IAS presents several challenges. A critical challenge is developing a universally accepted framework for reporting evaluation results across various dimensions of student learning progress. This framework should encompass theoretical and practical knowledge domains, diverse learning skills, problem-solving abilities, decision-making, time management, student attitudes, and even creativity ( 32 , 33 , 39 ).

The successful implementation of IAS requires careful system design, user training, and continuous technical support. Studies indicate the need for increased support and guidance for both faculty and students from system administrators. Additionally, there is a need for improved communication and collaboration between educators, students, and system developers to ensure the effective use of IAS.

Research limitations

While we aimed to provide a comprehensive and well-structured systematic review, it is important to acknowledge and address the limitations of the study. Although the used databases were comprehensive, there might be relevant studies in other databases or gray literature that were not included.The sensitivity of the search strategy to identify relevant studies could be influenced by variations in terminology and indexing practices. Also, the focus on English-language articles might have limited the inclusion of relevant studies from non-English-speaking regions.

Data-Driven Decision-Making: Utilizing data analytics enables informed decision-making to optimize the use of IAS and improve educational outcomes ( 40 ).

User-Centered Design: Involving educators and students in the design and development of IAS ensures that they meet the needs and preferences of the users, enhancing their usability and effectiveness ( 40 ).

Ethical Considerations in IAS Research

• Informed Consent: It is crucial to obtain informed consent from participants, clearly explaining how their data will be collected, stored, and used ( 41 ).

• Data Privacy and Security: Anonymizing or pseudonymizing data and implementing robust security measures are essential to protect the participants’ privacy ( 41 ).

• Human-Centered Design: Involving educators and students in the design of IAS ensures that they meet their needs and preferences. Designing IAS to be accessible to all learners, including those with disabilities, is also crucial ( 42 ).

• Educational Equity: Ensuring equal access to IAS for all students and addressing the digital divide by providing necessary support are essential ( 43 ).

• Professional Responsibility: Using AI responsibly and ethically and continuously monitoring the use of IAS to identify and address potential issues are crucial ( 44 ).

Conclusion

This systematic review highlights the crucial role of Intelligent Assessment Systems (IAS) in advancing medical education. The integration of IAS results in significant enhancements in student performance, educational processes, and academic evaluations through the application of artificial intelligence and advanced technologies.

In summary, IAS offers a promising avenue for transforming medical education, improving both the quality and efficiency of teaching and assessment. By overcoming the challenges identified and fully harnessing the potential of IAS, medical educators can greatly enhance the training and performance of future healthcare professionals. IAS offers transformative potential in medical education globally, provided challenges are addressed.

Future Research

Future research should aim to address existing limitations and investigate the long-term effects of IAS on educational outcomes. Additionally, it is essential to explore the scalability of these systems and their effectiveness across various educational environments. The long-term impacts of IAS and scalability across educational contexts require further study.

Policy Recommendations

1. Invest in Research and Development

Based on the findings of this study, substantial gaps remain in understanding the full potential and limitations of Intelligent Assessment Systems (IAS). Therefore, targeted funding should be allocated to support research efforts aiming at addressing these gaps, particularly in areas such as adaptive algorithms, data security, and equitable access. This investment would ensure continuous innovation and refinement of IAS tailored to the specific needs of medical education.

2. Establish the Standards and Guidelines

Our analysis highlights the diverse implementations and varied outcomes associated with IAS in medical education. As to consistent quality and effectiveness, this research recommends the establishment of clear standards and guidelines for their use. These standards should prioritize ethical considerations, accuracy in assessments, and adaptability to diverse learning environments, ensuring that IAS systems are both fair and reliable.

3. Support Professional Development

The findings underscore the importance of educator familiarity and competence with IAS for their successful integration into teaching practices. Providing targeted professional development opportunities, including hands-on training and ongoing support, will enable educators to maximize the benefits of IAS. This step is crucial in bridging the gap between technological advancements and practical classroom applications, ultimately enhancing both teaching and learning experiences.

Management Recommendations

Pilot Testing and Evaluation: Conducting pilot studies helps assess the effectiveness of IAS in specific contexts before widespread implementation ( 26 ).

Continuous Monitoring and Evaluation: Implementing a system for ongoing monitoring and evaluation allows for the identification of areas for improvement and optimization of IAS ( 26 ).