Evaluation of Tracheal Intubation Skill Acquisition in Junior Residents During Anesthesiology Training: An Observational Study

TL;DR

This study evaluates how junior medical residents improve their tracheal intubation skills during anesthesiology training using a standardized evaluation method.

Contribution

The study introduces a structured observational approach to assess skill acquisition in tracheal intubation among junior residents.

Findings

Junior residents showed significant improvement in tracheal intubation scores after two months of training.

The time required for tracheal intubation decreased significantly following the training period.

Factors like department affiliation and skill satisfaction influenced residents' satisfaction with anesthesiology training.

Abstract

Background: Anesthesiology training during the residency program is a requirement for junior residents; however, due to only one or two months of anesthesiology training, the experiences of junior residents tend to vary from individual to individual. Furthermore, the decision of skill acquisition level is left to the discretion of a clinical teacher. In most studies, researchers evaluate tracheal intubation based on the number of experiences, success rate, and time required for tracheal intubation. It is difficult to standardize because conditions for tracheal intubation rely on information about a patient’s airway and individual surgical cases. For this reason, it is also difficult to determine clear criteria for tracheal intubation proficiency. We hypothesized that skill acquisition level can be clarified by evaluating tracheal intubation training of junior residents.

Methods: An observational study was conducted on 31 junior residents who participated in the junior residency training in the department of anesthesiology. This research was conducted at our simulation center between November 2021 and March 2023. On the first day of anesthesiology training, participants performed a series of procedures, from tracheal intubation to the connection of artificial respiration, following the orientation for airway management with a simulator. During these procedures, each participant’s movement and laryngoscopy maneuver within the mouth were recorded on video; later, specialist physicians evaluated each performance using an evaluation sheet. On the last day of the two-month training period, participants repeated the same procedures as on the first day of anesthesiology training. Total score based on the evaluation sheet was selected as the primary endpoint, and the time required for tracheal intubation and scores by objectives were selected as the secondary endpoints. Furthermore, questionnaires were administered before and after anesthesiology training. Multiple regression analysis was used to analyze questionnaire results, which determined if there is a causal relationship between variables affecting participants’ satisfaction in anesthesiology training.

Results: The scores were compared between before and after anesthesiology training. When compared to the scores before anesthesiology training, the total score showed a significant increase (8.9±8.5, 95% CI (5.8-12.1), p<0.01), and the score in the time required for tracheal intubation showed a significant decrease after anesthesiology training (58.9±49.7, 95% CI (40.6-77.2), p<0.01). According to the questionnaire results from before and after anesthesiology training, a factor that affected an increase in the score of tracheal intubation skills was the time required for tracheal intubation (adjusted R²=0.1, Akaike information criterion (AIC)=87.5). Factors that affected participants’ satisfaction in anesthesiology training were affiliated department, number of skills experienced, and skills’ satisfaction (adjusted R2=0.56, AIC=68.5).

Conclusions: We verified the hypothesis that tracheal intubation skill acquisition following anesthesiology training can be visualized by using the evaluation sheet. In the future, it is necessary to consider management that can experience various skills based on the points of airway management education in the junior residency training. In addition, it is also necessary to establish an education system that can increase interest in anesthesiology training.

Figures

Comparison of the difference in total score and time taken before and after trainingn=31; paired t-test

Correlation heatmap between the goals in the evaluation form and each indicator

Path diagram: factors that affect the level of satisfaction with anesthesia trainingβ: regression coefficient

Tables

Table 1. Endotracheal intubation skill evaluation chartEach item is scored on a five-point scale, with a total of 20 items for a maximum of 100 points. Decimal scores are also acceptable.Scoring criteria are as follows: One point: unable to perform independently; critical errors such as esophageal intubation (beginner level); Two points: able to perform with physical assistance; Three points: able to perform with verbal guidance (general competency level); Four points: able to perform independently; Five points: performs skillfully (specialist level)

Goal		Score
Goal 1: To be able to assume a position suitable for tracheal intubation
1	Position the patient's head appropriately (sniffing position, inserting pillows for head and shoulders, etc.)	/5
Goal 2: To be able to properly insert a laryngoscope
2	Holding the laryngoscope correctly
3	Cross-finger opening
4	Insertion from the right corner of the mouth, the tongue is moved to the left and reaches the midsection.
5	Recognizes the epiglottis valley and can insert the blade
6	Appropriate application of force to the epiglottis valley
	Subtotal	/25
Goal 3: Obtain an appropriate airway field of view
7	Blade position is appropriate, neither deep nor shallow
8	Blade is in the midsection
9	Lift the epiglottis without tongue protrusion
10	The initial laryngeal deployment angle is appropriate (about 45 degrees)
11	Vocal cords are visible
12	Number of laryngeal deployments (Note! If you can do it once, you get 5 points)
	Subtotal	/30
Goal 4: To be able to correctly insert an intubation tube into the trachea and start artificial respiration
13	No contact with other tissues during tube insertion
14	Insert the tube to the proper depth
15	Inject cuff and confirm placement in trachea (chest elevated, etCO2, auscultation, cuff leak)
16	Initiate manual or mechanical ventilation
17	Proper skin fixation with tape
18	Number of times the tube is inserted (Note: If you can do it once, you get 5 points)
	Subtotal	/30
Goal 5: To avoid trauma during intubation
19	Not applying excessive force during laryngoscopy or tube insertion
20	Laryngoscope does not pinch lips or cause dental damage
	Subtotal	/10
	Overall score	points
	The time required	seconds

Table 2. Comparison of pre- and post training evaluationsn=31; paired t-test; * p<.01; †p<.001

	Evaluations	Pre-training	Post training	Mean±SE	95%CI
Primary outcome	Total score of Goals 1-5  (points)	58.1	67.0	8.9 ± 8.5 †	5.8 - 12.1
Secondary outcome	Goal 1: Positioning	2.9	3.5	0.6 ± 0.1 †	0.4 -0.9
	Goal 2: Insertion of a laryngoscope	13.0	15.8	2.8  ± 0.5 †	1.7 - 3.7
	Goal 3: Laryngeal view	17.5	19.8	2.3 ± 0.7 *	0.9 - 3,7
	Goal４: Intubation and connect to a ventilator	19.5	21.7	2.1 ± 0.5 †	1.1 - 3.1
	Goal 5: Injury	5.2	6.2	1.1 ± 0.2 †	0.6 - 1.5
	Time taken (seconds)	251.7	192.8	58.9 ± 49.7 †	40.6 - 77.2

Table 3. The factor affecting the satisfaction level of anesthesia trainingDependent variable: the satisfaction level of anesthesia training, *p<.05Goal 4: Intubation and connection to the ventilator skill; AIC: Akaike's information criterion; β: regression coefficient; VIF: variance inflation factor

Parameters	β	SE	p-value	95%CI		VIF
Department of specialty	-0.46	0.19	0.004*	-1.01	-0.21	1.41
Number of skills	0.4	0.08	0.01*	0.05	0.38	1.47
Satisfaction with skills	0.72	0.2	0.001*	0.56	1.39	1.53
Goal 4	-0.22	0.03	0.1	-0.12	0.01	1.17
R	0.78
R2	0.61
Adjusted R2	0.56
AIC	68.5

Full text

Introduction

Two months of anesthesiology training are mandatory in the residency program for junior residents. On the first day of training, the residents receive airway management orientation using a simulator, followed by tracheal intubation training in surgical anesthesia. Anesthesiology training also has the advantage of providing experience in other advanced technical skills, such as epidural anesthesia and central venous catheter insertion. Anesthesiology training also provides a foundation in safety management, including the prevention of medication errors and non-technical skills with multidisciplinary staff [1,2].

Tracheal intubation, typically among technical skills, is an essential skill not only during surgery but also during emergency care, intensive care, and lifesaving in general wards. Although surgical anesthesia provides daily experience with tracheal intubation, few departments outside of anesthesiology offer educational training. In addition, according to the American Heart Association’s cardiopulmonary resuscitation guidelines, tracheal intubation is an important technique for proper airway clearance and oxygenation during resuscitation, as well as for the prevention of aspiration pneumonia. Proficiency in this requires appropriate initial training and experience [3].

Anesthesiology training is short, only lasting one to two months, and the extent of experience beyond this is likely to vary. The number of times the procedure has been previously performed, the success rate, and the time required for tracheal intubation are used as indicators to evaluate proficiency in this technique. However, since these vary greatly depending on the characteristics of patient airways and specific surgical cases, it is difficult to define a general standard for proficiency in this technique. As a result, clear standards are currently lacking.

Using tracheal intubation skill acquisition as a decision, we referred to the educational guidelines of the Japanese Society of Anesthesiologists (Basic Course in Airway Management) [4] and the assessment tools of the American Council for Accreditation of Medical Education's Anesthesia Education Standards Form (Milestones) [5,6] to create an assessment chart tailored to our hospital environment. This chart was used to conduct a pilot study in 2021. In the present study, we verified the reliability and validity of this assessment chart. On the first and last day of training, tracheal intubation and ventilation training on a simulator using the assessment chart was conducted twice to document technical progress. In addition, an anesthesiology questionnaire was administered before and after training was used to document the number of procedures performed and the residents’ technical scores, as well as to allow residents to evaluate their satisfaction with their training.

The aim of this study was to evaluate skill acquisition in anesthesiology residents during intubation training using a simulator. The hypothesis is that residents’ skills and evaluation scores will be higher after training than before. A questionnaire will be administered pre- and post-training to comprehensively evaluate the causal relationship between the number of times the procedure is performed and skill acquisition, as well as resident satisfaction with the training content. The acquisition of appropriate tracheal intubation skills by residents is key to saving patients’ lives and improving medical safety in the future.

This study was presented as a conference abstract at the American Society of Anesthesiologists Annual Meeting, October 19, 2024.

Materials and methods

The study participants were comprised of 31 residents undergoing initial training in anesthesiology. The study was conducted at the simulation center of St. Luke's International Hospital, Tokyo, Japan, from November 2021 to March 2023.

Training consisted of a practical exercise using the Laerdal Airway Management Simulator (Laerdal Medical AS, Stavanger, Norway) on the first day, followed by a lecture-style explanation of the technique. The consenting residents were asked to perform a series of actions, from mask ventilation to tracheal intubation and ventilator connection. The training session was recorded from two angles: the residents' actions during the sequence from mask ventilation to tracheal intubation and connection to the ventilator, and the intubation procedure in the oral cavity. The training instruments consisted of a Macintosh laryngoscope with a disposable bronchoscope glued to it and an Ambu® bronchoscope monitor (Ambu A/S, Ballerup, Denmark).

We ensured that the faces of the residents were not captured in the images, taking care to prevent individual identification. The video recordings were subsequently scored by a third-party anesthesiologist using an evaluation chart. On the last day of training, residents were asked to perform the same simulator tracheal intubation procedure as on the first day. The session was recorded and subsequently scored by the same anesthesiologist. The final score was based on the successful completion of each of the five goals in the assessment chart (Table 1).

The primary endpoint was the overall score based on the assessment chart, while the secondary endpoint was the time required for the procedure.

Statistical analysis to confirm differences between the pre- and post-training scores was performed using a paired t-test. In addition, the overall evaluation and satisfaction scores were analyzed by multivariate analysis. The following is a pre- and post-training questionnaire that was used for the analysis: Please indicate your level of interest in anesthesiology on a numerical scale of one to 10. Please indicate your level of satisfaction with learning the technology on a numerical scale of one to 10. Please indicate your level of overall satisfaction with the anesthesiology training on a numerical scale of one to 10.

Multiple correlations were then calculated between the five goals in the assessment chart and each indicator, as well as the relationships between the pre- and post-training scores. This study was approved by St. Luke's International Hospital's research ethics review committee (approval number: 21-R087).

Results

In this study, tracheal intubation and ventilatory training were filmed before and after the first and last day of training, and the overall score and time required to complete the procedure were compared based on the goals on the assessment chart. Among the 31 participants, 15 were male and 16 were female, with 15 belonging to the internal medicine unit and 16 to surgery. Twenty were in their first year of initial training, and 11 were in their second year, with the second year being their first rotation in anesthesiology.

The overall score on the primary endpoint assessment chart increased significantly after training (8.9±8.5, 95% CI (5.8-12.1), p<0.01), and the time required from head positioning to intubation and ventilator connection decreased significantly (58.9±49.7, 95% CI (40.6-77.2), p<0.01) (Figure 1).

Comparison of the difference in total score and time taken before and after trainingn=31; paired t-test

From these results, the two hypotheses set forth are affirmed. Comparing the scores pre- and post training, these increased significantly for all five goals on the assessment chart (p<0.01). In addition, a heatmap of the correlation between each item showed a strong correlation between Goals 2 and 3 (r=0.83, p<0.05) (Table 2, Figure 2).

Table 2: Comparison of pre- and post training evaluationsn=31; paired t-test; * p<.01; †p<.001

Correlation heatmap between the goals in the evaluation form and each indicator

Multiple regression analysis was performed on each factor affecting the overall tracheal intubation score and the residents’ satisfaction with the anesthesiology training. The results showed that the only factor affecting the overall score was the time required for the procedure (adjusted R^2^ = 0.1, Akaike information criterion (AIC) = 87.5). The prediction accuracy of the model was judged to be quite low. Factors that influenced training satisfaction included the department in which they were affiliated, the number of times the procedure had been performed previously, satisfaction with the skills experienced, and Goal 4 (adjusted R^2^ = 0.56, AIC = 68.5) (Table 3, Figure 3).

*Table 3: The factor affecting the satisfaction level of anesthesia trainingDependent variable: the satisfaction level of anesthesia training, p<.05Goal 4: Intubation and connection to the ventilator skill; AIC: Akaike's information criterion; β: regression coefficient; VIF: variance inflation factor

Path diagram: factors that affect the level of satisfaction with anesthesia trainingβ: regression coefficient

Discussion

In this study, the acquisition of tracheal intubation skills by residents during their initial hospital training was evaluated using an assessment chart to document their progress and satisfaction. Many hospitals do not currently have any established standardized criteria to determine tracheal intubation skill. As a result, evaluation is typically based solely on the judgment of a specialist. However, this study proposes using an evaluation chart to evaluate and monitor the acquisition of tracheal intubation skills, including an overall score on the assessment chart after training and a record of the time required to complete the procedure. A comparison of the scores before and after training showed that the scores increased significantly across all five assessment criteria (Goals 1-5), suggesting that goal-setting is necessary for skill acquisition. In particular, the results showed a strong correlation between Goals 2 and 3, indicating a close relationship between laryngoscopy and laryngeal deployment.

Previous reports on anesthesiology training have suggested that 57 instances of tracheal intubation are needed to achieve a 90% success rate for this technique [7]. Other studies have also reported that between 43 [8] and 30 cases [9] are needed to achieve an 80% success rate. This is not likely to be achieved in a two-month training program. The number of actual attempts may be more critical than the duration of the training program. Many studies use the number of times the procedure has been performed, the success rate, and the time required to conduct the procedure as criteria for evaluating tracheal intubation skills [10,11].

However, since management in real patients varies depending on the patient's airway status and specific surgical procedures, it would be shortsighted to evaluate success solely based on the ratio of tube insertion and the time required. There is a need to comprehensively evaluate several actions, starting with head positioning of the patient, confirming that the tube is securely inserted, and ending the procedure only after smooth ventilation is confirmed. To this end, we consulted previous protocols studied in similar settings [6,12,13]. Anesthesiology guidelines from around the world may prove to be useful references, but cannot be reduced to specifics. Multidisciplinary procedure evaluation tools, such as Objective Structured Assessment of Technical Skills (OSATS) [14,15] and Direct Observation of Procedural Skills (DOPS) [16-18], score each item on a five-point scale and facilitate nominal and ordinal scale analyses. Current assessment charts analyze up to 20 items using an overall integer score. In the future, the use of larger training cohorts will be essential to solidify the foundations of this assessment while making necessary modifications to develop the best training guidelines.

The results of the training sessions were analyzed using multivariate analysis to identify factors that play important roles in improving the skills and training satisfaction of residents. The time required to complete the procedure was selected as a factor influencing the improvement of the overall score. However, because of the model's low accuracy results, it was not possible to identify factors affecting the overall score. Factors affecting satisfaction with anesthesiology training included the department of junior residents, the number of times the procedure was performed, satisfaction with the skills, and Goal 4. In particular, satisfaction with the skills and the number of times the procedure was performed had a significant impact on satisfaction with the training, suggesting that these two factors play important roles during anesthesia training. Since anesthesiology is the only field in which residents learn advanced skills needed to treat critical patients and respond to emergencies [19], it is clear that the residents had high expectations [20-24].

Japan has been experiencing a significant shortage in physicians since the 2000s due to government healthcare policies, including a shortage in anesthesiologists that continues to this day [25-26]. With day-to-day clinical work taking precedence, the establishment of a specialized training system may still take time. They promoted team medicine, which is now a policy, and collaborated with physicians and nurses while teaching residents and conducting research. Based on this collaborative approach, this study seeks to contribute to the construction of a fulfilling medical education system and the creation of an environment that increases resident satisfaction with anesthesiology training. The purpose of this study was to evaluate skill acquisition during tracheal intubation training conducted by anesthesiology residents.

The limitations of this study include insufficient rigor in the selection and exclusion criteria of participants, inadequate training of evaluators, and a lack of rigor in the research environment. In the future, we aim to establish a standardized environment to ensure reproducibility of techniques and assessments, enabling their use even in patient care sessions.

Conclusions

This study empirically evaluated the hypothesis that the introduction of a skill assessment chart for simulator tracheal intubation training would increase resident evaluation scores after the training compared to before training, reduce the time required to complete the procedure, and facilitate a visual aid for the degree of skill acquisition.

Questionnaires administered to the residents pre- and post training allowed us to evaluate the causality between the number of times the procedure was performed and the satisfaction with training among the participants. As a future direction, we are considering excluding the average number of attempts required to acquire sufficient intubation skills and using this as a target value for subsequent training sessions. Based on the results of this study, we hope to develop an education system that allows for skill-based learning and an attractive training program for medical residents.