Asynchronous Technical Feedback: A Workshop for Training Surgical Instructors

TL;DR

This paper introduces a workshop to train surgical instructors in giving effective feedback on learners' videos for asynchronous learning.

Contribution

The novel contribution is a 60-minute workshop curriculum designed to train instructors in asynchronous technical feedback.

Findings

Forty-six participants attended four workshop iterations across three cities.

85% of participants could identify a barrier to providing feedback after the workshop.

91% of participants could describe a situation to use asynchronous feedback.

Abstract

Asynchronous learning is an efficient method for surgical trainees to gain technical skills by practicing in low-stakes and convenient settings. Effective asynchronous learning requires feedback. Prior work has highlighted the need to train surgical instructors in providing asynchronous technical feedback, as this involves unique skills related to giving feedback on learners’ videos. While many existing curricula focus on optimal feedback practices, there remains a gap with regard to asynchronous technical feedback materials.

Following Kern's six-step approach to curriculum development, we developed a 60-minute workshop, for participants across multiple contexts, on best practices for effective asynchronous technical feedback. We conducted a pilot workshop and then iteratively adjusted the flow and materials for subsequent workshop sessions. We followed survey design principles to create a postworkshop questionnaire evaluating how well the workshop addressed three of the educational objectives.

Forty-six participants attended four iterations of the workshop across three cities. Seven participants attended the pilot session of the workshop, and 39 participants joined the subsequent workshops. Of these 39 participants, 33 (85%) completed the questionnaire. Twenty-eight (85%) of 33 participants indicated that they could state a barrier to providing technical feedback after the workshop, while 30 (91%) of 33 could provide a situation in which to use asynchronous technical feedback. Twenty-seven (82%) of 33 could state a way in which to improve the quality of asynchronous technical feedback.

Educators may use these materials to equip instructors with tools for effectively giving learners the feedback needed for asynchronous technical skill acquisition.

Figures

On the questionnaire (see Appendix F), participants reported that they had increased confidence in providing asynchronous technical feedback after workshop participation compared to before workshop participation (p < .001). Box plots represent the median (interquartile range) confidence rating on a scale of 1 to 5.

Responses to the questionnaire (see Appendix F) indicated that, after the workshop, most participants were able to state a barrier to providing asynchronous technical feedback, were able to provide a situation in which to use asynchronous technical feedback, and were able to state a way in which to improve the quality of asynchronous technical feedback.

Full text

Educational Objectives

By the end of this workshop, participants will be able to:

• 1.Identify characteristics of and barriers to providing high-quality technical feedback.
• 2.Describe evidence-based techniques to provide high-quality technical feedback.
• 3.Recognize technical skills for which asynchronous technical feedback can promote trainee practice and skill acquisition.
• 4.Apply evidence-based feedback techniques and produce high-quality asynchronous technical feedback.

Introduction

Asynchronous learning permits surgical trainees to acquire technical skills at their own pace, when time arises between clinical duties.^1^ Several published reports have described curricula and assessment evidence suggesting the promise of asynchronous technical skill development.^2–6^ However, feedback remains a key component of efficient and appropriate technical skill acquisition.^7^ Previous work has shown the inefficacy of technical skill practice without feedback.^8^ Indeed, the theory of deliberate practice incorporates timely feedback as a key pillar of skill development. This theory posits that inadequate feedback contributes to skill stagnation.^9^

Feedback, whether for synchronous or asynchronous technical skill development, can vary substantially in quality and usefulness. Effective feedback builds on relationships, incorporates both reinforcement and correction, focuses on actionable changes, employs multimodal communication, and is timely.^10–13^ Some of these factors change in asynchronous settings, during which instructors may provide feedback on learners’ standardized patient care interactions, notes, or technical skill videos. In this asynchronous setting, feedback cannot be presented in real-time with an interactive conversation between learner and teacher. As such, there are certain considerations specific to asynchronous feedback. These considerations may include the need for an initial in-person synchronous session and practice with any involved technology.^14,15^ However, several best practices for feedback translate across contexts.

Many existing curricula focus on how to provide feedback in a variety of settings.^16–18^ A number of useful models to facilitate feedback conversations have been proposed in these curricula and other work.^19,20^ Despite the known best practices for technical feedback and the published considerations related to the asynchronous setting, no other curricula–whether in MedEdPORTAL or other sources–have focused on technical feedback in asynchronous settings. As such, instructors providing asynchronous technical feedback may lack formal education on how to implement it. Our prior work specifically underscored the need for training instructors on known best practices in this area, as we observed substantial variation in asynchronous technical feedback quality.^21^

To fill the gap in training procedural and surgical instructors on asynchronous technical feedback skills, we have described the development, implementation, and evaluation of a workshop that builds from technical feedback generally to asynchronous technical feedback specifically.

Methods

Development

Following Kern's six-step approach to curriculum development, we began with problem identification and a needs assessment through a literature review and discussion with 12 surgical learners (Table 1).^22^ We used these findings to create the session educational objectives. Based on the objectives, we designed an interactive session involving a mix of didactic content, participant discussion, and hands-on practice. We reviewed and iteratively refined the educational strategies and materials prior to first implementation. Given our plans to trial this workshop in multiple contexts, we developed a facilitator guide to ensure uniform implementation (Appendix A). The main session materials comprised a set of 28 slides (Appendix B) and three videos for small-group review (Appendices C, D, and E).

Table 1.: Curriculum Development Based on Kern's Six-Step Approach

Implementation

We invited medical students, residents, faculty, and others with interest in asynchronous technical feedback to attend the workshop. We did not include any specific prerequisites for participants. We began the session by introducing the session objectives and asking participants key reflective questions about feedback and its challenges. We then turned to evidence-based and previously published methods to promote effective technical feedback,^10–13,16–20^ framed using the REACT model that we developed to outline best practices (Appendix B). The REACT model emphasizes the importance of several considerations for technical feedback: relationships, encouragement, action, channels/multimodality, and timeliness. After discussing these evidence-based strategies for providing technical feedback, we focused on asynchronous technical feedback. We highlighted the affordances and constraints of asynchronous technical feedback and emphasized how the previously discussed strategies apply to the asynchronous setting. We also had a group discussion about the logistics of using asynchronous feedback. After this discussion, we conducted a large-group practice, in which participants watched a video of a student performing a technical skill and then discussed feedback strategies together. Following the large-group practice, we divided participants into smaller groups for guided independent practice. Small groups reviewed a set of shared videos and practiced providing feedback using an online platform. Finally, we led a debrief about the small-group practice session and emphasized key session takeaways.

Serial Iteration

We first piloted the curriculum with a group of seven participants (representing faculty, residents, and medical students). We chose to start with a small pilot to identify major areas for workshop changes. At the end of the pilot, we discussed the session objectives, flow, and materials with participants. We made several changes to the workshop at this point, including adjusting the materials to better align with the objectives and changing the order of the session to better enable participant understanding. We conducted a second, third, and fourth session of the workshop following the initial pilot. Participants from these sessions had fewer suggestions for workshop improvement, though we did make some adjustments to the example videos and slides to improve clarity. We also trialed the workshop in 45-minute, 60-minute, and 90-minute versions, based on varied opinions about workshop length from participants.

Evaluation and Analysis

We created a postworkshop questionnaire for participants to evaluate the first three workshop educational objectives. Following the survey design process outlined by Artino, La Rochelle, Dezee, and Gehlbach,^23^ we first performed a literature review, and then conducted a focus group with 14 surgical education research fellows to understand and define the constructs in our questionnaire. We synthesized prior published questionnaires with findings from our focus group to develop items relevant to the workshop.^24–26^ We discussed items with experts among the author group, including experts in survey design and experts in asynchronous technical feedback. We piloted the questionnaire during the pilot session of the curriculum and refined the items. We did not retain responses from this pilot session because we made subsequent changes to the questionnaire. We distributed the final questionnaire to all participants in the second, third, and fourth iterations of the curriculum. This questionnaire contained three sections and 14 items focusing on attitudes, retrospective and postworkshop confidence, knowledge around technical and asynchronous technical feedback, and participant information (Appendix F).

We generated descriptive statistics from the results of the questionnaire and used a Wilcoxon signed-rank test to compare paired ordinal data. We used a Wilcoxon rank-sum test to compare medical student and non–medical student participant groups. For statistical analysis, we used Stata/IC 16.1 software for Mac (College Station, TX).

Context

We implemented this workshop in 2024 for educational and surgical audiences across multiple sites. Facilitators for this workshop all had some prior experience providing technical feedback, though they had varied experience with asynchronous technical feedback. While preparing for the workshop, all facilitators worked together to further practice and refine their skills with asynchronous technical feedback.

We conducted the workshop in classrooms with projectors at sites in three cities: San Francisco, California (United States), Santiago (Chile), and Orlando, Florida (United States). For small-group activities, we asked participants to bring computers with internet access capability. During these sessions, we used the free online platform C1Do1 to allow participants to practice providing asynchronous technical feedback. Only the session leader needed to create an account to use this platform for the workshop. Other online platforms (e.g., Practice XYZ) have similar functionality. Regardless of the specific platform used, a program to facilitate multimodal feedback helped with hands-on practice. We hosted two of the workshop iterations as offerings at conferences and two of the workshop iterations as independent events.

The University of California, San Francisco Institutional Review Board exempted this curricular development and evaluation from review (UCSF IRB no. 24-41011, 2024).

Results

Forty-six participants attended four iterations of the workshop across three cities. Seven participants attended the first pilot session of the workshop, and 39 participants joined the subsequent workshops. Of these 39 participants, 33 (85%) completed all or part of the questionnaire and were included in the final analysis. Participants represented a range of roles, including Medical students (n = 14), Residents (n = 10), attending (n = 7), and Other (n = 2). Participants who reported a specialty (n = 15) worked most commonly in general surgery (n = 11).

The questionnaire first assessed confidence and attitudes about the workshop. Though many participants (n = 17, 52%) reported having received no or very little prior instruction in providing asynchronous technical feedback, most participants (n = 18, 55%) stated that they were confident in providing asynchronous technical feedback before the workshop. The preworkshop confidence level, rated on a scale of 1–5 (1 = extremely unconfident, 2 = somewhat unconfident, 3 = neither confident nor unconfident, 4 = somewhat confident, 5 = extremely confident), was higher among medical student participants (median preworkshop confidence 4) than among all nonmedical student participants (median preworkshop confidence 3; p < .001). Seventeen (52%) of 33 participants reported that their confidence in providing asynchronous technical feedback increased following the workshop (p < .001; Figure 1). Of the seven participants who reported feeling unconfident in providing asynchronous technical feedback before the workshop, all seven reported an increase in confidence by the end of the workshop. Medical student and non–medical student participants reported similar levels of postworkshop confidence (median postworkshop confidence 4 in both groups; p = .54). Twenty-four (73%) of 33 participants reported that they would recommend the workshop to others, and 25 (76%) of 33 reported that they were likely to use the skills they learned in the workshop.

On the questionnaire (see Appendix F), participants reported that they had increased confidence in providing asynchronous technical feedback after workshop participation compared to before workshop participation (p < .001). Box plots represent the median (interquartile range) confidence rating on a scale of 1 to 5.

The questionnaire next assessed our educational objectives by determining participants’ postworkshop knowledge of feedback barriers and the use of asynchronous technical feedback (Figure 2). Twenty-eight (85%) of 33 participants were able to state a barrier to providing technical feedback after the workshop, while 30 (91%) of 33 were able to provide a situation in which to use asynchronous technical feedback. Twenty-seven (82%) of 33 were able to state a way in which to improve the quality of asynchronous technical feedback.

Responses to the questionnaire (see Appendix F) indicated that, after the workshop, most participants were able to state a barrier to providing asynchronous technical feedback, were able to provide a situation in which to use asynchronous technical feedback, and were able to state a way in which to improve the quality of asynchronous technical feedback.

In narrative comments (summarized in Table 2), participants reflected on several strengths of the workshop, including the easy-to-use REACT mnemonic for improving asynchronous technical feedback quality, the clarity of the presented information, and the hands-on nature of the session. Participants also suggested areas for improvement. The most common suggestion—proposed by 11 respondents in free-text responses—was related to either insufficient or excess time in the workshop. Most of those who commented on workshop duration participated in the 45-minute version of the session and felt it to be too short. Others suggested that the small-group practice be shifted to individual practice to give every participant the chance to provide asynchronous technical feedback.

Table 2.: Summary of Narrative Comments From Workshop Participants

Discussion

In this report, we have described a newly developed workshop to teach procedural and surgical instructors how to provide effective asynchronous technical feedback. The workshop can be easily implemented with the use of few materials and fills a gap in available materials to train instructors in an increasingly important skill. Most participants reported increased confidence in providing asynchronous technical feedback after attending the workshop. Furthermore, most participants were able to display relevant knowledge related to key aspects of the first three workshop educational objectives.

This workshop adds to the existing literature promoting feedback skills in medical education. Prior studies have outlined easy-to-use models for instructors to facilitate effective feedback, often for giving feedback in synchronous settings.^16–20^ Other work has emphasized the efficacy of programs training novices–such as some of those who participated in our workshop–to provide technical feedback.^27^ Our report contributes to this literature through its specific focus on providing asynchronous technical feedback, an increasingly important type of feedback given changes in training structures and the importance of equipping trainers with effective and diverse feedback strategies. Asynchronous technical feedback practice represents a promising way for trainees to attain skills in a low-stakes setting and at convenient locations and times for both trainees and faculty.^28,29^

As part of the evaluation and feedback step of Kern's approach to curriculum development, we adjusted the workshop across its four iterations.^22^ Following the initial pilot, we adapted the workshop in accordance with participants’ suggestions regarding materials and flow. Many participants had (occasionally conflicting) suggestions regarding workshop duration. After collating all responses and running the workshop in 45-, 60-, and 90-minute versions, we identified the 60-minute version as providing an appropriate balance.

There are a number of limitations and lessons learned. First, we measured knowledge-related outcomes for our first three educational objectives through a postworkshop questionnaire. We did not measure changes in participants’ behavior or results. Also, we did not directly measure the fourth objective. Participants practiced applying evidence-based feedback techniques to sample trainee videos during the workshop, though we did not assess their work or measure retention. Furthermore, we implemented this workshop in academic centers at three sites. The workshop is most likely to be relevant to similar academic settings. Additionally, we did not require participants to have prior background with technical feedback. As such, there was a wide range in participants’ roles and experiences, with more than one-third of participants being medical students. This created the challenge of balancing basic and advanced topics tailored to the specific participants. We found this topic to be relevant to many types of participants, particularly given the increasing use of peer feedback in synchronous and asynchronous technical feedback. Peer feedback allows both the feedback giver and recipient, even when inexperienced, to benefit from the feedback process.^21,30^ Nonetheless, use of suggested prerequisites to better frame the workshop for potential participants may address this problem. Alternatively, hosting the workshop in smaller sessions but with more homogenous groups may allow for a more focused level of discourse. For example, participants with limited technical experience (e.g., medical students) may be better able to review basic technical skill videos. Based on the group of participants, those implementing this workshop may choose to focus on just one of the small-group videos (Appendices C, D, and E). Finally, some participants had mixed responses with regard to recommending the workshop and changes in confidence following the workshop. Of note, many participants started the workshop with a high level of confidence, which may have limited any subsequent increase in confidence level postworkshop. Medical students reported higher preworkshop confidence levels on average. Confidence as an outcome measure has multiple limitations, as an individual may be confident but lacking in skill. This may have additionally been limited given that the questions regarding participants’ confidence levels pre- and postworkshop were both asked after the session. Future work could evaluate the efficacy of feedback given by session attendees.

Next directions include incorporating the workshop into annual training for procedural and surgical instructors who provide asynchronous technical feedback. Overall, as asynchronous technical feedback practice becomes increasingly prevalent, equipping procedural and surgical instructors with the skills to provide feedback in this setting will be essential.

Appendices

• Facilitator Guide.docx
• Slides.pptx
• Small Group Video 1.mp4
• Small Group Video 2.mp4
• Small Group Video 3.mp4
• Questionnaire.docx

All appendices are peer reviewed as integral parts of the Original Publication.