Neurosurgery Trainees’ Perspectives of the ‘Mock Viva’

TL;DR

Neurosurgery trainees find mock exams valuable for learning about the final exam format and improving performance through feedback, but junior trainees are less engaged due to stress and limited experience.

Contribution

This study identifies how mock viva exams can be optimized to improve trainee engagement and learning outcomes.

Findings

Mock viva exams provide insight into the FRCS exam format and performance feedback.

Junior trainees perceive limited educational value due to stress and lack of experience.

Observing others' assessments helps trainees understand their own performance relative to peers.

Abstract

This article was migrated. The article was marked as recommended.

The ‘viva voce’ examination has been a tradition in surgical training for over a century, and remains a key element of the ‘Fellowship of the Royal College of Surgeons’ (FRCS) Examination, which neurosurgery trainees must pass to complete their training. The aims of this study were to evaluate the educational value of an annual ‘mock viva’ as a formative assessment tool for neurosurgical trainees during their eight year training programme, to identify barriers to participating in the mock viva and to explore how these might be attenuated to increase levels of engagement
.

A mixed deductive and inductive methodology was employed for the study design and qualitative data analysis. Semi structured interviews were conducted with two cohorts of trainees (three pre-FRCS and three post-FRCS).

Six overarching themes emerged from thematic analysis of coded qualitative data – ‘insight’, ‘performance’, ‘simulation’, ‘stress’, ‘differentiation strategies’, and ‘assessment versus learning’.

Gaining insight into the format, standard, marking scheme, and processes of the final FRCS examination were considered key elements of the educational value of the mock viva and these were perceived to be acquired through high fidelity simulation of the FRCS exam, with high quality feedback on performance. The opportunity to observe the assessment of others offered insight into one’s performance relative to peers as well as insight the perspective of the examiner conducting the assessment.

Whilst they acknowledged negative ‘stress’ factors associated with the mock viva, post-FRCS trainees underscored the benefits of learning from substandard performances; of reflecting on these experiences and on the feedback they received, and they suggested that junior trainees lacked insight into the educational value of what are often perceived to be ‘negative’ experiences. Junior trainees were concerned that the potential learning value of the mock viva process was limited by their clinical knowledge and experience in the early stage of training.

These overarching themes point to areas which could be targeted to enhance the educational value of the mock viva and to address the paradox of poor uptake amongst junior trainees.

Tables

Table 1.. Participant Demographics

Participant	Level of training	FRCS exam status
P1	ST3	Pre-FRCS
P2	ST3	Pre-FRCS
P3	ST4	Pre-FRCS
P4	ST8	Post-FRCS
P5	ST8	Post-FRCS
P6	ST8	Post-FRCS

Table 2.. Thematic Analysis

Code references (Post FRCS participants in bold)	Sub-themes	Themes	Overarching Themes
P1; P3	Structure	Exam format	Insight
P1; P5	Marking Scheme
P3; P5	Classic exam cases
P3; P6	Usefulness of feedback	Frame of reference for performance
P1	Gaining perspective of examiner by observing others
P1; P2; P3	Level of knowledge relative to peers
P1; P2; P3	Standard required for FRCS
P5	Limitations of feedback and frame of reference
P1; P2; P4	Forces acknowledgement of unknowns
P4; P6	Lack of insight into what can be gained from the process of the mock viva	Junior trainees lack insight
	Lack of insight into their learning needs in the approach to the FRCS exam.
P2; P5; P6	Convincing performance		Performance
P2; P3	Confident performance
P1; P2; P4:	Practiced performance
P1; P5; P6	Developing a system	Exam technique	Simulation
P4; P6	Increases likelihood of success in FRCS exam	Practice
P2; P3; P5	Rehearsal
P1; P2; P3	Independent assessors	Must be high fidelity simulation of exam
P3	Vary the examining style
P2; P4; P6	Daily viva sessions in handover meetings as a panacea to the problems of the annual mock viva	Frequency of simulation
P4; P5; P6	Learning from failures	Challenge	Stress
P3; P4	Rising to challenge and competition
P3	Time pressured
P1; P2; P3; P4; P5; P6	Pressure as a negative factor	Anxiety
P4; P5	Pressure as a motivating factor
P1; P2; P4; P5; P6	Limited knowledge limits educational value of mock viva process	Content vs Process	Differentiation strategies for junior and senior trainees
P1; P2; P3; P5; P6	Engaging with the viva process is possible and necessary at the early stages of training, independent of level of knowledge
P1; P4; P6	Paradox of poor delivery in context of good knowledge
P2; P6	Support for coaching	Coaching for junior trainees
P3; P4	Against coaching
P1; P5	Learning the ‘hidden curriculum’	Learning opportunity	Assessment versus Learning
P1: P3	Practice rather than learning
P1	Assessment detracts from the learning opportunity	Assessment
P2	Primary aim is assessment
P2; P4; P5	Assessment promotes learning
P2; P5	Implications for future training	Judgement
P3	Opportunity to demonstrate competence

Full text

Introduction

Neurosurgery trainees take the Intercollegiate FRCS examination at the end of their training, which requires them to demonstrate knowledge and clinical competence commensurate with that of a newly qualified consultant neurosurgeon. One part of this summative process is the ‘Oral Examination’, which is a structured ‘viva voce’ designed to test knowledge, higher order thinking and decision-making skills.

The overall pass rate for the examination is 50-60% (JCIE Results, 2015-17). There is speculation that it is not lack of knowledge per se which is responsible for the low pass rate, but rather ‘poor confidence, which translates into weak delivery and hesitant performance’ (Choi, in Elwell et al, 2015).

It was with this principle in mind that an annual ‘Mock Viva’ was established in my deanery, with the aim of practicing the oral examination in a formative setting throughout the training programme, thus improving trainees’ performance in the final summative examination.

Whilst there appear to be clear advantages to ‘rehearse’ in the run-up to the final exam, the take-up rate for the annual mock viva amongst the cohort of neurosurgery trainees in my deanery is often lower than expected, particularly amongst junior trainees.

There is a small body of research from the USA demonstrating perceived advantages and modest objective improvements in first time pass rates for equivalent postgraduate surgical examinations with the implementation of mock oral examinations ( Aboulian et al., 2010; Fingeret et al., 2016; Smeds et al., 2018).

Beyond the primary objective of improving final pass rates, some evaluative studies have shown additional perceived benefits of mock oral examinations, including improvements in clinical reasoning, promotion of self-study and development of professional and communication skills ( Higgins et al., 2016; Pennell and McCulloch, 2015). These studies, although of limited quality, are based on educational settings closely related to the one in question.

Compounding the challenge of performing well in these high-stakes summative assessments is the inherent ‘stress’ factor. Although there is evidence that higher levels of stress can improve performance, research has shown that performance increases only up to a point and, paradoxically, for complex tasks, if levels of arousal become too high, performance actually decreases. This is thought to be due to negative effects of stress on cognitive processes like attention, memory, and problem-solving ( Yerkes and Dodson, 1908).

Thus, it follows that to be prepared for the oral examination candidates need to be performing at the level of ‘proficient’ or ‘expert’ in the Drefus model of adult skill acquisition ( Dreyfus and Dreyfus, 2004), to accommodate the negative impact of stress.

Context of Study

To investigate the paradox between the promising aims of the mock viva and surprisingly poor engagement with the process, I decided to evaluate trainees’ perceptions of the viva process.

As a senior neurosurgical trainee, who has recently completed the FRCS examination, I effectively have a ‘foot in both camps’, as both an exam candidate who has experienced first-hand the challenge of the FRCS viva, and in my new role of co-organiser and mock examiner in the next annual formative oral examination. Inevitably, I have my own views on this particular formative assessment; and whilst my new role as an examiner may bias the responses from some participants, it also gives me the opportunity to address any areas for improvement which may become apparent.

Permission was granted from my Training Programme Director to recruit participants from the current cohort of neurosurgical trainees.

Study Question

What is the educational value of the mock viva to neurosurgical trainees, and how can it be enhanced?

Objectives

• •To explore trainees’ perspectives on the value of the mock viva as an annual formative assessment tool in early years training.
• •To explore the barriers or potential down-sides to participating in the mock viva and how they could be attenuated to increase levels of engagement.
• •To compare the perspectives of trainees who are pre and post-FRCS examination on the utility the mock viva for preparing for the summative final examination.

Methods

I investigated the perspectives of two separate cohorts of trainees, because I hypothesised that trainees pre- and post-FRCS would have different views on these topics based on their different experiences, and using a deductive approach, I anticipated that triangulating the data collection from different groups would be key to answering my research question.

The first cohort of pre-FRCS neurosurgical trainees (ST1-6) were investigated with a series of semi-structured individual interviews. I had intended to use a focus group for this cohort of trainees with the expectation that building on the ideas of other participants through facilitated discussion would produce richer data. However, this was not feasible to arrange and I opted to conduct individual interviews instead. This strategy offered the advantage of anonymity and reduced the effect of bias in the responses. In order to build on the ideas of others and find out to what extent these perspectives were shared by the pre-FRCS cohort, I presented anonymised comments and ideas generated from earlier interviews to the remaining participants, in order to canvas opinion on these theories.

The second cohort of post FRCS trainees (ST7-8 or equivalent) were also investigated with semi-structured individual interviews. This method of data collection was chosen in preference to a focus group to protect the anonymity of potentially sensitive personal information, and to avoid conflict of interest, particularly in the context of some participants who had multiple attempts at the summative examination, or had yet to achieve a pass. Whilst this was time-consuming, it facilitated in depth, personal responses from participants, with less risk of bias from others.

I invited trainees from my own department to participate in the interviews, partly selected by convenience, but also because my department has within it a suitable range of junior and senior trainees to enable me to answer my study question. I selected three trainees for each group with the aim of obtaining a range of opinions and ideas, without compromising the efficiency and practicality of the data collection.

Interview questions for the first cohort were tailored with an inductive approach and were firmly rooted in the evaluation aims and in the expectation of generating new theories from the data (Appendix 1).

The data from the interview transcripts were coded and a thematic approach was used for analysis of the results. Codes from the analysis of the pre-FRCS cohort data were identified and used to construct a framework for the second cohort of interviews with post-FRCS trainees. Thus, a mixed deductive and inductive methodology was employed for my study design and qualitative data analysis.

This study had no direct relevance to patient care. Interview transcripts were anonymised and individual responses were kept confidential when discussing newly generated ideas with other participants.

As a senior trainee, I could relate to participants in both cohorts and as an interviewer I was able to facilitate a deep discussion about the training issues in a supportive and non-judgemental environment. However, the impact of my relationship as a fellow trainee could be confounded by my role as a mock examiner and could have influenced the responses from participants.

To ensure that these potential concerns were recognised and addressed, trainees were given an information sheet on the study and were asked to sign a consent form prior to participation (Appendices 2 & 3).

The final report was shared with my Training Programme Director (TPD), so any significant findings arising from the study could be highlighted for review.

Results/Analysis

There were six participants, as outlined in Table 1, who were interviewed in the following order.

Six clear overarching themes emerged from thematic analysis of coded qualitative data from the interview transcripts, as outlined in Table 2.

Comparison of responses from the two cohorts of participants revealed some notable differences. Whilst all the participants reported negative effects of stress and anxiety on their performance in the mock viva, it was only the post-FRCS participants who reported the educational value of learning from these ‘failures’ in their mock exam performances, and they did so unanimously.

As one post-FRCS participant described, “The best mock exams were when my knowledge was pushed to the very limit… Those exam scenarios stuck to my mind the best, so in fact, although it may not feel like it at that moment in time, you learn most from that and you might actually benefit the most from those” (P4, Appendix 4).

This senior cohort of trainees also exclusively reported the positive effect of stress as a motivating factor in promoting learning, and the role of the mock viva in increasing the chance of success in the FRCS exam.

In their responses, the post-FRCS participants all heavily emphasised the importance of the mock viva in helping them to develop a system for answering questions, which emerged as the theme of ‘exam technique’. They also emphasized the ‘performance’ nature of the exam and the need to develop a confident delivery such that one presents oneself to examiners as a credible colleague. As one senior trainee summed up, “You are being judged by consultant colleagues based on this performance, and so how have to convince them that you are one of them” (P5, Appendix 4).

The concept of developing ‘insight’ into the exam process, the standards and one’s relative performance was a repetitive theme in the responses. Whilst all the participants described the general benefits of finding out about the process and the standards, the post-FRCS cohort offered a more nuanced perspective on this topic. They felt that junior trainees were failing to appreciate the immense educational potential of the mock viva, and that their failure to overcome their anxieties about the process was counterproductive, as it is an essential process in preparation for the exam. They emphasised the importance of engaging with mock viva examinations early in training, so they can learn more about what they do not know. In describing this lack of insight in junior trainees, one post-FRCS participant pointed out, “If you know less, then this awareness of ignorance is less likely to be there” (P4, Appendix 4).

Participants from both cohorts argued that limited levels of knowledge and experience in the junior trainees, meant there was little to be gained from their participation in the mock viva as they had insufficient ‘content’ to meaningfully practice the ‘process’. Other post-FRCS trainees argued that the system can be learned on the most basic of clinical knowledge and should be put into practice early in training.

This perceived deficiency in ‘content’ was the source of many anxieties surrounding the mock viva, and fed into another overarching theme of whether the mock viva represented an ‘assessment’ or a ‘learning’ opportunity. As a formative assessment its function was perceived to be dependent on the quality of feedback given, and the learning experience of the trainee as an observer of others. Junior trainees reported value in comparing their performance with those of peers and from observing their performance in an imaginary role as an examiner (P1, Appendix 4). However, a number of trainees reported stress associated with judgement from peers and consultant colleagues, which could impact their future training, and felt threatened by the potential conflict of interest.

From these overarching themes a number of ideas for enhancing the educational value of the mock viva experience were developed. These included differentiated approaches to formative assessment for junior and senior trainees, such as coaching in viva preparation for juniors, and pre-defined ‘content’ for juniors, so that the focus of their viva is on simulating the ‘process’. Other suggestions for improvement arose from themes which highlighted the importance of high fidelity simulation, such as recruiting independent assessors from different units to vary the questioning styles and the case scenarios, and to eliminate bias or conflicts of interest.

Discussion

These six overarching themes highlight the complexities of learning a hidden curriculum through formative assessment and point to areas which could be targeted to enhance the educational value of the mock viva.

Participants from both cohorts described the exam as a ‘performance’, and suggest that one needs to learn how to ‘play the game’. This points to a hidden curriculum ( Jackson, 1968) which must be learned. And as one post-FRCS participant described, “No one teaches you that. It’s trial and error” (P6, Appendix 5). This concept reflects expert opinion that “Demonstrating competence is ultimately a ‘performance’ – involving ‘a plethora of soft skills’” (Elwell et al, 2015).

Gaining insight into the format, marking scheme, processes, and one’s performance relative to peers and the final FRCS standard are all considered key elements of the educational value of the mock viva and are perceived to be acquired through high fidelity simulation of the FRCS exam, with high quality feedback, and the opportunity to observe peers, which offers insight into performance from the perspective of the examiner. These perceived benefits are grounded in experiential learning theory and add to evidence from other studies which suggest that conducting mock oral exams in a public forum can add educational value for observers, even if they are not preparing for a summative examination process themselves ( Aboulian et al, 2010).

Whilst they recognised the negative stress factors associated with the mock viva, the post-FRCS cohort also pointed to the benefits of learning from poor performances; of reflecting on these experiences and on the feedback they received, and suggested that junior trainees lacked insight into the educational value what are perceived to be ‘negative’ experiences.

This concept mirrors the educational theory of experiential learning described in Kolb’s learning cycle ( Kolb, 1984) and emphasizes the importance of learning through the iterative process of reflection on experiences and of developing theories and action plans to improve performance. Repetitive simulation of the exam in a formative setting is likely to reduce the complexity of the task and thus will offset the heightened levels of arousal in the summative exam, which can have negative effects on cognitive processes like attention, memory, and problem-solving as postulated by Yerkes & Dodson (1908).

This concept is also reflected in petitions for routine viva practice as part of a daily handover meeting. It was felt that this would be a panacea for addressing the issues of ‘content’ and ‘process’ simultaneously, and facilitate differentiated instruction for heterogeneous groups of trainees. The strategy of frequent simulation “to acclimatise to the stress”, by being “quizzed regularly by colleagues or senior surgeons, to experience the adrenaline rush of being ‘put on the spot’ and to become accustomed to answering questions under pressure” is also advocated by exam expert David Choi (Choi, in Elwell et al, 2015).

The theme of developing more insight as one progresses from novice to expert, and ‘unknown unknowns’ become ‘known unknowns’ is also grounded in the Drefus model of adult skill acquisition ( Dreyfus and Dreyfus, 2004) and is, perhaps unsurprisingly, strikingly reflected in the official scoring sheet of the Examination Board (JCIE Marking Descriptors, 2016).

Recommendations for local practice

• •Introduce coaching in viva preparation for ST1-2 trainees
• •Limit the content to be covered in the mock viva as appropriate for each stage of training and allow preparation of this predefined content in advance of the session
• •Recruit independent examiners to eliminate conflict of interests and increase fidelity of exam simulation
• •Introduce routine daily viva sessions which will create opportunities for differentiated instruction and for trainees to observe and reflect on the performance of others
• •Train examiners in giving effective feedback

Areas for future investigation include implementation of these strategies and evaluation of their effectiveness.

Limitations

This was a small evaluative study and my role as both a fellow trainee and mock examiner had the potential to introduce bias into the questioning, the responses from participants and the data interpretation. However, the two sample cohorts of trainees are strongly representative of the body of trainees in my deanery and consistent themes have emerged from the data analysis which are convincing and generalizable to the study area.

Effective implementation of these recommendations will require a cultural change within the educational setting and this calls for strong leadership and engagement of all the stakeholders in the mock viva process.

Conclusion

This study has demonstrated a wealth of perceived educational benefits of participating in the mock viva, and has shed light on the factors which contribute to the paradox of poor uptake amongst junior trainees. Exploration of the themes which emerged from the data analysis, generated strategies for attenuating the negative stress factors, increasing levels of engagement and differentiating the instruction for junior and senior trainees to enhance the educational yield of the mock viva process.

Take Home Messages

• •Trainees perceive the mock viva to offer a wealth of potential educational value, through high fidelity simulation of the FRCS exam, high quality feedback, and the opportunity to observe peers, which offers insight into performance from the perspective of an examiner.
• •Whilst junior trainees perceived poor performances to be ‘negative’ and stressful experiences, post-FRCS trainees had a strong appreciation for the educational value of receiving critical feedback and reflecting on substandard performances.
• •The educational yield of ‘mock vivas’ could be enhanced using differentiated approaches for junior and senior trainees to alleviate stress factors and increase levels of engagement.

Notes On Contributors

Gráinne McKenna MA(Cantab) MB BChir FRCS(SN) is an ST8 neurosurgical trainee in London and is undertaking an MSc in Medical Education with UCL and The Royal College of Physicians.