Unlocking Mixed Reality for Medical Education: A See-Through Perspective on Head Anatomy

TL;DR

This paper introduces a mixed reality application for head anatomy education that enhances interactivity and spatial understanding through see-through 3D models, gesture controls, and automatic calibration, improving medical teaching methods.

Contribution

It presents a novel MR system with hierarchical learning design and automatic calibration for immersive, interactive anatomy education, addressing limitations of traditional methods.

Findings

Effective alignment of virtual models with real heads.

Enhanced interactivity and immersion in anatomy learning.

Supports progressive, intuitive learning of complex structures.

Abstract

Extended reality (XR), encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), is emerging as a transformative platform for medical education. Traditional methods such as textbooks, physical models, and cadaveric dissections often lack interactivity and fail to convey complex spatial relationships effectively. The emerging MR technology addresses these limitations by providing immersive environments that blend virtual elements with real-world contexts. This study presents an MR application for head anatomy education, enabling learners to intuitively interact with see-through 3D anatomical structures via hand gestures and controllers. Our hierarchical information design supports progressive learning, guiding users from basic anatomical labels to detailed structural insights. Additionally, the system incorporates an automatic calibration module that aligns virtual anatomical models with a real human head, thereby facilitating realistic human-model interactions. Experiments show that the system can effectively match the anatomical model with real-time scenes, thus enhancing the interactivity and immersion of medical education, providing an innovative tool for teaching anatomy.

Figures

Tables

Table 1. Table 2 : Overlaying error for different facial features.

	Number of head poses	F1				F2				F3				All
		mean(px)	mean(mm)	std(px)	std(mm)	mean(px)	mean(mm)	std(px)	std(mm)	mean(px)	mean(mm)	std(px)	std(mm)	mean(px)	mean(mm)	std(px)	std(mm)
P1	71	33.74	6.46	15.92	3.05	25.53	4.89	11.98	2.29	26.57	5.08	13.32	2.55	28.61	5.47	14.22	2.72
P2	39	62.16	9.79	23.72	3.74	41.64	6.56	18.79	2.96	40.35	6.36	20.70	3.26	48.05	7.57	23.16	3.65
P3	56	42.91	7.91	29.70	5.47	33.13	6.11	22.94	4.23	34.12	6.29	22.94	4.23	36.72	6.77	25.55	4.71
Mean			7.73		4.36		5.69		3.29		5.79		3.42		6.40		3.81

Equations

$$P_k=D(u_k,v_k)\cdot K^{-1}\left[\begin{array}{c}{u}_k\\ v_k\\ 1\end{array}\right],\text{vskip}-20.0pt$$

$$K=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right].\text{vskip}-20.0pt$$

$$A$$

$$P_j=P_o\cdot A\text{vskip}-20.0pt.$$

$$T_W^V=T_H^V{\left(T_H^C{T}_C^W\right)}^{-1}.\text{vskip}-20.0pt$$

Full text

1 Introduction

With the rapid development of computer technology, extended reality (XR), such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), is increasingly being used in education and medicine. VR creates a fully synthetic interactive environment that immerses users in a computer-generated space [1]. In contrast, AR overlays digital elements onto the physical world, allowing users to perceive and interact with both real and virtual objects [2]. MR combines the capabilities of VR and AR, merging the physical and digital environments to enable real-time interaction with 3D virtual content in the real world [3]. Over the past few years, MR has garnered increasing attention in the medical field [4, 5], as visualization and spatial perception of complex anatomical structures are crucial. MR is a hybrid of AR and VR, resulting from the integration of the physical and digital worlds [6, 7]. In this case, it addresses the limitations of VR, which excludes the real-world environment, and AR, which cannot interact with three-dimensional (3D) data packages. In the medical field, MR has great potential in enriching the real-world environment with virtual data on a single display and allows interaction through various methods [8].

Traditional medical education relies on textbooks, physical models, and cadaveric specimens. However, these approaches present several limitations, including low interactivity, constrained contextual environments, and challenges in visualizing complex anatomical structures underneath. Recent studies indicate that both students and educators perceive MR-assisted instruction as more effective than conventional methods in medical teaching [9]. MR enables a more intuitive and immersive learning experience by allowing students to interact with virtual 3D anatomical content in real time and observe internal structures layer by layer to gain a better understanding of spatial relationships that are often difficult to grasp through traditional static resources.

Based on these advantages, this study introduces an interactive MR application designed to enhance immersive teaching and intuitive learning of head anatomy through a see-through perspective. The application enables users to visualize detailed 3D anatomical structures superimposed on the upper chest area, while simultaneously maintaining awareness of their physical environment via a head-mounted display (Meta Quest 3). A schematic diagram, as shown in Fig.1, outlines this work.

To enable accurate spatial alignment, an estimation of the head pose based on MediaPipe [10] is incorporated into an automatic calibration module. This module aligns the virtual anatomical model with a real-world anatomical reference, providing a more intuitive and context-aware learning experience. Additionally, users can interactively explore anatomical features by adjusting the model’s transparency, scale, and the visibility of specific organ regions. Textual descriptions are integrated to supplement each region, offering layered anatomical knowledge to support self-paced learning. Our approach provides a valuable reference for immersive medical education using MR technology, with the potential to be extended to other anatomical regions and organ systems.

The main contributions of this paper are as follows:

We propose a MR-based medical education platform focused on head anatomy. The system offers an interactive learning interface that supports manipulation of 3D models and multilayered anatomical exploration, thereby promoting spatial understanding and learner engagement.

• 2.

We design and implement an automatic calibration module that spatially aligns a virtual anatomical model with a real human head. This feature supports personalized anatomical education directly on real individuals, offering promising potential to enhance educational value.

• 3.

We conducted an experimental validation with three participants to assess the system’s usability and accuracy. Both qualitative and quantitative results suggest the system holds promise for enhancing medical education.

2 Related Work

VR immerses users in a fully synthetic environment through head-mounted displays (HMDs), enabling complete detachment from the physical surroundings. This technology has found extensive applications in education, healthcare, engineering, and other domains. Coyne et al. suggests that VR technology is an immersive, comfortable, and engaging learning platform that can be used for team-based remote learning [11]. VR not only enables experiential learning by providing simulated virtual environments, but also supports the creation of works in these environments and triggers activities through user interaction, supporting dynamic forms of learning [12]. For instance, product design can leverage VR technology to create complex 3D models and products using 3D modeling tools and visualize them, facilitating interactive and real-time communication of product information [13]. VR is also applied in the medical field, particularly in surgery and education. In surgical training, VR technology can significantly reduce the likelihood of surgical errors, improving patient safety [14]. The use of VR technology also aids rehabilitation clinicians in remote therapy—telerehabilitation, allowing patients to exercise in a virtual environment at home and transmit data to their doctors [15]. In medical education, Haluck and Krummel describe a progressive learning model in which VR-based simulation learning is a crucial step. Students can practice components of a procedure or the entire operation before conducting the final stages of skills training in the operating room [16], which helps them gain confidence before the resident rotations.

In contrast, AR is an interactive display environment based on the real world, enhancing the user’s experience by utilizing computer-generated displays, text, sound, and special effects. AR supports the enhancement of the user’s view by overlaying data and information in real-time in the actual environment [17]. AR has been widely applied in the medical field. In medical rehabilitation, Coelho et al. created a physical mixed model and a mobile device AR application using imaging data from a patient who had undergone surgery for cranial deformity correction. This method was used by 38 experienced surgeons who considered it a valuable educational tool for teaching and pre-surgical planning [18]. Ferrari et al. utilized electromagnetic sensors to detect nickel-titanium tubes—one of the commonly used medical devices, enabling AR visualization of hidden deformable tubular structures in situ. The obtained alignment accuracy demonstrated the feasibility of this method, which can be used in advanced AR simulations, particularly as a tool for identifying and isolating tubular structures [19]. Ortiz-Catalan et al. employed AR technology to enable amputees to see a virtual arm appear on a screen, which could be controlled by the brain to move the previously severed limb, thus achieving therapeutic results [20].

MR combines elements of both VR and AR, containing both physical and virtual components [21]. MR is often described by experts as a sliding scale between an entirely physical environment with no virtual elements and a fully virtual environment [8]. MR is not confined to the virtual environments of VR or the single mode of interaction found in AR. Parveaua and Addaa categorized MR into three types: First, it consists of both real and virtual content, allowing data contextualization. Second, the digital content requires real-time interaction. Third, the content must be spatially mapped and related to three-dimensional space [22]. MR technology is promising for exploring the anatomical structure of the human body. McJunkin et al. demonstrated that MR headsets can display interactive 3D anatomical models of the temporal bone, anchored in real space. By manually aligning the fixed model with a real head, this technology shows potential as an image-guided tool for anatomy and lateral skull base surgeries [23]. In the field of tumor surgery, Scherl et al. applied MR in preoperative planning for minimally invasive electroporation and microwave ablation of advanced hepatic gastrointestinal tumors, optimizing surgical methods through remote and hospital-based analysis of the patient’s specific anatomical structures [24]. In otolaryngology and maxillofacial surgery, MR facilitates easier extrafascial dissection of parotid tumors, thereby reducing the incidence of complications [25]. MR-guided liver resection planning has improved understanding of liver vascular anatomy and tumour location, enhancing resection accuracy while preserving a larger residual liver volume [26]. The development of patient-specific VR and MR models using web-based services and their preoperative and intraoperative application demonstrates the significant potential of the surgical assistance tools [27].

3 Methodology

3.1 Overview

The proposed application is deployed on the Meta Quest 3 platform by using Unity 3D, enabling users to interact with an accurate 3D model of the human head in an immersive MR environment. As illustrated in Fig.1, the system presents a detailed anatomical structure that includes multiple organ systems and tissue types such as skin, bones, arteries, veins, lymphatic vessels, glands, and tendons. The 3D anatomical models used in this study were sourced from the project “Lymphatics Drainage of Head and Neck”, developed by the University of Dundee, School of Dentistry [28], and are licensed under CC BY-SA 4.0.

To enhance anatomical clarity and usability, the application adopts a color-coded visualization scheme. This visual differentiation aids users in quickly identifying specific structures and understanding their spatial relationships. The model supports dynamic visualization, allowing learners to view external features like skin as well as internal organs layer by layer. The system includes several core features aimed at promoting interactive learning:

(1) Gesture and Controller-Based Manipulation: Users can zoom, rotate, drag, and place the 3D anatomical model using either hand gestures or handheld controllers for flexible interaction.

(2) Detailed Anatomy Explanation: Each part of the head can be individually visualized, allowing users to isolate specific structures. For each part, corresponding educational content is provided, including detailed descriptions, functions, and related anatomical relationships.

(3) Real-World Model Alignment: An automatic calibration module enables spatial alignment of the virtual anatomy with a physical anatomical reference. This helps users relate digital content with real-world context and enhances comprehension of internal anatomical structures.

With the above features, this MR-based application offers an intuitive and engaging learning tool to support medical education.

3.2 Automatic Calibration

In addition to directly visualizing the virtual head model, the system is capable of automatically aligning the model with a real human head.

3.2.1 Head Pose Estimation

Initially, as illustrated in Fig.2 (a), the head pose is estimated using a combination of MediaPipe for facial landmark detection and the Perspective-n-Point (PnP) algorithm for 3D pose estimation. To apply the PnP algorithm, a set of 2D–3D point correspondences ($p_{j}\leftrightarrow P_{j}$) is required.

To obtain the 3D points $P_{j}$, the initial 3D coordinates $P_{o}$ of key facial landmarks are defined on a virtual anatomical model with dimensions of $16\times 23~cm$. The coordinate system is established with the nose tip as the origin, and all axes are aligned with the facial orientation. Next, a scaling factor derived a depth camera will be used to adjust $P_{o}$ to better approximate the real facial 3D coordinates.

In this study, an RGB-D camera (Intel RealSense D435) is used to capture both the RGB image $I$ and the aligned depth image $D$. The RGB image is processed by MediaPipe, which outputs a set of 2D facial landmark positions $\{p_{k}=(u_{k},v_{k})\}$. For each 2D point $p_{k}$, the corresponding 3D point $P_{k}$ in the camera coordinate system is reconstructed using the depth image and the camera intrinsic matrix $K$ as follows:

[TABLE]

where $D(u_{k},v_{k})$ denotes the depth value at pixel $(u_{k},v_{k})$, and $K$ is the camera intrinsic matrix defined as:

[TABLE]

Here, $(f_{x},f_{y})$ are the focal lengths and $(c_{x},c_{y})$ represents the principal point of the camera.

The width ($w$) and length ($l$) of the face can be easily obtained with a few reconstructed 3D points $P_{k}$. Then, the scaling matrix $A$ is constructed as follows:

[TABLE]

where $w_{o}$ and $l_{o}$ are the width and length of the anatomical model. Therefore, the 3D points of the real face can be approximated as:

[TABLE]

Since the target head remains unchanged, when observed by the Meta Quest 3, the 3D position satisfies $P_{i}=P_{j}$. The image captured by the Meta Quest 3 is also processed by MediaPipe, yielding the 2D positions of facial landmarks $\{p_{i}\}$. Using the previously obtained 3D points $\{P_{i}\}$ and their corresponding 2D image projections $\{p_{i}\}$, the PnP algorithm is employed to estimate the rotation and translation vectors that define the head pose $T_{H}^{C}$ with respect to the Meta Quest 3 camera.

3.2.2 Virtual-to-real Calibration

With the head pose and head size, the anatomical model can be adjusted to fit the real face. The scaling factor obtained from the depth camera is again used to adjust the model size. To maintain its shape, the anatomical model is uniformly scaled along three axes, with a factor of $A_{z}$ in Eq. 3.

The pose of the virtual model $V$ in the virtual world frame $W$, denoted as $T_{W}^{V}$, must be computed to accurately overlay the virtual model onto the real head. As illustrated in Fig. 2(b), the corresponding kinematic chain is established. The system contains two types of coordinate frames: the virtual components developed in Unity use a left-handed coordinate system, while the real components are defined in a right-handed coordinate system.

Therefore, the first step is to convert the left-handed frame to a right-handed one. Specifically, this is achieved by inverting the $z$-axis of the virtual model frame $V$. Afterwards, a rotation of $180^{\circ}$ about the $y$-axis is applied, which maps the left-handed virtual model frame $V$ to the right-handed frame $H$, ensuring consistency between the coordinate systems of the virtual and real environments.

In addition, since the Unity application is deployed directly on the Meta Quest 3 device, it provides a direct transformation from the camera frame $C$ to the virtual world frame $W$ by utilizing the onboard Simultaneous Localization and Mapping (SLAM) system. As a result, the transformation $T_{W}^{C}$ can be obtained in real time. Finally, with the estimated pose of the head $H$ in the camera frame $C$, the target pose of the virtual model in the world frame can be calculated as:

[TABLE]

This equation enables precise overlay and registration of the virtual model onto the real head within the MR environment.

3.3 Education Features

To enhance user engagement and facilitate anatomical learning, the system integrates a set of educational features designed specifically for MR environments. As shown in Fig.1(b), these features include interactive information display, selective visualization of anatomical regions, and an intuitive user interface (UI) for control and guidance.

The application supports selective visualization, allowing users to toggle the visibility of individual tissues through a control panel. Transparency adjustments are also available, enabling inspection of deeper anatomical layers beneath surface structures. An interactive wrist panel is mounted on the user’s virtual wrist, providing quick access to key functionalities. Upon launch, users are guided by an in-scene tutorial video and an optional onboarding panel, both of which can be reactivated as needed.

These integrated educational features create an immersive and learner-centered MR experience that supports both exploratory learning and structured knowledge acquisition.

4 Experiments and Results

One of the significant features of the proposed system is its ability to anchor anatomical content directly onto a real human head. This personalized visualization enhances educational value by allowing users to observe anatomical structures overlaid on actual individuals in real-time. To evaluate this function, three participants were recruited, each tested under various head poses. Both qualitative and quantitative analyses were conducted to assess the alignment accuracy and robustness of the overlaid anatomical content.

4.1 Qualitative Evaluation

As shown in Fig. 3, a user wearing the MR device was instructed to observe the heads of three participants. The system then automatically aligned the 3D anatomical model to the detected head pose in real time. From the captured frames, it can be seen that the virtual anatomical structures align reasonably well with the real-world head positions from multiple viewpoints. Furthermore, the overlay remained stable under moderate head rotations and translations, suggesting the robustness and reliability of the virtual-to-real calibration method used in the system.

4.2 Quantitative Evaluation

To quantitatively assess the alignment precision, we evaluated overlay errors at three key facial landmarks: the left eye corner (F1), chin (F2), and right eye corner (F3), as illustrated in Fig. 4. Table 4 demonstrates the estimated head size in image space and the corresponding real head size of different participants.

A total of 166 head poses from three participants were used for analysis. For each pose, the landmarks F1, F2, and F3 were manually annotated on the overlaid virtual model in recorded videos. Simultaneously, the same video frames without the overlay were processed using the MediaPipe framework to automatically detect corresponding facial landmarks. The overlaying error was then calculated as the Euclidean distance between the positions of each manually annotated landmark on the virtual model and the automatically detected landmark on the real face.

The results are summarized in Table 2, which shows the mean pixel errors of the three key landmarks on different participants. Based on the average pixel-to-millimeter ratio from the head dimensions, the corresponding errors in millimeter are estimated. Over the three participants, the mean errors were 7.73$\pm$4.36 mm for F1 (left eye corner), 5.69$\pm$3.29 mm for F2 (chin), and 5.79$\pm$3.42 mm for F3 (right eye corner). Therefore, the overall error of the three landmarks was 6.40$\pm$3.81 mm in physical space. Compared with the AR visualization error of 6.97 mm from [29] and the misalignment error of 16.50 mm from the method based on reflective-AR display [30], the findings indicate that the system achieves a reasonable level of anatomical alignment suitable for educational purposes in real-world conditions.

5 Discussion and Conclusion

This study presents the development of an MR application aimed at enhancing the understanding of head anatomy. By enabling real-time interaction with 3D anatomical models, the system provides an immersive and intuitive learning and teaching experience. A key innovation of the proposed system is the integration of an automatic calibration module that spatially aligns virtual anatomical structures with a real human head, enhancing contextual relevance and educational value.

However, the system currently lacks the accurate 3D coordinates of real facial landmarks, which may cause slight alignment deviations. Additionally, the reliance on a pre-defined 3D head model may exhibit significant misalignment when applied to individuals with different facial shapes. A potential solution to this problem is to adjust the three dimensions of the model with independent scaling factors, enhancing the model’s flexibility to some extent. Future work will focus on integrating high-precision depth-sensing technologies to improve the accuracy of landmark localization, as well as including a more diverse participant pool in terms of skin tone and facial shape to improve its generalizability. Furthermore, participants’ knowledge levels will be assessed before and after using the application to evaluate its educational effectiveness.

Acknowledgements

This work is partially supported by the Guangdong Basic and Applied Basic Research Foundation General Program under grant 2025A1515011594; in part by the Hong Kong Research Grants Council Collaborative Research Fund under grant C4026-21GF; in part by CUHK–Department of Electronic Engineering MSc Research & Development Project grant.