On-Body Visualization of Patient Data for Cooperative Tasks

TL;DR

This paper introduces an anatomically integrated in-place visualization system for patient data, enhancing cooperative clinical tasks by using a virtual body model to improve data access and understanding in neurosurgery.

Contribution

It presents a novel visualization approach that embeds patient data directly onto a virtual body model, tailored for clinical cooperation and based on formal design requirements.

Findings

Prototype evaluated by 10 neurosurgeons showed positive feedback.

The visualization improved task-specific data access and understanding.

Challenges in visual encoding and visibility were identified and discussed.

Abstract

Electronic health records (EHR) systematically represent patient data in digital form. However, text and visualization based EHR systems are poorly integrated in the hospital workflow due to their complex and rather non-intuitive access structure. This is especially disadvantageous in clinical cooperative situations that require an efficient, task specific information transfer. In this paper we introduce a novel concept of anatomically integrated in-place visualization designed to engage with cooperative tasks on a neurosurgical ward. Based on the findings of our field studies and the derived design goals, we propose an approach that follows a visual tradition in medicine, which is tightly related with anatomy, by using a virtual patient's body as spatial representation of visually encoded abstract medical data. More specifically, we provide a generic set of formal requirements for these kinds of in-place visualizations, we apply these requirements in order to achieve a specific visualization of neurological symptoms related to the differential diagnosis of spinal disc herniation, and we present a prototypical implementation of the visualization concept on a mobile device. Moreover, we discuss various challenges related to visual encoding and visibility of the body model components. Finally, the prototype is evaluated by 10 neurosurgeons, who assess the validity and the further potential of the proposed approach.

Tables

Table 1. Table 1: Data Categories: The raw data categories (left block), the final categories after discussion with the physicians (center block), and the visual attributes incl. anatomical reference (right block). Only the abnormal states are listed. Specific aspects are indicated as 1 : quantization, 2 : usage of proxy geometry, 3 : extended range due to explicit comparison.

Raw Data Categories			Final Data Categories			Visual Attribute	Anatom. Reference
Category	Prop. Type	Domain	Category	Prop. Type	Domain	(   range   )
Radicular Pain			Radicular Pain			Red
	Intensity	$\{1,\dots ,10\}$		Intensity	$\{1,\dots ,10\}$	Saturat.-Brightn. (3)1	Dermatome
	Trigger	binary		Trigger	binary	Texture Normal Pert.(1)
Muscle Strength			Paresis			Purple
	Intensity	$\{1,\dots ,5\}$		Intensity	{mild, moderate, severe}	Saturat.-Brightn. (3)	Muscle
T-Reflex			T-Reflex			Green
	Intensity	$\{1,\dots ,5\}$		Intensity	$\{1,\dots ,5\}$	Saturat.-Brightn. (5)3	Tendon2
Excretion Disorder			Excretion Disorder			Orange
	Intensity	binary		Intensity	binary	Saturat.-Brightn. (3)	Urethra or anus2
Paresthesia			Sensory Disorder			Cyan
	Intensity	$\{1,\dots ,3\}$		Intensity	$\{1,\dots ,4\}$	Saturat.-Brightn. (4)	Dermatome
Hypoesthesia				Paresthesia	$\{1,\dots ,3\}$	Texture Noise (3)
	Intensity	$\{1,\dots ,3\}$
Anaesthesia
	Intensity	binary

Equations

$$p=(\text{propType },\text{dataType },\text{domain}),$$

$$c=(\text{spatialRef },\text{props}),$$

$$\text{muscleStrength}=$$

$$\{(\text{intensity},\text{numerical},[0,\dots ,5])\}.$$

$$\text{radicularPain}=($$

$$\{(\text{intensity},\text{numerical},[0,\dots ,10]),$$

$$(\text{trigger},\text{nominal},\{\text{constant},\text{stress})\}\}).$$

$$T(c)$$

$$T$$

$$\forall V\in \mathcal{V},\forall c\in V$$

$$\forall c\in C,\forall t\in T(c)$$

$$\forall c\in C$$

$$\forall t\in T$$

$$\forall c\in V,\forall c^{\prime }\in V\setminus \{c\}:c.\text{spatialRef}\ne c^{\prime }.\text{spatialRef}.$$

$$\left|p.\text{domain}\right|$$

Full text

In this paper we introduce a novel concept of anatomically integrated in-place visualization designed to engage with cooperative tasks on a neurosurgical ward. Based on the findings of our field studies and the derived design goals, we propose an approach that follows a visual tradition in medicine, which is tightly related with anatomy, by using a virtual patient’s body as spatial representation of visually encoded abstract medical data. More specifically, we provide a generic set of formal requirements for these kinds of in-place visualizations, we apply these requirements in order to achieve a specific visualization of neurological symptoms related to the differential diagnosis of spinal disc herniation, and we present a prototypical implementation of the visualization concept on a mobile device. Moreover, we discuss various challenges related to visual encoding and visibility of the body model components. Finally, the prototype is evaluated by 10 neurosurgeons, who assess the validity and the further potential of the proposed approach.

1 Introduction

Hospitals today use electronic health records (EHR) to store patient data in a systematized, digital form and to share it between the different status-groups [13]. In many cases, EHRs are the basis for various automated procedures in hospitals such as clinical decision support, drug procurement and exchange of health information between different stack holders in a hospital [21]. Still, EHR systems do not necessarily improve the quality of health care [30]. This is underlined by a recent study, which assessed the physician’s beliefs about the meaningful use of the electronic health record [9]. Moreover, EHRs can hardly be used in cooperative situations like shift handovers, as they do not provide sufficient cognitive support [29]. Such cooperative situations have a very strong focus on efficient and effective information transfer. A very recent study documents fundamental problems related to the complex information structure of EHR systems that arise with their long-term usage in cooperative ward rounds [35].

While most EHR systems follow the text-intensive nature of EHRs, there are various approaches that apply methods from information visualization and visual analytics in order to improve clinical support and quality assurance for single patient EHRs or temporal and trend analysis for large EHR databases [6, 36]. The majority of approaches in both modes, i.e. single patient EHRs and large EHR databases, address the exploration of complete EHR data sets comprising data selection, reconfiguration of layout and visual data encoding, and detection of data correlations or outliers [26]. In clinical practice, however, even simple tasks that require filtering and data collection may be carried out inefficiently by medical practitioners and physicians with standard IT knowledge, due to the complex access structures of information visualization approaches [19]. A recent approach that aims at enhancing the health care practitioners’ performance in decision making and care planning by improving the visual data representation in EHR systems confirmed the overall low performance involving EHR systems even if better visual representation are utilized [12].

Our long-term interdisciplinary project deals with the fundamental question of analyzing and partly reconfiguring cooperative medical work practices, such as shift handovers and rounds on a neurosurgical ward. Our field studies revealed that

(1) selected EHR data is indispensable in this kind of cooperative setting,

(2) compared to text-intensive EHR tools, visualization methods are preferable to convey individual patient data in cooperative settings, however

(3) abstract visualization concepts for EHR data are considered as little intuitive and not focused on cooperative tasks.

There are only very few visualization techniques that potentially allow a more intuitive access to EHR data by being more tightly connected to the visual tradition of clinical practice, i.e. by considering the human body as a canonical spatial representation for clinical data. These approaches either use the human body as the central entry point for efficient access and insertion of EHR data items [17], follow a Google Earth like approach in placing body-referenced information as icons and text in 2D [32], or comprise 3D navigation and visualization methods for EHR data at different levels of detail (LOD) that partially relate to the human anatomy [1]. All these approaches are hybrid in that they present the final EHR information in textual or abstract form.

In this paper, we represent a novel visualization concept designed to engage with cooperative tasks on a neurosurgical ward, in which an efficient, task specific information transfer is needed. In contrast to previous hybrid visualization approaches for EHR data, we take advantage of the fact that each specific cooperative task only requires access to a comparatively small subset of EHR data. Therefore, we propose, implement and evaluate a comprehensive visualization concept for cooperation relevant neurosurgical medical data. In this regard, this paper contributes to

The concept of an anatomically integrated in-place visualization of abstract medical data including a formal definition of requirements for this kind of visualization. 2. 2.

The design goals and data categories for clinical neurological symptoms related to the differential diagnosis for patients with spinal disc herniation. This results from different field studies and an interdisciplinary iterative design process. 3. 3.

A prototypical implementation of the visualization concept and its evaluation by neurosurgeons.

2 Related Work

EHR systems based on interactive information visualization are commonly applied in the following application scenarios [11]:

(1) treatment planning,

(2) examination of patients’ medical records and their lifelong medical histories,

(3) representation of pedigrees and family history,

(4) patient-practitioner communication and shared decision making, and

(5) life management and health monitoring.

The patient-practitioner communication relates to designing adequate visual representations when conveying, for instance, risks of cancer. This fundamentally differs from the requirements in cooperative clinical situations addressed by our approach. Beyond the application scenario, information visualization based EHR systems are assessed using criteria such as [26]:

(1) data types, e.g. options for visualization of categorical and numerical data,

(2) multivariate analysis support, and

(3) number of patient records (one or multiple).

There are several approaches related to the visualization of single patient EHRs that have certain similarity at an abstract level, e.g. Plaisant et al. [25], Bui et al. [4], Craig[8]. These approaches commonly use multiple coordinated views (also applied to raw EHR data) to assess an overview of the patient status and medical history, by preventing complex menu and pop-up structures. In these approaches, time-related data are visualized on a timeline. Craig[8] specifically stresses the “loss of overview” as one of the main problems of EHR systems and proposes an interface design that resembles classical paper charts. This interface comprises three panes, displaying generic patient information, a timeline with “events” related to acquired medical data records, and the specific data related to the currently selected event. Ghassemi et al. [12] designed an interactive visualization tool for care planning and decision making in intensive care units (ICU) based on existing knowledge about workplace specific practices [16]. They evaluate their tool by comparison to a text-based baseline system. While their approach was slightly superior with respect to accuracy and time-to-decision, the overall task specific performance stayed low. Belden et al. [2] designed a medication timeline visualization in the context of ambulatory care of chronic disease. Their iterative user-centered design uses the participatory approach from Sedlmair et al. [27] involving workshops with clinical staff; see also Sec. 3. Related to the specific medication-related task addressed, their evaluation revealed an improved physician performance compared to a text-based presentation.

Another thread focuses on the time-related visualization of multiple patients, e.g. Guo et al. [14], Wang et al. [34], Wongsuphasawat et al. [37]. The main goal of these approaches is related to the analysis of event sequences using methods from visual analytics and statistics in order to find trends and pattern in the patients’ treatment history. These analysis methods can also be used to categorize individual patients, e.g. using cluster analysis and visualizing cluster transitions over time [14].

There are only a few approaches that utilize spatial representations in order to provide a more intuitive access to EHR data. Kirby et al. [17] presented one of the first systems that uses a visualization of the human body as the central entry point for efficient access and insertion of EHR data items. Sundvall et al. [32] present a prototype of a 2D EHR navigation and visualization framework based on Google Earth and OpenEHR. It supports so-called placemarks, the standard Google Earth approach to position body-referenced information as icons and text. An et al. [1] developed a 3D navigation and visualization method for EHR data that uses different levels of detail (LOD). The information contained in an EHR is reorganized according to a hierarchical data structure. In this hierarchy, specific user interfaces are assigned to each LOD level that are composed of level-specific graphical entities. Particularly, on the two topmost levels a virtual human body serves as the basis for visualization. Highlighting helps to identify organ systems (level 1) or organs (level 2) affected by some disease. Furthermore, by zooming into the second level, the description of the corresponding disease is represented by text labels and icons.

Based on the insight from our field studies (see Sec. 3.1) and the design goals deduced from there (see Sec. 3.2), we found that none of the existing approaches is suitable for our problem setting. The main gap lies in the lack of efficient and effective information transfer required to tackle specific clinical tasks, which is in agreement with recent studies [30, 9].

3 Design Method and Goals

In the context of our interdisciplinary long-term project on a neurosurgery ward, we investigate fundamental questions of analyzing and in part reconfiguring cooperative medical work practices. From a sociological perspective, articulation work [31] is at the core of clinical cooperation, i.e. the ongoing work of integrating distributed tasks and maintaining a coherent treatment trajectory by collection, processing and organizing patient data beyond formal divisions of labor. This information handling is crucial for the physician’s daily routine and mainly relates to the patient’s EHR data, which, however, barely supports workflows in cooperative settings.

We aim at the design and development of integrated visual representations of patient information in order to support and potentially even modify specific cooperative workflows. We address the fundamental problem of the hitherto low acceptance of text- as well as visualization based EHR systems by utilizing a participatory design approach, similar to Belden et al. [2] and Sedlmair et al. [27]. That is, we set up a design and implementation process that involves visualization researchers, sociologists and neurosurgeons in order to analyze specific cooperative real-world situations on the neurosurgical ward. Our design and implementation process comprises high-frequent interaction between these groups of experts, in particular during the initial definition of the overall goals of the intended visualization tool and the field studies that comprise observations of cooperative workflow situations and interviews. The insights of the field study are depicted in Sec. 3.1, whereas the deduced design goals presented in Sec. 3.2.

Moreover, we utilize participatory refinement of design & implementation similar to the principle of agile software development (see, e.g., [20]), in which visualization researchers, sociologists and neurosurgeons jointly advance and refine the visualization design on the basis of the formulated design goals (see Sec. 3.2) by utilizing prototype implementations of a visualization tool. For reasons of efficiency and in order to not bias the evaluation, three physicians (expert group) are involved in this stage, whereas the evaluation involves a distinct set of ten physicians (test group; see Sec. 6).

3.1 Field Study

In this first phase of data collection and processing, field observations and interviews were conducted. Here, the sociologists, and partially visualization researchers, observed neurosurgeons during their routine ward work to become acquainted with the workflows and relevance structures of their daily work. During these observations, specific cooperative work constellations, namely shift handover and ward rounds, were of particular interest. The main findings of the field studies are:

1. F1.

Clinical cooperative work is characterized by the need for efficient and effective information transfer between clinical staff with respect to specific clinical usage contexts in synchronous (i.e. face-to-face cooperation, e.g. at shift handover) or asynchronous modes (i.e. deferred cooperation with absent colleagues, e.g. at ward rounds). 2. F2.

A potentially deficient information transfer is ascertained in cooperative constellations, as they are dominated by oral, i.e. volatile, communication with little information integration from electronic data sources, namely EHR. The main reason is high time pressure and the inefficient access to the relevant EHR data. 3. F3.

There is a strong dependence of the relevance of an EHR datum on the clinical usage context. For instance, during a ward round the physician is primarily interested in a patient’s neurological status, e.g. symptoms related to the damage to sensory nerve fibers. 4. F4.

In addition to the instantaneous state of a patient, the evolution of the relevant data is of great importance in order to properly assess the course of the disease or therapy. 5. F5.

In order to allow for an efficient information transfer and to avoid overload and redundancy, physician have a clear focus on the abnormal with respect to both, the patient’s status as well as the evolution of specific patient data. 6. F6.

The physicians rely on the patient’s anatomy as spatial reference not only for diagnosis and surgery planning, using CT images for instance, but also for their articulation work, e.g. when presenting a patient in the morning meeting or handwriting information on anatomical sketches in examination forms. 7. F7.

During our semi-structured interviews we asked the clinical staff to assess the potential of various types of information presentation. In the process, existing text-based approaches were classified as just as unsuitable as overly abstract visualization concepts. 8. F8.

The physicians need ubiquitous access to the clinical data relevant to their cooperative work, e.g. on a mobile device.

3.2 Design Goals

Following the project motivation, the overarching goal of the intended visualization system is to facilitate efficient information transfer in the context of distributed medical tasks, which has been validated by the field studies (cf. F1, F2). The resulting design goals listed below have been jointly deduced by the visualization researchers, sociologists and neurosurgeons from findings of the field studies in Sec. 3.1:

1. DG1

Familiarity in novelty. The goal is to exploit the existing visual tradition, i.e. the usage of the anatomy as spatial reference (cf. F6) as far as possible and to prevent abstract visualizations (cf. F7) in order to achieve a high degree of intuitiveness. 2. DG2

Visual discriminability. Favoring non-abstract visualization concepts increases the problem to properly map data relevant in a given clinical usage context (cf. F3) in order to attain an effective information transfer (cf. F2). 3. DG3

Context-related synopsis. In cooperative settings it is important to provide a simultaneous visual access to all the data that are relevant (cf. F3) in the given clinical usage context for conveying the current patient status at a glance. 4. DG4

Intuitive comprehensibility. The visualization design has to focus on an intuitive and comprehensive interpretability in order to address the overall goal of efficient and effective information transfer (cf. F1 and F2). 5. DG5

Concise visualization. In order to avoid clutter and distraction by secondary details, we aim to provide only the data that are relevant in the current clinical usage context (cf. F3) and to focus on medically relevant, i.e. abnormal, value constellations (cf. F5). 6. DG6

Visualization of chronological changes. The visualization should provide access to the evolution of the patient’s data (cf. F4). 7. DG7

Mobility. The goal is to make the visualization available in all places using a mobile device (cf. F8).

4 The Visualization Concept

4.1 General Considerations

This section describes our visualization concept, which aims at the achievement of the design goals (see Sec. 3.2) in consideration of practical insights into the hospital workflow gathered in our field studies (see Sec. 3.1).

At the core of the concept is an anatomically integrated in-place visualization of medical data relevant to specific cooperative tasks. According to this visualization principle, an anatomical model serves as spatial representation of medical data that inherently refer to its structures, e.g. the clinical symptom paresis referring the affected muscle. Particularly, the visual attributes that appropriately encode the abstract data to be visualized are applied by rendering of the corresponding anatomical structures, changing their default or “natural” appearance.

Using the human body as spatial reference, we directly address the design goal DG1 of exploiting the existing visual tradition, which is tightly bound to the anatomy. On the one hand, this approach reduces the freedom in assigning medical data to visual attributes compared to InfoVis (see, e.g., [3, 7]), since the attributes ’spatial position’ and ’form’ are defined by the model and are no longer free parameters. On the other hand, the focusing on a specific cooperative context and on data relevant therein (cf. DG5), significantly reduces the amount of data to be visualized and, consequently, the number of required visual attributes. This data preselection in combination with in-place visualization provides a context-related synopsis at a glance (cf. DG3). Moreover, we expect that the spatial embedding of abstract medical data in the anatomical context makes its relation to the underlying real phenomena more evident and facilitates its interpretability (cf. DG4).

However, the intended anatomically integrated in-place visualization poses two main challenges, i.e.

How to design the mapping from abstract medical data to visual attributes under the restriction regarding location/geometry, such that visual discriminability (DG2), context-related synopsis (DG3) and intuitive comprehensibility (DG3) can be achieved in practice? 2. 2.

Even though a significant subset of medical data can be inherently related to an anatomical structure, how to deal with medical data that has no canonical relation to a specific anatomical structure, such as a hemogram?

In the rest of the paper, describing our visualization concept, its prototypical implementation and evaluation, we cover medical data with inherent anatomical reference. The integration of data without anatomical reference using, for example, more classical methods from InfoVis, will be addressed in future work.

4.2 Formal Definition and Requirements

The following provides a formal definition of the visualization problem and the requirements that any specific realization of the anatomically integrated in-place visualization has to fulfill.

4.2.1 Medical Data

The medical data to be visualized is structured as follows.

• •

Property. A property $p$ is defined as triple

[TABLE]

where $p.$propType refers to the underlying medical concept, $p.$dataType is a nominal, ordinal or numerical data type and $p.$domain is the domain set of elements of the corresponding $p.$dataType.

• •

Category. A category $c$ is a pair

[TABLE]

where $c$.spatialRef is the reference to the corresponding anatomical structure and $c$.props are all properties of $c$. $C$ is the set of all categories.

• •

View. A view $V\subseteq C$ is defined as the categories relevant to the given usage context inside the cooperative workflow. $\cal V$ is the set of all views.

Note: Views automatically reduce the amount of information to be visualized, and thus the necessity of user’s navigation, by preselecting the context relevant data.

The following examples illustrate the structural concept introduced above.

The category ‘muscle strength’ refers to the anatomical structure ‘muscle’, comprising a single property with the type ‘intensity’ that documents the strength a patient can create in a specific muscle, quantified in 6 numerical values.

[TABLE]

The second example describes the category ‘radicular pain’, i.e. pain caused by irritation of a nerve root and related to the skin region that is associated with the latter, i.e. dermatome. This category comprises one numerical and one nominal property.

[TABLE]

There are several fundamental relationships between views, categories and properties:

• •

A property type is unique within a category. Property types obtain their medical meaning only in combination with the category they are used in.

• •

A property type can be shared between properties of several categories. This expresses similarity of medical concepts, such as ‘intensity’ of clinical symptoms in the prior examples, even if the respective property domains can be distinct.

• •

A category is unique within a view.

• •

A category can be shared between several views, as it can be relevant in different usage contexts.

In oder to be able to properly specify the mapping of medical data to visual attributes, we define the set $T(c)$ of property types in category $c$ and the set $T$ of all property types as

[TABLE]

4.2.2 Visual Attributes

Since shape and position are mainly predefined as the anatomical structures that are used for visualization in our concept, the geometric visual attributes for data encoding are restricted to transformations or deformations such that they do not degenerate the respective 3D object. The other available attributes are color components, namely hue, brightness and saturation, textures and transparency, as well as time using, e.g., animations.

Formally, $a$ denotes a visual attribute, $A$ the set of all visual attributes, and $a$.range the discrete and finite set of distinctively perceivable values of $a\in A$. Note, that $\left|a\text{.range}\right|$ is commonly smaller than the number of displayable visual attributes. For example, the visual attribute ‘hue’ is a floating point value, however, the human vision is able to distinguish only up to eight hue values w/o reference (see, e.g., Kuehni et al. [18]). Similarly, we found that the values of visual attributes ‘saturation’ and ‘brightness’ become hardly distinguishable for a range cardinality of 5 and higher, if no reference color is given in the same scene. The presence of a visual reference, e.g. colorbars, near the visualization can significantly increase the number of distinguishable visual attributes and/or their values. However, displaying of colorbars for all attributes by default would introduce a distraction factor leading to visual clutter. On the other hand, calling these reference colorbars on demand requires as much user interaction as is necessary for accessing raw data by means of textual overlays, whereby the latter provides a more precise information. Thus, we avoid the use of explicit visual references and integrated textual overlays in our approach (see Sec. 4.3).

4.2.3 Visual Encoding / Mapping

The mapping of medical data to visual attributes comprises three levels, i.e.

(a) encoding of categories,

(b) encoding of property types,

(c) encoding of property values.

Consequently, the mapping has to allow for the visual retrieval of this information. The entire mapping process can be represented as:

Select a visual attribute $a^{C}\in A$ to represent categories. 2. 2.

Find a mapping $M^{c}\colon C\rightarrow a^{C}.\text{range}$ that maps all categories to values in the range of the selected visual attribute. 3. 3.

Find a mapping $M^{t}\colon T\rightarrow A\setminus\{a^{C}\}$ that maps all property types to the remaining visual attributes. 4. 4.

For each $c\in C$ and each $p\in c$.props find a mapping $M^{d}_{p}\colon p.\text{domain}\rightarrow(M^{t}(p.\text{propType}))$.range that maps the property domain values to the range values of the corresponding visual attribute.

We define $M^{c}$ and $M^{t}$ on the global set of categories $C$ and property types $T$, respectively, in order to achieve an intuitive comprehensibility according to DG4.

4.2.4 Injectivity Requirement

In general, injectivity is a pre-requisite for any visual encoding / mapping in order to lead to an unambiguously comprehensible representation (see, e.g., Ziemkiewicz et al.). We distinguish the following situations where either the mapping injectivity is strongly required or its violation has to be recognized and appropriately tackled.

Local Injectivity.

In the case the mapping $M^{c}$ or $M^{t}$ is not injective within a given view $V$ or category $c$, respectively, it is impossible to trace back the categories or property types from their visual representation. Local injectivity can be guaranteed if

1. all categories of a view are mapped to distinct values of $a^{C}$, and

2. all property types of a category are mapped to distinct visual attributes.

Formally, this is given, if

[TABLE]

Global Injectivity.

While local injectivity guarantees the visual distinctiveness of categories and properties inside each single view and category, respectively, the global injectivity ensures the uniqueness of visual encoding across respective contexts, i.e. categories and views for property types and views for categories. Formally, we have

[TABLE]

Due to the restricted distinctiveness of visual attributes, global injectivity is hardly achievable, still it should be pursued as far as possible.

Spatial Injectivity.

Obviously, several categories of a view can refer to the same anatomical structure. Formally, spatial injection in view $V$ is defined as

[TABLE]

The violation of spatial injectivity rules out simultaneous visualization of the respective medical data and needs to be handled explicitly (see Sec. 4.3).

Property Domain Injectivity

A non-injective mapping of a property domain to the respective visual attribute range causes quantization and, consequently, leads to a loss of information. In some cases a quantized visualization can be acceptable, in particular in the context related synopsis (cf. DG3), in which a qualitative overview is sufficient. In any case, quantization needs to be detected and reported. Formally, for a given property $p\in c.\text{props},\,c\in C$, quantization can be detected as follows:

[TABLE]

In general, quantization cannot be prevented and needs to be handled explicitly (see Sec. 4.3).

4.2.5 Visibility Restrictions

In general, a 3D visualization with free camera motion intrinsically affects the visibility of geometric objects representing anatomical structures. In our case, we have two causes for restricted visibility.

1. An anatomical structure that relates to relevant medical data (target) may be occluded by another anatomical structure. A muscle affected by paresis, for example, is commonly hidden below skin.

2. The spatial extension of an anatomical structure is too small in relation to the entire scene, so that the visualization cannot be clearly recognized. An example would be a tendon.

Both aspects are handled in our visualization prototype as described in Section 4.3.

4.3 Further Visualization Concepts

Independent from the specific mapping that we introduce in Sec. 5, there are further aspects that we added to our anatomically integrated in-place visualization concept in order to achieve the design goals postulated in Sec. 3.2.

Data selection.

Besides two already described data selection mechanisms, which apply automatically, i.e. focusing on the abnormal and usage-dependent views, we provide the user with the possibility to additionally filter the data by their categories. Note that the usage-dependent preselection reduces the available categories to a manageable amount. This filter allows to tackle, inter alia, the visualization of multiple medical data by means of same anatomical structure, i.e. spatial non-injectivity (see Sec. 4.2.4).

Alternating visualization.

In the constellations where a simultaneous visualization of multiple medical data on the same anatomical structure is not possible, i.e. the spatial injectivity is not fulfilled (see Sec. 4.2.4), an alternating visualization with additional user control to select one of the alternatives, e.g. through the data category filter, can be applied (see Fig. 4a-4b).

Textual overlays.

In some situations, the physician may want to access the underlying information explicitly on demand, i.e. in textual form. We enable this by textual overlays on top of the corresponding anatomical structure in order to, for example, resolve the quantization problem (see Sec. 4.2.4) that results in a simplified visual representation, or to support the physicians in the getting acquainted with our visualization tool.

Proxies.

The small object extension problem (see Sec. 4.2.5) can be tackled by means of an appropriately scaled proxy that is projected to the body surface over the location of the target anatomical structure and rendered with the corresponding visual attributes.

Transparencies.

In order to handle depth occlusions of anatomical structures carrying relevant information by other anatomical structures (see Sec. 4.2.5), we use view dependent (semi-)transparency for the occluder, in case the occluder itself is not carrying relevant information, while preserving the surrounding anatomical context.

The specific technical approaches taken to implement textual overlays, proxies and transparencies are described in Sec. 5.2.2.

5 Prototype Implementation

Based on the design goals stated in Sec. 3.2 and the visualization concept introduced in Sec. 4, we present a prototype implementation on a mobile device. As a proof of concept, we focus on patients with spinal disc herniation. Moreover, we concentrate on clinical symptoms that are relevant for the retrieval of a patient’s neurological status, which turns out to be the common usage context for the asynchronous cooperative situations such as ward rounds (cf. F1) and defines the main view of our prototype.

5.1 Development Environment

As geometric model for the anatomically integrated visualization we use plasticboy’s anatomical 3D human avatar111www.plasticboy.co.uk/store/Human_Male_Female_Anatomy_Complete_V05.html. The model already includes the main organ systems subdivided into the corresponding anatomical structures. However, the granularity of the hierarchical anatomic structure is partially too coarse and has to be extended in order to allow for a proper spatial mapping. Particularly, in the context of neurosurgery, the dermatomes constitute very important anatomical structures that are not reflected in the model. Hence, we have set up an anatomy refinement procedure based on indexed texture maps that is applied to the existing 3D geometries, i.e. polygonal meshes. Our procedure comprises 3D painting functionalities commonly available in modeling tools such as Maya. By “painting” the required anatomical sub-structure on the geometry of the (super-)structure and saving the resulting segmentation as indexed texture, these index textures can be used for looking up the anatomical sub-structures of the current fragment in the fragment shader.

Our rendering framework uses the Vulkan API under Android allowing an efficient resource management, which is important especially on mobile devices. In order to fit all high resolution model textures to the limited mobile graphics memory, we initially convert them into the ASTC format [22].

5.2 Prototype Features

5.2.1 Mapping of Spinal Disc Herniation Data

Raw Data Categories.

In collaboration with the neurosurgeons, we defined seven data categories (see Table 1) that are of high relevance with respect to the representation of neurological status of a patient with spinal disc herniation, i.e. in the respective view (cf. Sec. 4.2.1). The type of the main property to be visualized, common for all these categories, is the intensity with which the respective symptom manifests, whereas its domain is individual for each symptom, i.e. category.

The category radicular pain has a further property with the type trigger, which states if the pain is constant or only occurs under stress, e.g. during movements, whereby the former is assumed to be the normal, i.e. default situation for pain that does not need any visual indication.

Data Category Refinement.

During the first trials with the prototype and discussions with neurosurgeons, two changes to the initial raw data categories have been applied. First, the three categories related to sensory disorder, i.e. paresthesia, hypoesthesia and anaesthesia have rather complex interrelations. For example, hypoesthesia and anaesthesia can be considered as different stages of sense decrease, whereas paresthesia is in a certain sense orthogonal because it does not describe a decrease of sensation but rather its abnormality, e.g. tingling, and, thus, it can occur in combination with hypoesthesia. Therefore, the new category sensory disorder represents anaesthesia and hypoesthesia as a joint property with the type ‘intensity’ and has an additional property with the type ‘paresthesia’. Second, insufficient muscle strength, measured in the Medical Research Council scale $0,\ldots,5$, is used in daily clinical practice as indication for paresis that trigger for potential urgent actions such as emergency surgery. Therefore, we adopted this practice by using the category paresis with the ordinal data type comprising the values ‘mild’, ‘moderate’, ‘severe’.

Mapping of Data to Visual Attributes.

The final medical data categories in Table 1 are visually encoded in consideration of the rules deduced in Secs. 4.2.3 and 4.2.4, that is, the mapping functions $M^{c}$ and $M^{t}$ are at least locally injective. We selected hue as the visual attribute $a^{C}$ to encode category. The category mapping $M^{c}$ takes into account the distinctiveness of the resulting five hues with regard to each other as well as to their context in the anatomical model, e.g. a red color is not suitable for visualization on muscles because it highly coincides with their natural, i.e. healthy, appearance. The shared property type intensity is mapped in all categories to a composite visual attribute saturation-brightness in the HSV color space, i.e. $M^{t}$ is also globally injective. We decided to combine two respective visual attributes in a single one to increase the visual discriminability (DG2) especially in cases of the visualization of a single property value that is difficult to assess if no color reference, e.g. colorbar, is present (see discussion in Sec. 4.2.2). Additional properties are encoded by means of textures. We select texture such that they are visually as complementary to the visual attribute color as possible in order to allow for a simultaneous visualization. In case the additional property is non-binary, e.g. the property with the paresthesia type in sensory disorder (cf. Sec. 1), the required range of visual values corresponds to the texture’s frequency and amplitude; see also Fig. 3.

There are several features of the mapping to visual attributes that need to be mentioned (cf. Table 1):

(1) The 10 intensities of the pain category result in a visual quantization, i.e. the corresponding property domain mapping is not injective (cf. Sec. 4.2.4),

(2) the anatomical reference for the categories T-reflex and excretion disorder are too small and require a proxy geometry (see also Sec. 5.2.2 and, e.g., Fig. 4), and

(3) we can slightly increase the saturation-brightness cardinality by $1$ for the main property of the category T-reflex as abnormal intensity values occur in anatomical pairwise constellations, e.g. on the left and right leg and the main indication is their difference.

5.2.2 Rendering Implementation Details

In this section, we briefly describe some implementation details related to the realization of specific visualization features, partially already discussed in Sec. 4.3.

3D Solid Textures.

The interference of the textures, applied for visualizing additional data properties, with the color components hue (visual attribute to encode categories, $a^{C}$) and saturation-brightness (property type intensity) that are already assigned to the same anatomical structure should be as low as possible (cf. DG2). This is achieved by using textures that either sparsely vary the intensity or create lighting effects by means of a normal perturbation, i.e. procedural normal maps. The texture coordinates provided by the 3D human body model refer to texture atlases, which introduce geometrical distortion and whose segmentation does not follow the anatomical structures, i.e. we have no means of applying textures directly to the given coordinates. Therefore, we use 3D solid textures and the vertex positions as texture coordinates to achieve a proper and distortion-free texture parametrization [24]. In particular, we chose the procedural form of solid textures that allows to synthesize fragment color on the fly in a shader without any consumption of GPU global memory on our mobile device, which has limited hardware resources. The examples in Table 1 (see Fig. 3 and Fig. 4c) are stochastic textures based on Perlin’s noise (Texture Noise) and on normal perturbation using cycloidal functions (Texture Normal Pert.) (see  [23, 15]).

Anatomical Proxies.

The main idea in generating anatomical proxies is to utilize projective textures [28], e.g. appropriately scaled circles, on the skin surface above the anatomical structure that is too small for a direct visualization (cf. Sec 4.2.5 and the patellar reflex, e.g., in Fig. 4). To achieve this, several pre-processing operations are performed. Firstly, we compute a principle component analysis (PCA) of the target anatomical structure on the GPU using the vertex data stored in buffer objects in graphics memory, thus preventing unnecessary data transfer. The PCA’s center point serves as lookat point of the projection, whereas the PCA’s main axes are used to realize optional shifts or scales of the proxy consistently with the object’s extent, e.g. in the case of the triceps tendon reflex the lookat point is shifted to the muscle attachment on the elbow. Secondly, we calculate the minimum and maximum skin depth values in projector space by rasterizing the avatar’s skin into two depth buffers using a fragment shader and a subsequent minimum and maximum finding in a compute shader. In order to restrict the texture projection to the avatar’s skin surface closest to the anatomical structure and to prevent projection onto the avatar’s far side, we use the midpoint of the minimum and maximum depth value inside the projector footprint as depth threshold in projection space and discard all fragments that are greater or lesser, depending on whether the target anatomical structure lies near to the front or back body side, respectively.

Occlusion Handling

In order to visualize hidden anatomical structures in an integrated overview of the most relevant information without requiring specific navigation or zooming efforts, we dynamically decrease the opacity of areas above the occluded target structure, depending on the current camera transformation (cf. Sec. 4.2.5). Similar to Viola et al. [33] and Burns et al. [5], we use an image space approach that allows an efficient detection of occluding fragments in real time on mobile hardware. First, all anatomical structures that carry information are rendered in an offline step creating their footprint in the depth buffer, which is implemented as Vulkan storage image. Subsequently, the opacity of the fragments that are nearer to the camera than the corresponding depth value in the footprint is modified. In our approach we set the opacity close to zero in order to prevent interference with the color of the underlying structures that might degrade a proper recognition of referenced medical data. Furthermore, we extend the transparency region by a margin with increasing opacity depending on the pixel distance to the footprint edge; this improves the visible perception of the transparent cutouts.

For a better depth perception, the anatomical hierarchy is also taken into account during occluder detection. In particular, only anatomical structures that are part of the same body region are considered as potential occluders, e.g. the left tibialis anterior muscle can be seen through the structures of the left lower leg but not through the right foot (see Fig. 1). For this purpose, we additionally store the body region’s ID in the appropriately extended depth buffer and compare it at rendering time in the fragment shader with the ID of the potential occluder.

Finally, the (semi-)transparent fragments are blended into the framebuffer image. Here, the correct rendering order is crucial. Therefore, we sort the fragments in image space in front-to-back order using a depth peeling approach [10]. In order to reduce the number of render passes, we use the Vulkan subpass concept combining respective peeling and blending steps.

We accelerate the above algorithm by extending the depth/region buffer with a bit mask generated in the first depth peeling pass that masks out image regions that do not contain target objects, i.e. anatomical structures that carry medical information. The pixels that do not overlay the footprint with its extended margin will not become transparent fragments and will be discarded in all subsequent peeling and blending passes.

Picking and Context Menu

The prototype includes a context menu for accessing the advanced features such as the data category filter (see Fig. 5a) and the overlays with textual data (see Fig. 5c). The filter provides for the user the possibility to hide/show data visualizations by their category as described in Sec. 4.3, e.g. for handling of the spatially non-injective cases (see Sec. 4.2.4). The overlays display textual data corresponding to the visualization on a anatomical structure selected by user. This, in turn, requires an appropriate mechanism for picking of 3D objects. In our prototype the picking is realized in the image space and is combined with the offline step of the occlusion handling described above. More precisely, the ID’s of target anatomical structures are written in the extended depth buffer during occlusion handling. The ID of the current pixel position can be read out, copied back to the CPU memory and used for retrieval of the associated data.

Visualization of Temporal Changes

For assessment of healing progress a slider with dates of available clinical examinations is integrated below the main 3D view (see Fig. 4). Moving the slider, the user can navigate to the date of interest or scroll consecutive examination results. By selecting a date, the entries with the corresponding timestamp are retrieved from the database and the virtual body is rendered with updated visual attributes.

6 Prototype Evaluation

6.1 Goals and Procedure

The main evaluation objective is to verify the efficiency and effectiveness of the prototype concerning transfer of the cooperation relevant information in general, and its conformity with the design goals (see Sec 3.2) in particular. In order to test the aforementioned aspects, the evaluation is structured as follows.

(1) After a brief introduction to the overall context, the participants are directly confronted with item Task 1.

(2) Afterwards the legend is explained to the participants (see Fig. 3) and they have to perform item Task 2 and item Task 3.

(3) Finally, the participants have to reflect on the practical experience answering a questionnaire that comprises two main blocks, i.e. QNR 1 and QNR 2.

The three practical tasks are designed as follows:

1. Task 1.

The goal of the first task is to assess the potential of our visualization approach for intuitive interpretability (cf. DG4). Here the physicians need to give their spontaneous interpretation of two visualizations (see Fig. 2) without previous knowledge about the meaning of the visual attributes. 2. Task 2.

During the participatory refinement with the expert group (see Sec. 3), we collected seven real-life descriptions of neurological statuses of patients with spinal disc herniation (cf. DG3, DG5). All seven statuses are described in textual form similarly to the case presentations in the department morning meetings and one status is visualized with our prototype (see Fig. 4a and 4b). In this task, the physicians have to assign the given visualization to the corresponding textual description. A specific challenge arises from the fact, that the visualized status is very similar to another one in the list (cf. DG2). More precisely, the respective datasets differ only by the relatively small muscles in the right leg that are affected by paresis, which are anatomically adjacent, and both are partially occluded by the L5 dermatome.

The given dataset allows to test the effectiveness of the prototype features such as alternating visualization (representing the radicular pain and sensory disorder in same dermatome, in this case), dynamic occlusion handling (over the affected muscles), proxy geometries (for the patella tendons), and 3D navigation. Moreover, in order to solve this task correctly, the participants must also use the category filter, which hides the occluding dermatome (see Fig. 5a and 5b), and, if necessary, activate the textual overlay (see Fig. 5c), e.g. to identify the exact muscle strength level or muscle name. 3. Task 3.

The last task is to read the healing progress (cf. DG6) from two visualizations that show successive patient states. The first one equals Task 2 and corresponds to the preoperative status of a patient. The second one represents the postoperative neurological status of the same patients. Switching between both visualizations by means of the timeline, the physicians have to recognize (cf. DG2) and to describe the respective changes. Analogously to Task 2, there is an additional level of difficulty due to the partially remaining paresis that is still occluded in the postoperative status (see Fig. 4c).

The final questionnaire that is to be answered by the participants comprises the following two main blocks:

1. QNR 1.

The first questionnaire is dedicated to the evaluation of the usage-dependent view (cf. DG5). Particularly, the physicians assess the relevance of the pre-selected data categories in the given cooperative settings as well as their completeness. 2. QNR 2.

In the second block the physicians assess the benefit of the prototype for the transfer of information regarding a patient’s neurological status. Moreover, they are asked to explicitly specify advantages or drawbacks of the prototype’s features, such as using the anatomy as spatial representation for data visualization (cf. DG1), simultaneous visualization of multiple data (cf. DG3) and implementation on a mobile device (cf. DG7).

Ten neurosurgeons from the same department took part in the evaluation, none of which was involved in the participatory refinement process described in section 3, i.e. this test group only had a very basic understanding of the overall aim of the visualization prototype. The group consists of three assistant physicians, one specialist, five senior physicians and one chief physician. The interviews were conducted in groups of two persons or, partially, individually, and have been filmed for documentation and analysis. The response rate of questionnaires has been nine out of ten.

6.2 Results

In the following we present the main insights of the prototype evaluation, structured according to the aforementioned tasks.

The results of Task 1 demonstrate that the anatomical integration (cf. DG1) and the usage-dependent view (cf. DG5 allow a rather intuitive interpratation by the physicians (cf. DG4). They could successfully focus on the set of visual attributes related to the respective anatomical structure in the given context and interpret the underlying neurological information to a large extend w/o any pre-knowledge. For instance, the purple color in the muscle (see Fig. 2b) was consistently associated with paresis (10/10) and the interpretation of the red dermatome (see Fig. 2a) varied between pain (8/10) and sensory disorder (2/10). At the same time, the data category tendon reflex has often been misinterpreted (6/10). However, several participants rated this category as less relevant (see QNR 1), which is why it was less anticipated. In particular, the option ‘useful but dispensable’ in the given view got by tendon reflex ca. 45% of all votes, whereby the result of the respective questionnaire column for other categories vary between 11% and 20%. Moreover, the visualization of the tendon reflex is anatomically weakly integrated due to the required proxies, which also affects its interpretability.

To sum up, the prototype shows a considerable degree of intuitive comprehensibility (see DG4), which also was confirmed in the next tasks where the participants were able to successfully handle it after a very short learning phase. Therefore, we conclude that there is a strong relation between anatomical integration and view-dependent relevance of the data on the one hand, and intuitiveness of their visual interpretation on the other hand.

In Task 2, i.e. the assignment of a visualization to textual descriptions, all physicians could easily (10/10) limit the conceivable variants to both cases: the correct case and one that was very similar, as described above. However, for the decision between both cases the most participants needed further hints, e.g. to use the category filter and the zooming function. Notably, after receiving these hints, several participants (6/10) could familiarize themselves with the prototype features and independently apply them in Task 3 for the correct recognition of the remaining paresis.

In summary, the visualization prototype allows the physicians to successfully read the neurological status of a patient with spinal disk herniation (cf. DG3, DG5). At the same time, a limited visibility of target anatomical structures, particularly several small targets structures close to each other (see Fig. 5b) or occlusions (see Fig. 4a-4b), can affect the discriminability of the respective visualization (cf. DG2) and consequently complicate its reading. Apparently, the users need a higher degree of familiarity to handle these problematic situations and to apply prototype features such as 3D navigation and category filtering. We will further investigate approaches with a more intuitive access to the synopsis of the patient’s status (cf. DG3, DG4) in these kinds of more complex situations.

The healing progress in Task 3 (cf. DG6) was interpreted correctly by all participants (10/10). Most physicians also emphasized a high practical relevance of this kind of visualization. Note, that the correct interpretation of relative value changes, i.e. improvement or deterioration, was possible without consulting the legend or calling the textual overlay.

Moreover, several aspects addressed in the questionnaire blocks QNR 1 and QNR 1 have lead to additional insights discussed in the following.

The evaluation of QNR 1 validated the relevance of the pre-selected data categories for the specified view (cf. DG5), only the tendon reflex has been rated controversially. The latter is explained by individual variations in the clinical examination practices: while some physicians rarely apply the tendon reflex test, others consider it useful for obtaining additional insight into the patient’s status. Regarding the completeness, five physicians proposed to extend the view with further categories or properties, e.g.’pathological reflex’ as category and ’duration’ as type of a symptom property. These suggestions, again, mainly reflect differences in the individual professional procedures. Further suggestions, e.g.’previous surgeries’, indicate the interest of using the prototype in further contexts by adding new views.

All, i.e. nine out of nine, responses of QNR 2 positively assess the potential of the visualization concept for enhancing and accelerating of the information transfer in the targeted cooperative situations. Especially, rating positively the simultaneous visualization (cf. DG3) and the use of the human anatomy (cf. DG1), the physicians stated concrete advantages such as providing of multiple pieces of information at a glance and consulting the model for refreshing their anatomical knowledge. They also mentioned potential drawbacks in relation with these concepts such as a growing visualization complexity by increase of the amount of data in a single view or a long learning curve.

The physicians see the main contribution of the implementation on a mobile platform (cf. DG7) in the possible time saving thanks to the high availability of patient data. At the same time, they expressed their concerns regarding the necessity to carry an additional device in their coat pocket.

In summary, it can be stated that the anatomically integrated in-place visualization of medical data was appreciated by the most participants: only one of them could not discern any benefit in the use of our approach in cooperative situations in comparison to textual data. In contrast, one neurosurgeon unsolicitedly drew the analogy between our prototype and the MR images that are used for cooperative surgery planning, as in both cases the specialists immediately “see” the relevant information. This clearly suggests, that the visualization concept can also be used in synchronous cooperative settings, for instance in the discussions during the department morning meetings.

7 Conclusion and Future Work

In this paper we presented a novel concept for anatomically integrated in-place visualization of medical data. The concept is designed in accordance with the requirements arising from specific tasks in cooperative clinical workflow, namely transfer of cooperation relevant neurosurgical information between colleagues. Our approach allows for a spatially integrated comprehensive visualization of abstract medical data, such as clinical symptoms, on a 3D human avatar using their inherent references to affected anatomical structures and an appropriate visual encoding. Preselecting patient data as a function of their relevance in the given clinical usage context, i.e. view, provides an at-a-glance synopsis of relevant information to physicians.

The evaluation of the prototypical implementation of the visualization concept by a group of neurosurgeons revealed positive feedback, in particular concerning the use of anatomy as spatial representation of data and the potential speed-up of information assessment. At the same time, it revealed some limitations of the current solution in the situations where the target anatomical structures are not sufficiently distinguishable without user interaction.

Beyond the extension of our prototype to further clinical usage contexts, we intend to address several of the issues that have arisen in the evaluation, such as the flexibility to adapt to individual professional procedures and the problem in distinguishing adjacent anatomical structures.

Acknowledgments

The authors would like to thank the medical staff at the neurosurgery department of the Jung-Stilling hospital for their willingness and engagement in our field and user studies. The work is funded by the German Research Foundation (DFG) in the context of the Collaboratice Research Center (SFB) 1187 “Media of Cooperation”, sub-project A06 “Visual integrated medical cooperation”.