TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy

TL;DR

TempoCave is a visualization tool designed to analyze dynamic brain networks from fMRI data, aiding clinical psychologists in understanding neurological disorders and treatment effects through interactive exploration of connectome datasets.

Contribution

It introduces a novel visualization application that supports dynamic connectome analysis, tailored for clinical use in understanding brain activity related to mental health treatments.

Findings

Analyzed pre- and post-treatment connectomes from 27 subjects.

Provided new insights into the dynamic behavior of the human brain.

Supported comparative analysis of brain network metrics.

Abstract

We introduce TempoCave, a novel visualization application for analyzing dynamic brain networks, or connectomes. TempoCave provides a range of functionality to explore metrics related to the activity patterns and modular affiliations of different regions in the brain. These patterns are calculated by processing raw data retrieved functional magnetic resonance imaging (fMRI) scans, which creates a network of weighted edges between each brain region, where the weight indicates how likely these regions are to activate synchronously. In particular, we support the analysis needs of clinical psychologists, who examine these modular affiliations and weighted edges and their temporal dynamics, utilizing them to understand relationships between neurological disorders and brain activity, which could have a significant impact on the way in which patients are diagnosed and treated. We summarize the core functionality of TempoCave, which supports a range of comparative tasks, and runs both in a desktop mode and in an immersive mode. Furthermore, we present a real-world use case that analyzes pre- and post-treatment connectome datasets from 27 subjects in a clinical study investigating the use of cognitive behavior therapy to treat major depression disorder, indicating that TempoCave can provide new insight into the dynamic behavior of the human brain.

Full text

Introduction

The human brain is comprised of trillions of structured neurophysiological connections. Clinical neuroscientists create connectome representations of this vastly complex biological network via functional magnetic resonance imagine (fMRI) scans [24], producing a map of the connectivity between cortical and subcortical brain regions. Understanding the structure and functionality of the connectome is a fundamental goal of cognitive neuroscience and neuropsychology, and provides insight into the behavior of various brain conditions [31]. Even when displaying the (relatively) low-resolution network representations generated by non-invasive neuroimaging techniques (where a node represents a brain region “voxel” containing millions of neurons), the visual complexity can be overwhelming. Nonetheless, this network of nodes enables graph-theoretic methods that extract useful connectome metrics, and the edges linking the nodes provide finer details about the strength of the connectivity between regions. This complexity is exacerbated when analyzing dynamic connectomes [9, 28], in which a user investigates temporal patterns across the sequence of frames generated during a dynamic fMRI scanning session. Moreover, comparing two (or more) connectomes is necessary when evaluating pre- vs. post-treatment changes [7], negative vs. positive connections [38], or healthy vs. unhealthy connectomes [29], which introduces additional visualization challenges.

In this paper, we introduce TempoCave, a visualization application that facilitates the exploration and analysis of time-series connectome datasets. We worked closely with our neuroscience collaborators to identify aspects of their analysis workflow not yet supported by existing tools, and to determine which tasks are most relevant for making sense of dynamic connectomes in clinical contexts:

T1: Measuring the dwelling time of brain regions, defined as the length of time a region spends in the dominant community.

T2: Measuring the flexibility or particular brain regions, defined as the number of times a region changes modular affiliation.

T3: Analyzing the connectivity between brain regions that is used to define modular affiliations, including edges that are defined by either a negative or positive correlation between nodes.

T4: Understanding the overall dynamics, or “stickiness” of the connectome, in terms of how dwelling, flexibility, and connectivity metrics change over time.

T5: Enabling the comparison of multiple dynamic connectomes, for example, comparing a connectome to a group average, or comparing an individual patient’s connectome pre- vs. post-treatment.

These tasks have broad relevance to psychiatry, as a wide range of neurological disorders are linked to the disruption of normal brain connectivity. An important overarching goal of clinical neuroscience is to find relationships between brain activity and neurological disorders, which can then be leveraged to make diagnoses, to guide treatment plans, and to better understand the human brain. We developed TempoCave iteratively, adding relevant visualization features as we worked successively with individual patients’ connectomes and group average connectomes for autism spectrum disorder, anxiety disorder, and major depressive disorder. In this paper, we focus on connectome datasets gathered during a clinical study on rumination, a mental disorder characterized by repetitively and passively focusing on symptoms of distress and its causes. The consequences of prolonged rumination include anxiety and depression [5, 21, 33].

Rumination-focused cognitive behavior therapy, or R-CBT, assists individuals in realizing that their rumination about negative experience can be unhelpful, and coaches them on how to shift to a more helpful style of thinking. For example, patients undergoing R-CBT are asked to remember previous positive mental states, such as a time they were completely absorbed in an activity— the opposite of ruminating [20, 34]. R-CBT appears to be effective at supporting emotion regulation in patients suffering from depression. A preliminary study (described in more detail in Sec. 3) finds that patients who continued treatment remained in remission after 8 weeks, whereas patients who did not continue treatment had a higher likelihood of relapse. TempoCave facilitates insight into how R-CBT (and other clinical interventions) alters brain behavior. In order to conduct these analyses, clinical neuroscientists make detailed measurements of the brain activity of patients and compare the patients’ connectomes before treatment and after treatment, and also compare them against baseline healthy connectomes. Visualizing the dynamics of connections within and between brain regions, measured by dwelling time and flexibility, helps clinical psychiatrists examine the “stickiness” of these connections, where comparatively high measurements of stickiness can imply an unhealthy connectome.

To our best knowledge, TempoCave is the first application to support comparison tasks for multiple time-series connectome networks in order to better understand rumination. Our contributions include: (a) a delineation of analysis tasks relevant to reasoning about dynamic connectome datasets; (b) the introduction of a new visualization tool to support connectome-based comparison tasks for both static and dynamic networks; (c) techniques for observing and analyzing community affiliation of nodes in a dynamic network; (d) an analysis pipeline that supports easy data loading of multiple connectomes, including those in alternative topological spaces [37] or generated using different modularity identification algorithms [12, 38]; (e) a real-world use case illustrating how TempoCave is used in a clinical setting to elucidate new insight into neurological aspects of rumination. Fig. TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy presents an overview of TempoCave comparing two dynamic connectomes.

1 Background & Related Work

TempoCave is inspired by previous approaches for visualizing networks [4], comparing graphs [15], and exploring connectome datasets [27]. Keiriz et al. [18] survey the landscape of connectome visualization, focusing mainly on static datasets, and characterizing them in terms of their main visualization modality (emphasizing volume, surface, or graph representations), as well as identifying tools that include support for virtual reality (VR) environments. TempoCave presents connectomes as 3D networks, similar to approaches presented in Connectome Visualization Utility [22], BrainNet Viewer [35], Connectome Viewer Toolkit [14], and the AlloBrain project [32]. However, TempoCave specifically focuses on visualizing nodes and edges to make it easier to reason about metrics associated with modular affiliation and connectivity.

TempoCave provides features for dynamic data analysis, presenting a synchronized playback mode that highlights differences between two connectomes at different points in time. A number of 2D visualizations have been used to explore dynamic connectome data [17, 2, 26, 10], and Beck et al. [4] summarize approaches to visualize 2D dynamic networks. Other tools instead utilize a 3D layout for investigating dynamic connectome data [16, 23, 25]. For example, Xing et al.’s Thought Chart [36] presents distinct 3D trajectories for different task conditions and provides a comparative analysis that generates a summary view of how much one dynamic connectome differs in comparison to others. Similar to our implementation, Arsiwalla et al. introduce BrainX3[3], an interactive and immersive 3D visualization of dynamic connectomes, but which does not support comparison tasks, a main feature of TempoCave.

Alper et al. [1] investigate visual encodings of edge weights within an adjacency matrix to support comparisons of brain connectivity patterns. TempoCave further emphasizes comparisons of modularity affiliation metrics that summarize the edge weights in the network. Other recent brain visualization tools focus on neurobiological tasks, such as Ganglberger et al.’s [13] BrainTrawler, which provides tools to conduct integrated analyses of genomic data and mesoscale neuroscience datasets at the level of individual neurons. TempoCave aims to support tasks T1-T5 relevant to diagnosing and treating patients with neurological disorders, extending previous work that focused on static connectomes [18], which also presents a 3D view and enables a user to explore data immersively [8, 11].

2 The TempoCave Application

TempoCave is an interactive tool that enables clinical psychiatrists to load, visualize, and analyze dynamic connectomes, solving several technical challenges associated with the analysis tasks described in Sec. TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy. TempoCave supports the investigation of multiple connectomes at once, superimposing connectomes to make comparisons, and providing details about modular affiliation and edge dynamics that are needed in order to understand some dynamic datasets.

Our clinical neuroscience collaborators capture connectome datasets using high-resolution functional magnetic resonance imaging (fMRI) scans. The fMRI data is pre-processed to extract node and edge information, such as connection strength, and to perform various dimensional reduction steps. Dynamic datasets are obtained using scans taken at regular intervals over a short period of time. In the clinical study presented in Sec. 3, 200 frames are captured over the course of a $\sim$6 minute scanning session, and then further processed using the PACE algorithm [38] to determine modular affiliations.

TempoCave consists of a selection interface (Fig. 1 left), and an interactive inspection interface (Fig. 1 middle and right). The selection interface displays show an overview “carousel” of the available connectomes, along with a list of all the available layouts (generated through different dimension reduction algorithms, such as Isomap or t-SNE). The user can select two (in desktop mode) or more (in VR mode) connectomes for analysis and comparison. The inspection interface shows the selected connectomes with each connectome having their own settings panel, which provides options: to change the representation of the connectomes, to update the classification of different brain regions, to filter edges based on their connection strength, and to toggle edge bundling to mitigate visual clutter.

Summarizing the modular affiliation or “stickiness” of the nodes in dynamic connectomes is an important aspect of analyzing a patient’s connectome, as is investigating the amount of time that brain regions are associated with particular communities (dwelling time) and the frequency that brain regions change affiliation (flexibility). Connectomes demonstrating patterns of more rapid modular change are believed to indicate healthier connectomes. TempoCave automatically processes the summary statistics for each dynamic connectome upon being loaded into the application. For each node in the connectome, changes in the affiliation across the time steps are used to determine the flexibility metric. Dwelling time is calculated by identifying the maximum time a node is associated with a specific module. TempoCave uses color-coding to represent both dwelling time and flexibility, supporting T1 and T2. Fig. 2 shows how TempoCave is used to analyze flexibility in a rumination study.

To analyze changes in dwelling time, flexibility, and edge connectivity over time, TempoCave provides controls to scrub through the time steps. Users can play and pause, move forward or backward one or more time steps, adjust the playback speed, and, if two connectomes are available for inspection, there is an option to synchronize the playback settings. At each time step nodes are colored based on their modular affiliation. The edges change their width based on the strength of the connectivity between two nodes, and are colored with a gradient representing the regions they are associated with. The edges can further be classified into positive connections if the regions are correlated or negative connections if they are uncorrelated. TempoCave supports an optional color-coding to show the negative and positive edges for each time step, as shown in Fig. TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy. The dynamic visualization features support tasks T3 and T4. Fig. 3 shows an example of how the edge connectivity and modularity change over time in an analysis session using TempoCave.

TempoCave presents an overlay comparison mode supporting T5, where two connectomes are juxtaposed to form an integrated layout. Each node in this superimposed view is split into two halves, corresponding to left and right connectomes. As the user moves through the different frames, each half of the node’s colors change separately based on the modular affiliation of the associated connectome. The edges can be activated by clicking the corrresponding half of the node. To distinguish the connectivity of two connectomes, we use a solid line for left connectome and a dashed line for the right connectome. A comparison of a pre- vs post-treatment connectome is shown in Fig. TempoCave: Visualizing Dynamic Connectome Datasets to Support Cognitive Behavioral Therapy.

We developed TempoCave using the Unity Engine, which renders 3D data at real-time frame rates and provides out-of-the-box solutions for immersive applications. We evaluated a range of visual encodings and interaction modalities to determine useful representations and interactions that supported the analysis of dynamic connectomes. For example, we initially included animation as a primary encoding for dynamic network features [30]. While our collaborators found the animation engaging, ultimately it was distracting and introduced visual fatigue when they needed to scrub through many time steps. Instead, we color-coded edges to represent both modular affiliation and edge weights, giving the users the option to choose which encoding was most useful for a particular analysis session. We also experimented with a wide range of shapes to represent brain regions, hoping to more clearly distinguish nodes from different connectomes in the comparison view. However, our users found that it was easier to interpret a node when rendered as a multi-colored sphere. Contrary to expectations, we discovered that our users preferred to have fewer available encodings overall, but more options to control which data elements these encodings represent.

3 Use Case: Investigating R-CBT Treatment

As an initial validation of TempoCave, we explored a dataset from an ongoing clinical study that measures the effectiveness of rumination cognitive behavioral therapy (R-CBT) for adolescent patients with at least one previous episode of major depression disorder (MDD) [6]. In this study, 27 patients were recruited for comparing 8 weeks of R-CBT data against a control group of 15 healthy participants. The patients were in remission at the start of the study, and the main goal of the study was to measure the effectiveness of R-CBT at preventing a relapse of depression. Of the 27 patients with MDD, 14 were administered a treatment of R-CBT, and none had a relapse during that time. Of the 13 who were not administered R-CBT, 4 patients had an episode of MDD, providing initial evidence of the efficacy of R-CBT.

Using TempoCave, clinical psychiatrists observed the modularity dynamics of individuals’ connectomes. Participants from a healthy control group were found to have a higher overall flexibility than patients with MDD, and that on average there is no significant difference between the pre- and post-treatment connectomes of MDD patients who received R-CBT. However, as depicted in Fig. 2, the post-treatment connectome of MDD patients who did not receive R-CBT (and relapsed) shows much less flexibility than was observed in their pre-treatment connectome. Fig. 2 highlights the supramarginal gyrus and angular gyrus, two regions that are indicated in depression disorder. Fig. 3 shows an overlay comparison view of a patient’s pre- and post-treatment connectomes who received R-CBT, where the angular gyrus is selected. Looking at the right half of the selected node (from the post-treatment connectome), clinical psychiatrists found that the angular gyrus changed modularity three times (from t=2 to t=14), but that in left half of the selected node (from the pre-treatment connectome) the modularity of angular gyrus changed only once (from t=8 to t=14). This visual analysis corroborates the hypothesis that R-CBT mitigates rumination, as indicated by the temporal dynamics of dwelling time and flexibility metrics.

4 Conclusion & Future Work

Our initial use case already indicates that TempoCave helps clinical neuroscientists form new hypotheses about dynamic connectome datasets, and in particular that the comparison mode is useful for providing insight into patterns that emerge when investigating a patient’s response to treatment. Future work will explore a wider range of use cases in various clinical contexts. Additional definitions of modularity could generate network metrics that may be useful for understanding brain dynamics. For instance, recent work by Kim and Lee [19] introduce an “inconsistency” metric which can be used as an alternative definition of node centrality. Furthermore, while TempoCave provides textual labelling of nodes and edges, our collaborators indicated the need for including additional annotation options, which would make it easier to share results or hypotheses with other clinicians and to include as figures in presentations and articles. We also plan to extend our approach to other (non-connectome) dynamic datasets, and to conduct user studies to determine the effectiveness of our visual encodings as a more general approach for highlighting dynamic network features and comparing networks. TempoCave is available via our open source GitHub code repository at https://github.com/CreativeCodingLab/TempoCave, along with source code, detailed instructions on how to load in custom datasets, and additional documentation.