Interactive Visualisation of Hierarchical Quantitative Data: An Evaluation

TL;DR

This study evaluates and compares the effectiveness of treemaps, icicle plots, sunburst charts, and a new sundown chart for visualizing hierarchical quantitative data, revealing user preferences and performance differences.

Contribution

It introduces and assesses a semicircular sundown chart, providing insights into visualization effectiveness and user preferences for hierarchical data.

Findings

Treemaps were least preferred and slowest in navigation.

Icicle plots and sundown charts had similar performance.

Users slightly preferred icicle plots over sundown charts.

Abstract

We have compared three common visualisations for hierarchical quantitative data, treemaps, icicle plots and sunburst charts as well as a semicircular variant of sunburst charts we call the sundown chart. In a pilot study, we found that the sunburst chart was least preferred. In a controlled study with 12 participants, we compared treemaps, icicle plots and sundown charts. Treemap was the least preferred and had a slower performance on a basic navigation task and slower performance and accuracy in hierarchy understanding tasks. The icicle plot and sundown chart had similar performance with slight user preference for the icicle plot.

Full text

\CCScatlist\CCScatTwelve

Human-centered computingVisualizationTreemaps; Visualization design and evaluation methods

1 Introduction

Hierarchically organised quantitative data occurs in wide variety of contexts such as budget reports, CPU resource use, share portfolio allocation, and disk use. A number of different visualisations have been suggested to analyse such data [33]. The best-known is the treemap [23]. This is included in most modern data visualisation tools including Tableau and Microsoft Excel. Alternatives include the icicle plot, recently used to show “stack-traces” [20], and sunburst charts, used to visualise space allocation on a hard-disk [12].

However, to date there has been no comprehensive evaluation comparing the effectiveness of these three visualisations. Past evaluations  [39, 4, 18] have used small flat datasets, provided only very limited interactive navigation and/or used low resolution displays. The main contributions of this paper are:

• •

A novel visualisation, sundown chart, a semicircular variant of sunburst chart that is better suited to the wider aspect ratio of modern displays and reduces user disorientation during zooming.

• •

A carefully designed interactive navigation framework that is consistent across the four visualisations. This combines overview + detail, in-place zooming, breadcrumbs and uses unit visualisation [14] to reduces user disorientation during zooming.

• •

A hybrid WebGL and D3 implementation that allows responsive interactive visualisation of large quantitative hierarchies with more than 50,000 nodes using standard internet browsing tools.

• •

User evaluations of the visualisation designs. An initial pilot study with 6 participants compared all four visualisations. Feedback indicated that the sunburst chart was the least preferred. A second controlled study with 12 participants compared icicle plot, sundown chart and treemap. Treemap was clearly the worst, leading to slower performance task accuracy as well as being the least preferred. The icicle plot and sundown chart had similar performance with a slight user preference for the icicle plot.

2 Related Work

Treemaps, see Figure 1(c), use a space-filling design in which area denotes the quantitative node attribute and inclusion the node hierarchy [23]. They can be used to show both categorical and quantitative data: here we focus on quantitative data. The original “slice-and-dice” algorithm [23] led to nodes with high aspect ratio and unstable layout when the level of detail changed. Subsequent algorithms were designed to reduce aspect ratios [43, 8] and to improve layout stability [6, 38].

Icicle plots, see Figure 1(a), were originally developed as a method for presenting hierarchical clustering [27]. They use length to represent the quantitative attribute and juxtaposed layers to denote the hierarchy. Designs based on icicle plots have had a resurgence in recent years, e.g. SyncTRACE [24], and Flame Graph [20].

Sunburst charts, see Figure 1(d), are a radial version of the icicle plot. Each node is represented by a curved segment whose angle reflects the node quantitative attribute. An advantage of the radial layout is the increased area to display nodes at deeper levels of the hierarchy. The “Aggregate treemap” [10] and “Information Slices” [3] were early implementations of radial-space filling designs. The Aggregate treemap used an incomplete circle, and “Information Slices” used cascading semi-circular discs. The original sunburst chart was introduced in [40]. Like treemaps, the sunburst chart has achieved some popularity with recent integration into Microsoft Excel and use in applications such as “DaisyDisk” [12].

Task taxonomies developed for graphs and general visualisation techniques [28, 35, 37, 2] provide a foundation for understanding the tasks specific to visualisations of hierarchical quantitative data. There are two main kinds of tasks: understanding the hierarchy structure (tree topology), e.g. depth or size of hierarchy, finding local relatives or distant relatives such as the least common ancestor, navigation through the hierarchy; and understanding the quantitative node attributes, e.g. finding nodes with a specific value or extremums, comparing values. In addition, there is the basic navigation task of traversing the hierarchy. It has been suggested that there is a conflict between designs that effectively represent hierarchies and those conveying the quantitative node attributes [34].

User evaluations: a number of studies have evaluated new more stable techniques for treemap layout  [45, 29]. Recommendations for improvements and best practice in the design of treemaps have been made, in terms of labelling, interactions [36], improvements to navigation [21] and the provision of overviews [15].

Other researchers have produced design guidelines for creating perceptually effective treemaps, based on extensive user studies evaluating performance on value estimation tasks (comparison of node size) [26]. The studies compared bar charts and treemaps, finding that treemaps become more effective than bar charts for size comparison tasks when the tree has thousands of leaf nodes. They recommended avoiding treemap layouts with extreme aspect ratios.

A few papers have compared treemaps and variants with Windows Explorer style visualisations [25, 42].

We are aware of only three previous studies comparing the effectiveness of treemaps, sunburst charts, and icicle plots [39, 4, 18].

One paper from 2000 [39] compared treemaps and sunburst charts. It found a strong user preference for the sunburst chart and some indication that it had performance benefits for hierarchy understanding. However, little interaction was provided and the displays were low-resolution (800$\times$800).

Another study from 2001 compared treemaps, sunburst charts, icicle plot and node-link diagrams [4]. Only hierarchy structure understanding tasks were considered. The treemap was slower and less accurate and the icicle plot and node-link were preferred. However, the implementations provided limited functionality for navigation, were limited by computational power and screen resolution (1024$\times$768) and used a small dataset ($<$200 nodes) in the evaluation.

More recently, a study compared the usability of various visualisations including treemaps, icicle plots and sunburst charts for search and exploration of large digital collections [18]. A JavaScript InfoVis Toolkit [1] providing a common platform for the visualisations used. This was limited in terms of navigation and provided no overview of the structure. The results indicated that icicle plots and tree designs were more effective for visualising trees with greater depths. It was also noted that the dataset being visualised caused computational issues “compromising the performance of the platform.” Tasks relating to the identification and comparison of the quantitative attributes of the underlying data were not performed.

Interactive navigation of larger hierarchical datasets has been considered by a number of researchers. Most commonly, users can collapse or expand nodes [31] and/or zoom to lower levels of the hierarchy using zoom-and-replace, where the view associated with the selected node replaces that of its parent [7, 9, 12]. We believe this behaviour and the lack of overview can lead to issues with navigation. Flame Graph [20] uses zoom-in-place in which the selected node expands to fill the viewport but does not provide an overview. SyncTRACE [24] also uses zoom-in-place but provides a two-level overview. Information Slices uses linked cascaded views at increasing levels of detail to provide contextual information [3] while two-level views with different spatial arrangements have been suggested in  [40]. However, none of these designs have been used in user studies comparing different visual representations for quantitative hierarchical data.

Given the importance of hierarchical quantitative data and the fact that icicle plots and sunburst charts are increasingly used in practical applications, it is timely to conduct another comparison with treemaps, the most widely used visualisation for hierarchical quantitative data, using larger datasets and providing more comprehensive support for navigation in such larger datasets.

3 Visualisations

In order to fairly compare the different visualisations, we implemented them using a consistent design framework. This framework was designed to overcome the previously identified difficulties of navigation within larger datasets.

Sundown Chart: as part of the study we also introduced a hybrid of the icicle plot and sunburst charts that we called the sundown chart, see Figure 1(b). The sundown chart is a semi-circular design, with the root node placed at the centre of the design, and child nodes fanning out from the root node. The design is a rotated version of a single Information Slice [3] and is similar to a “fan chart” that is often used to represent genealogical hierarchies [13], with the additional encoding of the quantitative variable in a similar manner to the sunburst chart.

The sundown chart design provides the benefits of both the icicle plot and the sunburst chart. Similarly to the icicle plot, the aspect ratio is better suited to the 16:9 aspect ratio of a computer monitor, when compared to the 1:1 aspect ratio of the sunburst chart. Like the sunburst chart, the radial design provides proportionally more space for the smaller nodes at lower level of the hierarchies. But compared to the sunburst chart labels are orientated more consistently.

Navigation: we chose to provide an Overview + Detail view of the hierarchy. For each visualisation, a compact overview of the hierarchy was shown above the main detailed view. For the treemap, both height and width were reduced in the overview while for the other visualisations only the non-quantitative dimension.

Like previous quantitative hierarchical data visualisations, we allowed the user to navigate the hierarchy using zoom and pan and drill-down from parent to child node and roll up to return. Zoom and pan is either from the detail view, or by using the “handles” shown in the overview to manipulate the position of the detail view. Inspired by Flame Graph [20] and SyncTRACE [24] we chose to use zoom-in-place which increases the zoom in the same dimensions as the node dimensions as this preserves more contextual information. For example, zoom actions in icicle plots increase the zoom on the x-axis only, increasing the width of the node. For sunburst charts, where the node size is proportional to the angle of the arc segments, the angle is proportionally increased. For treemaps, zoom takes place in both the x and y axes, proportional to the area of the node.

In order to support zoom-in-place, we chose to avoid aggregation or elision of the nodes in either of the views, and instead showed all nodes. This was inspired by “Interactive information visualization of a million items” [16] and “Unit Visualisations” [14]. This allowed us to smoothly animate zoom-in-place transitions and ensured layout stability, as all of the nodes are already present.

An advantage of our zoom-in-place design is that, in the icicle plot, sundown and sunburst charts, higher levels of the hierarchy are not removed from the visualisation when the user selects a lower-level node. This results in a stack of “breadcrumbs” remaining visible as the user zooms to lower level nodes, showing the path between the root node and the currently selected node. In order to level the playing field we also provided an explicit breadcrumb trail for the treemap design.

We used a dynamic “label what you can” approach. Labels are centred on the visible portion of the nodes, so that during interaction labels remain visible even as the underlying node is panned outside of the current viewpoint. Where partial nodes are shown, the label indicated the “off-screen” portion of the node with chevrons. Tooltip display of labels was also provided.

The user was provided with the ability to tag nodes as this was required for the study. Tagged nodes appear in the overview as “tag-dots” and as node outlines in the detail view.

For all designs, we included a scale to assist in size comparison tasks and to clearly show the magnification factor in the detail view.

Other Design Choices: following InterRing [44], we used a structure-based colouring strategy to colour nodes along a “cubehelix” spectrum [19] in all visualisations. Hue discriminates between different branches and emphasizes the hierarchical structure. Other colouring strategies remain to be explored, e.g. [41].

We used a squarified treemap layout, based on the guidelines produces by Kong et al. [26]. To further emphasize hierarchical nesting we used relative colour saturation, with deeper nodes appearing more saturated (similar to the colours implemented by Rosenbaum and Hamann [32]).

Implementation: initially, the visualisations were implemented using the popular D3 JavaScript library. However we found that the visualisations created using D3 were limited to about 800-1000 elements, beyond which the animations become unresponsive.

We then explored WebGL. This utilises the client machine’s GPU (Graphical Processing Unit), and is able to render significantly more elements than D3 can while supporting responsive animation and interaction. The final designs were implemented using a hybrid WebGL and D3 model, in which the nodes and visual elements were implemented using a WebGL layer, with labels superimposed using D3. Using this hybrid model, we were able to produce highly dynamic and smooth visualisations that do not need to aggregate or elide nodes with visualisations containing in excess of 50,000 nodes, with rendering rates at 50-60 frames per second.

4 Evaluation

We evaluated our visualisations first in a pilot study then a larger controlled study.

4.1 Pilot Study

A pilot study was conducted with six participants. Five were male, one was female. All were post-graduate data science students with some experience in data visualisation. Three participants were in the age range 18-25, two in the age range 25-35 and one over the age of 45 years. They were shown each of the four visualisations and asked general questions relating to the aesthetics of the visualisation, then to perform a range of tasks that require interacting with the visualisations. The study took approximately 90 minutes.

Feedback from the participants was used to improve the visualisation designs and to refine the tasks used in the full user study.

We used the CHI Browse-Off categories hierarchy dataset [30]. This has 15 levels, 7,548 nodes of which 6,278 are leaf nodes.

All visualisations apart from the sunburst chart received positive feedback. That for the sunburst was mixed. While the initial impression of the radial design were positive, participants found that the visualisation was more difficult to interpret. The relatively large amount of white space required by the full 360° design with a 1:1 aspect ratio resulted in smaller sized nodes overall, and subsequently less labeling of nodes. Additionally, techniques used to zoom-in-place lead to “disappearing” nodes along one radial axis (i.e. for zoomed views, partial nodes are shown on both sides of the 12:00 position) that was perceptually more difficult to understand than partial nodes displayed at the “edges” of the design seen in the sundown chart and icicle plot designs. Based on this negative feedback, the sunburst chart was excluded from the controlled user study to allow more time for the evaluation of the other designs.

4.2 Controlled User Study

We then conducted a larger controlled user study to compare the effectiveness of the three visualisations in terms of speed, errors and user preference. The visualisations, including an overview, scales and other design elements are shown in Figure 1.

Participants: Twelve participants were recruited through a closed-group on social media for Monash information technology students and staff. None had taken part in the pilot study. While most of the participants had seen treemaps previously, only two had seen icicle plots. No participants had previously seen sundown charts. Eight participants were in the age range 18-25, one in the age range 25-35 and three over the age of 45 years. Eight were male, four female.

Tasks: Four tasks were evaluated. These checked basic navigation (Q1), understanding of quantitative node attributes (Q2) and hierarchy structure (Q3, Q4):

• •

Q1 - “Navigate to a specific node” This task required the participant to navigate along a explicitly given path of the hierarchy to reach a leaf node, and then “tag” the node.

• •

Q2 - “Compare the size of two nodes” This required the participant to tag the larger of two pre-tagged nodes. The size difference between the two nodes was between 10 and 40%.

• •

Q3 - “Identify the closest relationship between Node A and Node B” Participants were asked to choose the relationship that most precisely described that between two tagged nodes: sibling, parent, child, ancestor.

• •

Q4 - “What is the least common ancestor of Node A and Node B” Participants were asked to tag the least common ancestor of two tagged nodes.

Procedure: The user study was in the following process:

• •

Participants were welcomed, introduced to the study, and had the evaluations process explained.

• •

For each visualisation, training was provided and then twelve trial tasks were presented to the participant, three items for each of the four task types.

• •

Following the completion of the evaluations tasks for all visualisation, the user was asked to answer a post-study questionnaire: 1) How would you rank the visualisations in terms of aesthetics, i.e. visually pleasing, engaging? 2) How would you rank the visualisations in order of usability, e.g. ability to navigate and interact with the visualisation? 3) Which visualisation do you prefer overall?

Presentation order of visualisations was counterbalanced. Training was delivered to participants using an in-application interactive tutorial. The training presented the participant with background information on quantitative hierarchical data, a description and example of the structure of each visualisation, as well as instructions on how to use the available interactions. Sample visualisations were used to confirm the participant’s understanding of the structure and attributes of each visualisation. Finally, practice questions were provided to the participant, along with feedback relating to the accuracy of the answer.

We used a dataset based on the CHI Browse-Off dataset, where each leaf node was assigned a random quantitative value. In the original dataset, all leaf nodes had the same value and we wished to increase the complexity of the data. 3 sets of 12 questions were randomly generated and hand-curated to ensure similar level of difficulty. Presentation of these question sets were counterbalanced between the three visualisations.

All users were tested using the same computer set-up, consisting of a Mac computer with a 26-inch monitor (a resolution of 2560$\times$1440), and a standard mouse equipped with a scroll wheel.

The study took about 1 hour to complete.

Measures: The time taken to complete each task (from initial rendering to answer selection), accuracy (binary–correct/incorrect), and mouse-interactions were logged by the application.

4.3 Results and Discussion

Answers were not normally distributed. Significance was tested with the Friedman test because we have more than two conditions and a post-hoc test was used to compare pairwise. Response times were log-normal distributed (checked with histograms and Q—Q plots), so a log-transform was used before statistical analysis [22]. We chose linear mixed modeling to check for significance [5] and applied Tukey’s HSD post-hoc tests to conduct pairwise comparisons [17, p. 575-582]. A within-subject design with random intercepts were used for all models. For user preferences we again used the Friedman test and post-hoc test to test for significance.

The Friedman test revealed no significant effect of visualisations on accuracy for Q1, Q2 and Q4, but a significant effect for Q3 ($\chi^{2}(2)=6,p=.0498$). Figure 2 (a-Q3) shows icicle plot (97.2%) and sundown chart (97.2%) were more accurate than treemap (91.4%). Post-hoc tests showed for both $p=.0855$.

The linear mixed modeling analysis showed significant effects of visualisations on time for Q1, Q3 and Q4.

Q1: $\chi^{2}(2)=17.47,p=.0002$. Icicle plot (avg. 32.6s) and sundown chart (avg. 35.2s) were significant faster than treemap (avg. 53.0s). All $p<.05$.

Q3: $\chi^{2}(2)=23.95,p<.0001$. Icicle plot (avg. 20.0s) and sundown chart (avg. 20.6s) were significant faster than treemap (avg. 44.2s). All $p<.0001$.

Q4: $\chi^{2}(2)=11.18,p=.0037$. Icicle plot (avg. 20.3s) was significant faster than treemap (avg. 33.0s) with $p=.0005$. While sundown chart (avg. 25.8s) tended to be faster than treemap, see Figure 2 (b-Q4), it was not statistically significant.

The three visualisations had similar performance for Q2. We had expected that efficiency and accuracy would be greatest for the icicle plots and least for the treemap based on previous work [11]. It suggested that size comparisons for one-dimensions (line length) are easier than angular comparisons, which are in turn easier than comparisons in two-dimensions (area). The lack of difference may because of that all three visualisations provided a scale.

User preference and feedback: For aesthetics, see Figure 3(a), the Friedman test revealed a significant effect of visualisations on preference ($\chi^{2}(2)=11.54,p=.0031$). Participants liked the visual design of icicle plot and sundown chart significantly more than treemap. All $p<.05$.

For usability, see Figure 3(b), the Friedman test revealed a significant effect of visualisations on preference ($\chi^{2}(2)=7.17,p=.0278$). Participants preferred the usability of icicle plot significantly more than treemap with $p=.0218$. Icicle plot was tended to be more preferred than sundown chart, however this was not significant.

Overall, see Figure 3(c), the Friedman test revealed a significant effect of visualisations on preference ($\chi^{2}(2)=9.5,p=.0087$). Participants preferred the icicle plot overall, significantly more than treemap with $p=.0218$. Icicle plot was also tended to be preferred to the sundown chart, however this was not statistically significant.

The final section of the study allowed participants to give feedback on the pros and cons of each visualisation as well as the overall design. Qualitative analysis of these comments reveal (overall):

Icicle plot was found to be intuitive and easy to learn. Some participants also mentioned it uses the screen space more effectively than a sundown chart but less than a treemap.

Sundown chart was commented as novel. However, we observed almost all participants tended to occasionally rotate their heads and some participants reported feeling slight motion sickness.

Treemap was found difficult to start with, but “it got better”. Some participants found breadcrumbs useful. Many participants had difficulties navigating the hierarchy.

5 Conclusion

We have compared three common visualisations for hierarchical quantitative data, treemaps, icicle plots and sunburst charts as well as a semicircular variant of sunburst charts we call the sundown chart. In a pilot study we found that the sunburst chart was least preferred. In a controlled study with 12 participants we compared the remaining three visualisations. Overall treemap was clearly the worst, leading to slower performance on a basic navigation task and slower performance and accuracy in hierarchy understanding tasks. It was also the least preferred by users. The icicle plot and sundown chart had similar performance with a slight user preference for the icicle plot. Interestingly we found no evidence of the conflict suggested by Schulz [34] between designs that effectively represent hierarchies and those conveying the quantitative node attributes: icicle plots and sundown charts appear to work well for both tasks.

A limitation of the current study is the use of randomised quantitative data in the user studies which may be unrepresentative of real-world data sets. Additional studies are required to confirm the findings for other datasets including different shaped hierarchies.

Acknowledgements.

This work was supported by the Australian Research Council Discovery Project grant DP140100077.