How Sensemaking Tools Influence Display Space Usage

TL;DR

This study investigates how a sensemaking tool with linked concept-graphs affects users' display space usage and organization strategies during intelligence analysis tasks.

Contribution

It introduces a bidirectionally linked concept-graph tool and demonstrates its impact on spatial organization and display space efficiency.

Findings

Users use less display space with BLC

Fewer open windows when using BLC

Spatial organization shifts to BLC nodes

Abstract

We explore how the availability of a sensemaking tool influences users' knowledge externalization strategies. On a large display, users were asked to solve an intelligence analysis task with or without a bidirectionally linked concept-graph (BLC) to organize insights into concepts (nodes) and relations (edges). In BLC, both nodes and edges maintain "deep links" to the exact source phrases and sections in associated documents. In our control condition, we were able to reproduce previously described spatial organization behaviors using document windows on the large display. When using BLC, however, we found that analysts apply spatial organization to BLC nodes instead, use significantly less display space and have significantly fewer open windows.

Figures

Full text

1 Introduction and Related Work

Extracting facts out of collections of documents (information foraging) and synthesizing information (sensemaking) are tasks widely encountered by knowledge workers. Professional analysts employ document analysis software (“sensemaking tools”), such as the Sandbox for Analysis [WSP*∗*06] or Jigsaw [SGL07]. This type of software often supports the exploration of documents through external cognition, which combines internal and external representations to perform cognitive tasks [SR96]. Externalization of one’s internal knowledge reduces the internal memory load and supports cognition by being able to directly perceive the information [Kir95]. Mind maps, concept maps, and similar visual representations of knowledge are commonly used externalization strategies, as is the spatial organization of work artifacts [Mal83, Kid94]. Indeed, Goyal et al. [GLF13] showed that users’ sensemaking performance improved significantly when provided with a visualization of shared entities across documents compared to when only provided with a note-taking tool.

Text analytics tools, such as Jigsaw [SGL07] or nSpace’s Sandbox [WSP*∗*06], categorize text entities and present them as graphs, scatter plots, or fairly free-form “shoeboxes”, in which the connections between entities are made explicit by color coding and visual links. Mind mapping tools (e.g., VUE [SK05]) enable users not only to add text nodes, but also images and document links to the mind map for externalizing their mental concepts.

As an alternative approach to complex sensemaking tools, large displays provide “space to think” [AEN10]. In absence of other tools, information foraging and sensemaking is facilitated through spatial organization of information in documents and relationship extraction from multiple documents [AN12].

Multiple researchers have found improved performance in analysis tasks when using large displays compared to small displays [CSR*∗*03, RJPL15], and documented increased subjective satisfaction [AEN10, BB09]. Large display users employ sophisticated strategies to exploit the available space for spatial cognition, such as dividing the space into focus and context areas [Gru01, BB09], placing application windows as reminders [HS04], as well as clustering or piling windows [AEN10, WGSS11]. Large displays thereby act as externalized memory, as users employ the space to organize and memorize information [AEN10]. In a collaborative environment [IF09], a large display can support the spatial arrangement and mutual awareness of opened documents.

Bradel et al. [BEK*∗*13] investigated collaborative sensemaking on a large display using either Jigsaw [SGL07] or a simple document viewer with highlighting and annotation. They observed that users had fewer documents open with Jigsaw compared to the document viewer, but speculated that this difference was caused by the different window management behaviors of the two sensemaking tools. However, an alternative explanation could be that the users employed different externalization strategies in Jigsaw, so that the actual need for multiple document windows was reduced.

In this work, we investigate the influence of sensemaking tools on knowledge externalization strategies using a lightweight graph-based tool, the bidirectionally linked concept-graph (BLC). BLC supports sensemaking with arbitrary online information sources and allows users to externalize their knowledge through a graph. It combines features of CLIP [MT14] (attaching lists of document references to nodes and edges) and ScratchPad [Got07] (referencing specific websites or passages within), and adds window layout and visual cueing capabilities that make BLC attractive for use on large displays. In particular, we are interested in understanding users’ spatial organization strategies and whether and how users make use of the available large display space.

Results of our user study indicate that there are strong individual differences how analysts structure their knowledge, irrespective of the sensemaking environment. However, we show that display space usage does depend on the sensemaking environment. From our findings, we conclude that spatial organization is a popular knowledge externalization strategy, but can happen at different scales, i.e. analysts organize abstract nodes representing facts or concepts when they have the ability to do so, and resort to organizing windows and hence make use of available display space when they do not.

2 Study

We conducted a study comparing sensemaking strategies on large displays for an intelligence analysis task with or without BLC. BLC allows users to organize browser-based information sources as mental concepts and relationships between these concepts in an interactive node-link diagram (Figure 1). The nodes are laid out manually, which is meant to act as memory aid and as an external representation of the user’s internal knowledge [Kir95], which can help to make inferences [LS87].

Users can easily add document references to nodes and edges. References may refer to specific passages within these documents. This allows users to quickly go back to the exact piece of evidence they were previously investigating. When users reopen documents, windows are placed close to the referenced node. Hence, users can make effective use of the large display to organize their sources without considerable management overhead. When sources are open, visual links [GSL*∗*14] can connect corresponding keywords between the graph and the document content. All these features help to organize large data sets into an “abstracted space to think”, and expand it on demand.

The goal of our study was to find differences in knowledge externalization strategies between users of BLC and a condition similar to the “Analyst’s Workspace” [AN12] – a large-display sensemaking environment to support spatial organization. It has been shown [BEK*∗*13] that users of the powerful Jigsaw analytics software rarely employ manual spatial organization. Our study differs, as we use a light-weight general-purpose concept graph instead of Jigsaw, and as all documents are opened in new windows in both experimental conditions. Our hypothesis was that — despite having the same window management — users of the control condition would spatially organize document windows on the large display (as observed by Andrews et al. [AEN10]), while BLC users would externalize knowledge through spatial arrangements within the concept graph.

2.1 Study Design

In a between-subjects design, we assigned users to one of two conditions: In the control condition, CC, documents were shown in regular application windows with the ability to search within the corpus. In addition, visual links [GSL*∗*14] could be invoked to connect text fragments between all open document windows. Links can be triggered from a selected word, phrase, or by searching for a term. In contrast to Andrews et al. [AN12], we did not perform named entity extraction for visual links, to avoid introducing a confounding factor between the study conditions. We chose to use visual links, as they have been shown to improve performance when recognizing related items on the screen compared to simple highlighting [SWL*∗*11] — an aspect that is especially pronounced in large display setups. Users were provided with an empty Google Doc (an online text editor) to take notes.

In the BLC condition, users could record information by creating nodes and edges in the graph and by adding notes to them. Deep linking between the concept graph and the document windows allowed for quick switching between the graph and the source information. Visual links could also be used for connecting arbitrary selected text between documents, as in CC, but also to connect node labels in the BLC with the web documents. No separate text document for note-taking was provided.

BLC differs from CC in the following aspects: (1) it provides visual abstraction of the contained information, (2) it organizes the information in a graph structure, (3) it provides deep linking between the graph nodes and edges and their associated source information, and (4) it automatically places document windows according to the user’s graph layout.

We chose a between-subjects design, as this allowed us to use only one task, limit the length of the analysis session, and avoid learning effects. On the downside, between-subjects designs can distort the results due to individual variability. We will therefore not only report statistical significances, but also present the quantitative results visually, and provide qualitative results.

We used the task descriptions and data from the 2011 VAST MiniChallenge 3 [GCH*∗*11]. The data comprised around 4.500 text articles, of which 13 contained manually generated news regarding a terrorism threat in the fictitious Vastopolis area. The task was to identify any imminent terrorist threats in the Vastopolis metropolitan area and to provide detailed information on the threat.

2.2 Apparatus

The study was conducted on a PC with six 22” monitors (1920$\times$1080) in a $3\times 2$ arrangement. The user was sitting  70cm away from the central monitor. The display setup is  155cm wide; the displays covered a visual angle of  95*∘* (Figure 2). To search through the data, we provided users with Recoll (http://www.recoll.org/), a browser-based full-text search tool. Selecting a document in Recoll opened it in a new window with the same size as the Recoll window. At the beginning of the session, the Recoll window was placed in the middle of the upper central monitor. For BLC, the empty BLC window was placed on the lower central monitor. For CC, an empty Google Doc was used instead. All 20 users (10 female; age 22-49) were knowledge workers (students, researchers, or administrators). Sixteen users had a computer science background; all users were familiar with sensemaking tasks, such as literature research. Some users reported to have experience with dedicated tools for sensemaking or information management, such as Evernote, Mendeley, OneNote, or Trello. After an introduction to the tools and a training session, users worked for an hour and were then asked to present their findings. After the session, they filled out a questionnaire, followed by a semi-structured interview.

2.3 Analysis

Sessions and interviews were recorded, and all BLC activities (concept or edge creation, adding or removing references), link activities (creation and deletion), window activities (opening, closing, moving, resizing), and query terms were logged. We compared questionnaire items, usage frequencies of the tools, display usage parameters (average/maximum number of open windows and display coverage, respectively) and the number of correctly identified plot elements in the analysis task either by Independent Samples t-tests or by Mann-Whitney U tests, if the assumption of normality was violated. We report significant differences, but do not mention explicitly when differences are not statistically significant. All interviews were transcribed and analyzed using open coding. Additionally, we qualitatively analyzed all concept graphs and Google Docs. We also analyzed all query terms and terms used for linking.

3 Results

To assess the task performance of the users, we counted the number of correctly identified hints. The ground truth consists of 13 short news documents. If a reference to one of these files was added to the concept graph or the Google Doc or content of one of these files was mentioned in the interview, we counted this as correct.

All users made an effort to follow leads and to extract a potential terrorist plot — albeit not necessarily the correct one. Only few users identified elements of the ground truth plot (three BLC users and four users of CC). The average number of opened documents out of the set of the 13 ground truth documents was low with 1.3 in BLC and 1.6 in CC. Three BLC users and four users of CC did not open any ground truth document at all. Users’ subjective satisfaction with the outcome was rated similarly for the two groups, with 3.4 in BLC and 3.0 in CC on average, on a 5-point Likert scale.

3.1 Usage of Sensemaking Tools

CC users conducted a significantly higher number of file queries (35.2/18.2). However, the number of opened files was similar in BLC (31.1) as in CC (29.3). The number of distinct files that were opened was almost equal (21.5/21.3).

Structuring approaches were diverse across all participants, but we could observe recurring strategies. BLC users created a more or less detailed concept graph, while CC users collected text snippets and notes in the provided Google Doc. Almost all users applied some groupings on their findings. While this is inherently supported by BLC, all but one CC users also logically grouped blocks of text in their documents (e.g., by source document or abstract concepts, such as “bioterrorism” or “airport”). Additionally, users tried to maintain links to the original files. BLC users had 6-33 file references in the graph. Similarly, all but two users in CC noted file names manually in the text files.

Many users mentioned in the interview that directly linking the source files to nodes or edges in BLC was helpful. They revisited on average 7.4 files through such references. BLC users rated “It was very easy to find the relevant passages in the key documents again.” significantly higher than the control group (4.5/3.7). However, the question “I had a very good overview of the documents I had already visited” was rated low by both groups (2.8/2.4). Some users criticized that the search tool Recoll did not visually mark files that have already been opened. BLC users explicitly noted that having numbers as file names made it hard to recall what the content of the particular file was. The node references therefore only showed up as numbers. With conventional web sources, BLC showed a favicon or, if not available, the first letter of the sources domain name.

3.2 Display Space Usage

We measured display space usage as the percentage of display space covered by application windows in one-minute intervals. The average display space covered by application windows over the entire task in the control group was significantly higher than in the BLC group (61% vs. 38%, see Figure 3). This difference was similarly pronounced for the maximum display usage with 71% control to 49% for the BLC group, which is also statistically significant.

Figure 2 shows heat maps of how the displays were used during the analysis session. To create these figures, the position and size of each window was aggregated for all users. Note how, in BLC, the windows are concentrated on the two center screens, while the peripheral four monitors were used only occasionally.

We found a considerable difference between the number of open application windows in the two conditions. The number of open application windows was significantly higher for CC than BLC (6.7/3.8 average); see Figure 3. The maximum number of open windows was also significantly higher for CC (10.3/5.9). Note, however, that the number of opened files (and therefore also the number of opened windows) was almost identical between those two groups (29.3 vs. 31.2) This implies that the difference was not caused by the number of visited documents, but by the way the document windows were managed. Most BLC users only kept windows open occasionally. For instance, one user explained that “I usually closed them right after usage to keep the space tidy.” CC users tended to keep documents with relevant content open for a longer time compared to BLC users, or never closed them.

We also interviewed users about display arrangement strategies. While only four BLC users mentioned a specific strategy how to arrange windows, seven users of CC were able to describe their window management strategies. We grouped interview responses into two different categories. The most popular strategy was to spatially group windows according to common concepts (described by two BLC and four CC users). Two BLC and three CC users partitioned their display into functional units, such as a main and peripheral area. However, these differences are only partially reflected in the questionnaire results. Overall, CC group rated the large display only slightly more pleasant to use (3.7 vs. 4.3), while, overall, the software provided was rated equally well by the two groups.

4 Discussion and Conclusions

We observed that users of both groups used a variety of strategies for organizing their findings and demonstrated comparable performance. However, users of CC had a significantly higher number of open document windows compared to BLC users. CC users therefore also utilized significantly more display space to spatially organize document windows. This implies that the outcome of the task was hardly influenced by the provided sensemaking tools, but the process was.

The spatial organization strategy systematically differed between the groups. Most CC users exhibited well-known spatial organization strategies with a large amount of document windows [BB09, AEN10, WGSS11]. Only few BLC users did that; most of them closed the text documents after reading. Instead, BLC users mainly used the graph to organize their findings. This means that both groups applied some sort of spatial organization, but it depended on the level of abstraction at which scale this spatial organization was performed. In fact, the concept graph can be considered a visual abstraction of the window layout, which itself is a visual externalization of the users’ mental model of the gathered information.

Thereby, we confirm a previous finding [BEK*∗*13]: Users have fewer open windows and use less space with a dedicated sensemaking tool. However, this is not exclusively caused by different window management strategies of the sensemaking environments, as suspected by Bradel et al. In our study, documents were all opened in a new window in both conditions, yet still, users handled them differently.

We believe that the ability to easily return to the relevant passage within the original document was a major factor why BLC users reduced the number of open document windows. CC users tried to maintain a link to their original data sources by adding the file names to their text document. However, finding the relevant passages required more interaction steps than clicking on a reference and following the link in BLC. On the other hand, no user commented on the automatic window layout offered by the BLC. We therefore think that this feature had a negligible effect on the users’ sensemaking strategies.

We infer that spatial organization is an important sensemaking strategy, but can happen at different scales. Meaningful visual abstractions can compress external knowledge representations. We hypothesize that the ability to easily return to the original information is a prerequisite for this compression. To formally evaluate this hypothesis, it will be necessary to compare sensemaking strategies with and without the ability to reference external documents in the future. Other important future investigations concern the scalability of the concept graph for longer activities, as well as its suitability for data with inherent chronological order or geographic reference. We expect that, due to its free-form spatial interface, BLC will also be appreciated for such kinds of tasks, but that more specialized sensemaking tools will lead to improved task performance.

Like Bradel et al. [BEK*∗*13], we were not able to show any measurable performance benefits (such as improved task completion time or correctness) of BLC. We speculate that this is due to the complexity of the sensemaking task, which only very few users could solve. However, our results show that users adapt their sensemaking strategies to the available environment. This is of high relevance given the heterogeneous “device ecologies” users are facing.

We conclude that modern desktop managers should provide efficient means to visually abstract arbitrary information and restore it on demand. Such as feature may decrease the need to perform extensive window switching [HS04], or to expand the display space to increase the amount of visible information.

Acknowledgements

We thank Marc Streit and Markus Steinberger for helpful discussions. This work was supported in part by the Austrian Science Fund (FWF T 752-N30), and the US National Institutes of Health (U01 CA198935).