Investigating Direct Manipulation of Graphical Encodings as a Method for User Interaction

TL;DR

This paper explores how users perform direct manipulation of graphical encodings in visualizations, providing empirical insights and guidelines for designing intuitive interaction methods.

Contribution

It presents a qualitative study identifying strategies for direct manipulation of graphical encodings and offers design implications for visualization tools.

Findings

Identified common strategies for direct manipulation operations

Observed cross-operation similarities in manipulation strategies

Provided guidelines for designing interaction techniques

Abstract

We investigate direct manipulation of graphical encodings as a method for interacting with visualizations. There is an increasing interest in developing visualization tools that enable users to perform operations by directly manipulating graphical encodings rather than external widgets such as checkboxes and sliders. Designers of such tools must decide which direct manipulation operations should be supported, and identify how each operation can be invoked. However, we lack empirical guidelines for how people convey their intended operations using direct manipulation of graphical encodings. We address this issue by conducting a qualitative study that examines how participants perform 15 operations using direct manipulation of standard graphical encodings. From this study, we 1) identify a list of strategies people employ to perform each operation, 2) observe commonalities in strategies across operations, and 3) derive implications to help designers leverage direct manipulation of graphical encoding as a method for user interaction.

Tables

Table 1. Table 1: The 15 basic operations that have been used in previous work for direct manipulation of graphical encodings in 2D scatterplot, bar chart and histogram. We use all 15 operations in our study. The last column ( Phrasing ) contains the exact sentence participants were told in our study. All the operations started with “ How would you interact with this system to show that you are interested in: ”

Encoding	Visualization	Operation	Phrasing
Position	Scatterplot	${𝐎}_{\mathrm{𝟏}}$ Assign a data attribute to an axis [37, 18] ${𝐎}_{\mathrm{𝟐}}$ Switch from a scatterplot to a bar chart [37] ${𝐎}_{\mathrm{𝟑}}$ Navigate the values of a point over time [20] ${𝐎}_{\mathrm{𝟒}}$ Adjust the value of a point [3]	Assigning the horsepower attribute to the x-axis Switching from a scatterplot to a bar chart Checking if this specific car has ever had the price of 20,000? Adjusting the value of this specific car to 30,000
	Bar chart	${𝐎}_{\mathrm{𝟓}}$ Group the bars into one bar [39] ${𝐎}_{\mathrm{𝟔}}$ Sort the bar chart [37, 24]	Merging the bars representing SUV and Wagon cars Sorting the bar chart in an ascending order
Size	Scatterplot	${𝐎}_{\mathrm{𝟕}}$ Change the size of all points [37, 21] ${𝐎}_{\mathrm{𝟖}}$ Assign a data attribute to the size of points [37]	Change the size of all data points, so that they are all equally bigger Assigning city mile per gallon attribute to the size of points
Color	Scatterplot	${𝐎}_{\mathrm{𝟗}}$ Change the color of all points  [37] ${𝐎}_{\mathrm{𝟏𝟎}}$ Assign a data attribute to the color of all points [37]	Changing the color of all points to red Assigning the cylinder attribute to the color of all points.
	Bar chart	${𝐎}_{\mathrm{𝟏𝟏}}$ Change the color of all bars  [37] ${𝐎}_{\mathrm{𝟏𝟐}}$ Assign a data attribute to the color of all bars [37]	Changing the color of all bars to red Assigning the car type attribute to the color encoding
Height	Bar chart	${𝐎}_{\mathrm{𝟏𝟑}}$ Navigate the values of a point over time [20] ${𝐎}_{\mathrm{𝟏𝟒}}$ Adjust value of a bar [3]	Checking if the number of sedan cars have ever had the value of 35? Adjusting the number of the SUV cars to 15
Width	Histogram	${𝐎}_{\mathrm{𝟏𝟓}}$ Expand the range of a bin in a histogram [39]	Expanding the range of this specific bin from 2009 to 2010?

Full text

\CCScatlist\CCScat

K.6.1Management of Computing and Information SystemsProject and People ManagementLife Cycle; \CCScatK.7.mThe Computing ProfessionMiscellaneousEthics

\vgtcinsertpkg

Introduction

The visualization community has advocated for the need to design more natural and fluid interactions for data visualization tools [12]. A recent line of research investigates how to enable users to convey their intended operations by direct manipulation [41] of the graphical encodings used to represent the data (e.g., [37, 7, 20, 31, 29, 21, 33, 35, 47, 10, 18, 13, 43, 42, 3]). For example, DimpVis [20] allows users to directly interact with the length, angle and position of the visual representations, as a means for temporal navigation. With DimpVis, users can drag a bar in a bar chart vertically to adjust its height. This makes it possible to explore the values for the bar over time, and to search for times at which the bar had a particular value. The appeal of direct manipulation can be attributed to multiple factors. First, direct manipulation interfaces benefit users’ experience by not requiring people to shift their attention from the visual features of interest when interacting [20, 28]. Second, they simplify the interface by obviating the need for additional control panels or widgets and opens the screen real estate [4, 22, 20].

To help designers create effective tools that implement this form of interaction, there is a need for empirical evidence to guide their design decisions. What type of direct manipulation strategies do people perform to convey their interest in performing visualization operations? Are there strategies that are consistently employed to perform a given operation? Are there some that conflict across operations? Addressing these questions through empirical studies will lead to design principles and guidelines to support further development of visualizations.

We take an initial step to address these questions by conducting a qualitative study in which 10 participants each performed 15 operations on three standard visualizations (scatterplot, bar chart, and histogram). By using a think-aloud protocol and video analysis, we obtain rich qualitative data from which we extract a list of strategies people employ to perform each operation. We then identify strategies that have consensus, and strategies that are conflicting with each other. Our analysis of the results further sheds light on four high-level categories of strategies (exemplification, declaration, instrumentation and selection) from which we derive implications to help designers leverage direct manipulation of graphical encoding.

This work contributes the following to the area of interaction design in visualization: (1) a qualitative characterization of user-defined direct manipulation strategies for performing different operations on visualizations, (2) insight into users’ mental models when using direct manipulation of graphical encodings as a method for user interaction, and (3) a set of actionable implications for designing interactive data visualization tools. Our results will help designers leverage direct manipulation of graphical encodings as a method for user interaction.

1 Background

We first discuss the concepts of direct manipulation [41] and instrumental interaction [4], two key concepts on which research about direct manipulation in visualization is strongly relying. We then discuss the work related to direct manipulation of graphical encodings.

1.1 Direct Manipulation and Instrumental Interaction

Direct manipulation enables users to directly act on the visual objects of interest [41] and provides immediate visual feedback in response to physical actions (e.g., dragging a folder on the desktop using the mouse). Direct manipulation actions are simple and support continuous flow of interaction. Instrumental interaction [4] generalizes the principles of direct manipulation by introducing instruments, that act as mediators between users and objects of interest. For example, sliders or check boxes can be used as instruments for filtering data points. Researchers who have applied instrumental interaction principles to visualizations [22, 20, 28, 12]; highlighted the need to minimize the spatial indirection (the distance between the interaction source and the target object), and the temporal indirection (the delay between invoking an operation and observing the result of that operation) [28].

1.2 Direct Manipulation of Graphical Encodings

Building on the concepts of direct manipulation and instrumental interaction, embedded interactions [38] are interactions in which users directly manipulate visual marks (e.g., bars in a bar chart) in visual representations rather than widgets and menus to perform a task. The goal of embedded interaction is to tighten the gap between one’s intent and the execution of that intent – thus reduce what Hutchins et al. call the gulf of execution [17]. It is propitious to congruent interactions [28]: interactions whose action, reaction, and feedback are cognitively congruent to the intent of the user. These approaches can make interactions more discoverable, resulting in easier to learn and use applications [28].

There is a growing interest in the infovis and visual analytics communities to develop this form of interaction for visualizations. A large set of visualization authoring tools enable graphic designers to construct customized visualizations (e.g., [23, 26, 19, 32]). These tools use direct manipulation as a method for creating customized visual glyphs. For instance, Data Illustrator [23] enables users to adjust the size of the circle by directly dragging its outline. There are also visualization tools that enable users to perform their analytical tasks using direct manipulation of graphical encodings, for example by letting users convey their interest in sorting a bar chart by dragging the tallest bar to the extreme left or right [37, 24, 34]. We found 16 papers that explicitly investigate direct manipulation of graphical encodings as a method to let users perform their analytical tasks [37, 7, 39, 20, 31, 11, 29, 21, 35, 34, 33, 47, 10, 18, 13, 43, 42, 3].

We organize this body of work according to which visualization type they support. We further structure our discussion of each visualization type according to the encodings that previous work has explored and the meanings that were assigned to their direct manipulation.

1.3 Visualization type: 2D Scatterplot

Direct manipulation of position [37, 20, 3], size [37, 21], and color [37] of data points in 2D scatterplots have been explored.

Position. Direct manipulation of position in scatterplots has been implemented through drag and drop of data points. This has been used to allow users to convey their interest in: navigating the temporal dimension of a dataset [20]; changing the axes of the scatterplot or to switch to a bar chart [37]; and adjusting the values of data points [3]. Researchers in visual analytics have explored direct manipulation of position in scatterplots that represent outputs of statistical models, for example to steer clustering models [10]; to steer distance and similarity functions by moving data points closer or further to each other [11, 43, 47, 13]; and to define and modify axes of scatterplots by dragging and dropping data points to either side of the x or y axes [18].

Size. Direct manipulation of size has been implemented through drag of the outline of a data point’s representation. Prior work enabled users to directly manipulate the size of data points shown in a scatterplot to demonstrate their interest in changing the size of data points [21, 37], and mapping a data attribute to the size of data points [37].

Color. The direct manipulation of color has been implemented through using a relatively indirect color picker in a contextual menu close to the visual mark. This has been used to change the color of all data points in a scatterplot, and to assign a data attribute to the color of points [37].

1.4 Visualization type: Bar Chart and Histogram

Direct manipulation of position [39, 37, 24], height [3, 20], and width [39, 7] of bars in bar charts and histograms have been explored.

Position. Direct manipulation of position in these visualizations has been implemented through drag and drop of bars. This has been used to execute abstract operations: to sort a bar chart by dragging the tallest/shortest bar to its extreme left or right side [37, 24]; and to merge two bars by dragging and dropping a bar on top of another one [39].

Height. Direct manipulation of height has been implemented through drag of the top border of a bar. Direct manipulation of the height of bars has been used as a method to convey an interest in navigating the temporal dimension of a dataset [20]; to adjust the values of data points in bar charts [3]; and to adjust the weight assigned to an attribute as a method for steering the underlying model used for computing the visual representation [18].

Width. Direct manipulation of bar width has been implemented through drag of the left/right borders of a bar to interactively merge and split the bins of a histogram [39].

Color. Direct manipulation of color has been implemented with a color picker in a contextual menu close to the visual mark, to change the color of all bars or to assign a data attribute the color of bars [37].

1.5 Other Visualization Types

Most research on direct manipulation of graphical encodings has focused on 2D scatterplots and bar charts / histograms, probably due to the simplicity and widespread use of these visualizations. However, some researchers have also enabled direct manipulation of graphical encodings in other visualization types. This includes the direct manipulation of: position of cells in table visualizations to either steer the underlying ranking model [46] or explore rankings [31, 44]; angle of a pie chart segment to navigate the time dimension [20]; rows and columns in matrix visualizations [42, 30]; nodes in tree visualizations [45]; and position of tokens in unit based visualizations [16].

1.6 The Next Step: Collecting Empirical Data

Because direct manipulation of graphical encodings is a recent topic of research, existing work has explored sparse points in the design space by selecting operations to be invoked and decided which strategies should be used to invoke each operation. It is now appropriate to empirically study these designs. Table 1 summarizes the operations and associated strategies from previous work for 2D scatterplots, bar charts, and histograms. Although we focus on visualization operations and discard analytical ones, this list gives a starting point for gathering empirical data regarding how people accomplish such operations.

2 Preliminary Studies

We conducted two preliminary studies to determine which approach to use to empirically investigate direct manipulation of encodings.

2.1 Preliminary Study 1: Paper-Based Study

In our first pilot study, we asked three participants (2 male, 1 female) to verbally explain how they would perform a series of visualization tasks using direct manipulation of graphical encodings on paper prints of visualizations. Providing participants with paper-based visualizations makes it possible to remove constraints that come with any implemented system, thus give more freedom and expressivity to participants.

We printed a bar chart representing the Cars dataset [15]. We explained the concepts of marks, encodings, labels, and direct manipulation of graphical encodings to the participants. We then explained the visualization and the data. We gave each participant three operations to perform in a random order (e.g., How would you show that you are interested in sorting the bar chart in an ascending order?). We asked participants to verbalize how they would perform each operation only using direct manipulation of the encodings used in the visualization.

Overall, participants found it challenging to explain their strategies without being able to actually perform the operations. For example, one participant said: “Should I imagine that I can change the width of the bar? […] then what is the system response?”, and another “I have no idea what happens if I move this bar!” Moreover, participants often did not restrict themselves to direct manipulation of graphical encodings only. For example, to sort the bar chart, one participant said he would first drag the tallest bar to the right side of the bar chart and added: “Now I expect to see a drop-down menu that has the ‘sort’ option.” – thus combining direct manipulation of graphical encodings and WIMP.

Based on the results of this first study, we decided to provide an interactive tool that supports direct manipulation of different graphical encodings used in visual representations. This would ideally help us avoid participants’ confusion as to what interactions are available to them and how those interactions are implemented. In addition, we could encourage the participants to think of strategies that rely solely on direct manipulation of encodings, if no other interactions are available.

2.2 Preliminary Study 2: Partial Implementation

We conducted a second pilot study with four new participants (4 male) using a prototype with limited functionality. We developed a web-based interactive bar chart showing the Cars dataset [15] and supporting direct manipulation of graphical encodings used in bars (e.g., changing the height of a bar). We explained the visualization, data, and available interactions to the participants. Then we asked them to perform the same three operations as in the first pilot study using the system.

Participants were able to perform 8 out of 12 operations (4 participants $\times$ 3 operations) using direct manipulation of graphical encodings. Although participants found some operations challenging to invoke (e.g., assigning a new data attribute to the axis), this pilot study indicated that using an interactive prototype is an appropriate way of eliciting people’s strategies for performing operations using direct manipulation of graphical encodings.

3 Study Design

In this study, we investigate how people perform the visualization operations listed in Table 1 using direct manipulation of graphical encodings. We describe the visualization and encoding types we used, the operations from Table 1, the dataset we used in the study and the software implementation. Then we describe our participants and settings, study procedure, and data collection method and analysis.

3.1 Visualization and Encoding Types

Given its qualitative and observational nature, we included a small number of (three) visualization types in the study to keep it within reasonable time. We selected scatterplot, bar chart and histogram because: i) they are among the most commonly used visualizations [14]; and ii) direct manipulation of graphical encodings has been well researched with these visualizations, as shown in Table 1. We studied the five encodings that have been explored in previous work for these visualizations: position and color for scatterplot and bar chart; size for scatterplot; height for bar chart; and width for histogram.

3.2 Operations

We included in our study all 15 operations, $\mathbf{O_{1}}$ to $\mathbf{O_{15}}$, listed in Table 1. We made this choice to compare the strategies implemented in previous work for each operation with the strategies our participants would perform. We also made the choice to focus on operations for visualization construction (i.e., that are designed to specify the visualization), and excluded operations that are designed to steer underlying models used for computing visualizations (e.g., InterAxis [18]).

3.3 Datasets

Most operations listed in Table 1 are generic enough to not depend on the dataset being used. For example, assigning a data attribute to an axis can be performed on any scatterplot regardless of the underlying tabular dataset. Operations that involve navigating temporally, however, require a dataset that contains at least one temporal dimension. Based on this constraint, we selected the widely used cars [15] and movies [1] datasets, which have also been used in related studies (e.g., [37, 18, 20]).

3.4 Interactive Tool and Interactions

We developed an interactive visualization tool using JavaScript and the D3 library [5]. An Upload (\faUpload) button supports uploading the pre-defined visualizations we used in our study. A Reset button (\faRefresh) supports resetting the visualization. We added interactivity to the encodings used in visualizations. For instance, participants could change the position, color, width, and height of the bars in a bar chart. Or, they could change the size, position, and color of circles in a scatterplot.

The tool shows one visualization at a time. Participants can directly manipulate the encodings in the visualization. For example, after the interviewer has uploaded a bar chart and asked a participant to perform an operation, the participant can use any of the provided interactions to convey their intention to perform this operation. The tool only enables participants to manipulate the encodings and does not recompute the visualization (similar to a drawing interface like Adobe Illustrator). Enabling the participants to convey their actions without the system reacting to those actions makes it possible to observe participants’ unrevised behavior, and drive system design to accommodate it.

To add interactivity to the five graphical encodings (position, height, width, size, color) investigated in this study, we kept our implementation of interactions as close as possible to previous work (Figure 1 illustrates these interactions for the bar chart and the scatterplot):

Position. We afforded the repositioning of circular data points in a scatterplot or bars in a bar chart or histogram anywhere on the screen through drag and drop. 2. 2.

Height and Width. Clicking on a bar in a bar chart or histogram makes four small handles (black circles) appear on the four sides of the bar. Dragging the left and right handles changes the width of the bar; dragging the top and bottom handles changes its height. 3. 3.

Size. Clicking on a circular data point in a scatterplot makes a small handle (black circle) appear on the perimeter of the circle. Dragging the handle changes the size (radius) of the circle. 4. 4.

Color. Right clicking on a mark (circular data point in scatterplot, bar in bar chart or histogram) makes a color picker appear. Picking a color from the color picker changes the color of the mark.

3.5 Participants and Settings

We recruited 10 non-color blind participants (4 females, 6 males), aged 20–30 (mean $24.8$) via email and word of mouth at our university. They were students enrolled in computer science (5), physics (1), psychology (2), and mechanical engineering (2). All reported being familiar with reading visualizations, and eight had created visualizations before. Four took the information visualization course taught at our university. Some participants had experience with visualization tools such as Microsoft Excel (8), D3.js (4) and Processing (1). None of them had participated in the pilot studies. Participants sat approximately 30–40 cm from a 13” LCD display with a resolution of 2560 $\times$ 1600 pixels equipped with a mouse and keyboard. The visualizations were shown in full screen.

Given the qualitative nature of the study, we determined participant numbers based on empirical saturation [8] – which can be reached with as low as 6 participants [27]. Two authors of this paper watched the screen-recorded videos after each session to get a sense of the strategies the participant used to perform the operations. In addition, during each session the interviewer took notes of high-level strategies that the participant employed to perform each operation. As we progressed through our study, we discussed these notes, identifying whether the observed strategies were repeats or newly observed strategies. The sessions with participants 9 and 10 generated limited new strategies, suggesting that we had reached empirical saturation. We then discussed our informal findings as a group and decided to conclude the study.

3.6 Procedure

1. Introduction (~10 min). Participants were briefed about the purpose of the study and their rights. After filling out the study consent form and a questionnaire on demographics and visualization expertise, they watched a three-minute video explaining the concepts of marks, encodings, labels, and axes. Participants could replay the video as they liked. Then, they were given a sample of the movies dataset printed on a sheet of paper. After the experimenter had explained to them the meanings of rows and columns, they were asked to familiarize themselves with the data for two minutes and ask any question they might have.

2. Training (~10 min). In this phase, participants were shown a scatterplot, a bar chart, and a histogram all created with the movies dataset (one visualization at a time, in a random order). We explained how the system supports manipulation of different encodings used in each visualization. For instance, we showed participants that they can drag the left or right boundary of a bar to manipulate its width. We then asked participants to perform an operation on the visualization. Depending on the visualization, we asked participants to: assign a data attribute to an axis of a scatterplot; sort a bar chart in a descending order; and expand the range of a bin in a histogram. For example, the interviewer asked the participants: “How would you interact with this system to show that you are interested in sorting the bar chart in a descending order?” The interviewer’s role in this phase was not to guide the participant, but solely to answer their questions. Such questions included: “Am I supposed to show how I am going to do this task by manipulating these glyphs”? “Should I assume that the system is going to detect my interaction?” The interviewer did not suggest participants strategies for performing the given operations, nor gave examples or hints on how to perform an operation.

3. Main study (~30 min). Participants familiarized themselves with the cars dataset like they had with the movies dataset. Then, they were shown a scatterplot, a bar chart, and a histogram created with the cars dataset, one at a time with the order randomized. For each visualization, they were asked to perform all operations associated with the visualization (see Table 1), one at a time in a random order. In total, each participant performed 15 operations ( 8 with the scatterplot + 6 with the bar chart + 1 with the histogram). For example, the interviewer asked the participants: “How would you interact with this system to show that you are interested in changing the color of all points to red?” Participants were asked to perform each operation by only manipulating the graphical encodings in the visualization on the screen. Participants could also verbally explain how they would perform the operation when they could not perform it with the supported interactions. They could also suggest more than one strategy for performing each operation.

4. Wrap-up (~5 min). The experimenter thanked the participants who received a $10 gift card. Participants were invited to ask additional questions about the study.

3.7 Data collection and Method of Analysis

We screen- and audio-recorded the study. During the main study, the experimenter took notes of participants’ actions. We analyzed the 298 minutes of screen-capture videos using open coding [9] in three phases.

In the familiarization phase, two coders reviewed the videos together to identify common and unexpected patterns that informed our decision to focus the analysis on strategies. For example, they observed that participants came up with many more unique ways of switching the visualization from a scatterplot to a bar chart, than for other operations.

In the coding phase, once the two coders had a good understanding of the data, they agreed that strategies for performing operations were an appropriate lens through which to analyze the data and defined the ‘intended strategies’ as the unit of analysis. This meant that two types of information had to be coded for: which operation the participant is performing; and which intended strategy the participant either employs or suggests. An intended strategy is the expression of an intention that a participant performed physically and/or explained verbally to complete an operation. This definition allowed us to capture intended strategies that participants demonstrated using the prototype as well as those that they verbally explained, e.g., “In that case, I would select the entire screen [all data points]. Then once they are in the same group of selection, I am assuming that increasing the size of one object should increase the size of all.” To ensure a reliable coding process, both coders coded parts of a video together until they fully agreed on the coding. Then one coder went through all videos and coded both operations and intended strategies. The second coder then checked the coding for two randomly selected videos and confirmed the coding. In total 203 intended strategies were coded across all 15 operations.

In the analysis phase, both coders watched the video snippets of the 203 intended strategies to derive archetypal strategies. For each instance of intended strategy, they discussed whether it fell under any of the existing archetypal strategies or if it formed a new one. For every new archetypal strategy, the coders sketched a storyboard of it on paper. This process created 48 mutually exclusive archetypal strategies. The two coders further named each archetypal strategy based on its sketch. Figure 3 and Figure 3 summarize these 48 archetypal strategies (that we call ‘strategies’ in the remainder of the paper).

We provide online 111https://encodingstudy.github.io/ all relevant materials for this study: datasets, software for running the experiment, and sketches of the strategies employed by the participants.

4 Results

We present the strategies participants employed in our study to perform the operations using direct manipulation of graphical encodings. All but one participants could identify at least one strategy for each operation, with one participant failing to identify a strategy for one operation only. Participants suggested 4–5 strategies per operation on average. Most strategies are based on direct manipulation of the graphical encodings using well-known interaction patterns such as repositioning, resizing (that includes resizing the radius of points, the width of bars and the height of bars), and recoloring. A few strategies, however, rely on direct manipulation of other visualization elements such as axes, labels, and tick marks (participants verbally explained these strategies).

We identified 48 unique strategies across the 15 operations, shown in Figure 3 (the exact phrasing used in our study for each operation i provided in Table 1). Below, for each operation we present the commonalities, variety, and unexpectedness of the strategies that participants employed. We provide raw sketches drawn during the video analysis procedure in supplemental materials. For simplicity, in this section we use “points” to refer both to the data cases in the dataset and to the visual marks representing these data points in the scatterplot (circles in our study); we use “bars” to refer to the visual marks representing data points or aggregations of data points in the bar chart and the histogram.

Assign a data attribute to an axis ($\bm{O_{1}}$), to the size of points ($\bm{O_{8}}$), and to the color of points ($\bm{O_{10}}$) or bars ($\bm{O_{12}}$). We grouped together these operations that are about assigning a data attribute to an encoding. Figure 7 shows the main strategies used for invoking these operations.

We identified four strategies that participants employed to assign a data attribute to a scatterplot axis ($\mathbf{O_{1}}$). Seven participants used Reposition a few Points, where they rearranged a few points to sort them based on their values. This strategy has already been supported in previous work [37]. Three participants used Reposition & Resize, where they first rearranged a few points to sort them horizontally and then resized them based on their values. Here they manipulated the size of points to express their intent to sort the points from lower to higher values. For example, after sorting and resizing the points, P2 said: “So the size represents the horsepower, but at the same time the position is also horsepower. By the size [changing the size] I am trying to reinforce that the higher value should be on the right.”

One participant used Reposition inline & Recolor, where he repositioned a few data points to sort them horizontally and then colored the points based on their values to inform the system about his thought process. He stated: “I am coloring them just to show the system they have been sorted.” Another participant also first repositioned a few data points to sort them based on the range that they fall in and then colored them (Reposition by range & recolor).

To assign a data attribute to the size of points in the scatterplot ($\mathbf{O_{8}}$), seven participants used Resize by Values, where they resized a few points based on their values. For example, P2 made smaller three points with a low value; then he made the point with the highest value bigger than all the other. Other participants used similar strategies: P1 resized a single random point (Resize point), and P2 and P8 first resized a few points based on their values before coloring them (Resize & Recolor). One participant also said that he would first select all the points and then resize one of the selected points (Select & resize).

Most participants employed Recolor by Values to assign a data attribute to color, both with the scatterplot ($\mathbf{O_{10}}$, 8 participants) and with the bar chart ($\mathbf{O_{12}}$, 10 participants). This strategy consists of coloring a few points/bars with different values using different colors. P5 suggested Recolor & Enwidth with the bar chart, where he first colored two bars then increased the width of one of the bars to cover all the remaining bars. He explained: “if I want to imply changes to be made to all bars, I drag the width of the bar.”

Switch from a scatterplot to a bar chart ($\bm{O_{2}}$). Figure 7 shows the four most used strategies to invoke this operation out of the eight we identified. Five participants used Reposition Points Vertically, where they rearranged a few points to position them vertically without any overlap. Two used Recolor one Group, where they colored a few points that fall in a specific range using the same color.

Three used Resize & Reposition to Axis, where they resized a few points then moved them to the tick marks shown on the axis. They explained that they consider each resized point to be a bar where the value is mapped to the radius of the circle instead of the height of the bar, e.g., “this looks like the bar chart the bars are circular” (P5) and “So basically each circle is a bar” (P2). Two participants used Stack Points Vertically, where they rearranged a few points to position them vertically where they have some overlaps (i.e., stacking the points vertically similar to visualization by demonstration [37]).

Navigate a data point over time ($\bm{O_{3}}$ & $\bm{O_{13}}$). Figure 7 shows the main strategies used to check whether a data point has ever had a target value. With the scatterplot ($\mathbf{O_{3}}$), five participants used Reposition to Target, where they moved the point to the target value like in DimpVis [20]; five used Resize Point, where they resized the data point; and four used Reposition to Tick Mark, where they moved the point and dropped it on the tick mark representing the target value.

With the bar chart ($\mathbf{O_{13}}$), nine used Reheight to Target, where they changed the height of the bar to the target value like in DimpVis [20]; one used Recolor & Reheight, where she first changed the height of the bar to the target value and then colored the bar; and four used Retop to Target, where they moved the bar vertically so that the top of the bar aligns with the target value.

Adjust the value of a point ($\bm{O_{4}}$) or a bar ($\bm{O_{14}}$). Figure 7 shows the most frequent strategies for these operations. Participants used four different strategies to adjust the value of a point in the scatterplot to a target value ($\mathbf{O_{4}}$). Three used Reposition to Target, where they moved the point to the target value, like in previous work [3]. Six participants extended this strategy by adding another step to it: four further colored the point (Reposition & Recolor) and two other participants resized it (Reposition & Resize).

To adjust the value of a bar in the bar chart ($\mathbf{O_{14}}$), nine participants changed the height of the bar to the target value (Reheight to Target), like in previous work [3]. Two used Retop to Target, where they moved the bar so that the top of the bar aligns with the target value.

Group two bars into one bar ($\bm{O_{5}}$). Figure 11 shows the three strategies (out of four) used by more than one participant. Nine used Overlay Bars, where they dragged one bar on the other one, like this has been proposed in previous work [39]. Participants found this strategy to be intuitive, e.g., “Is there an easier way to this question? There might be others but this seems like the most intuitive one” (P1). Two used Stack two Bars, where they stacked one bar on top of another, e.g., “after grouping two bars, values of the two bars should add up” (P3). Here we use “stack” to indicate the piling of visual marks on top of each other without overlap. Three participants used Recolor a few Points, where they colored the two bars using the same color.

Sort a bar chart ($\bm{O_{6}}$). Figure 11 shows three of the five strategies participants suggested to sort a bar chart in ascending order. Nine participants used Reposition Bars by Height, where they moved a few bars based on their heights. For example, P1 stated: “dragging two or three bars one after another should essentially indicate that I am rearranging them based on their values.” Three participants used Reposition Tallest & Shortest, where they dragged the tallest bar to one extreme of the visualization and the shortest bar to another extreme, like in previous work [37, 24]. Some of them felt that moving both tallest and shortest bars was not necessary. For instance, P3 said: “I pick sedan which is the biggest one and put it on here [extreme right]. This makes sense and then the system should be able to pick up what I was trying to do. And maybe if you want to make it more sure, you pick up the lowest one also put it here [extreme left].” P6 used Recolor by Values, where she colored three different bars with ordinal colors. She explained: “keep the first one white, then make the second one yellow. So I am assuming this is the increasing order” (P6).

Change the size of all points ($\bm{O_{7}}$) or the color of all points ($\bm{O_{9}}$) / bars ($\bm{O_{11}}$). Figure 11 shows the main strategies participants used when asked to execute these operations.

There were two popular strategies to change the size of all circles in the scatterplot ($\mathbf{O_{7}}$). Eight participants used Resize a few Points Equally, where they resized two to four random points to make them equally bigger. For example, P2 stated: “If I have to change the size of all of them [data points in scatterplot], then probably I should make a group of circles and they should not share the same axis, they should be very random.” Four suggested verbally Select & Resize, that consists of selecting all circles and resizing one of them.

There were four main strategies to change the color of all points in the scatterplot ($\mathbf{O_{9}}$) or bars in the barchart ($\mathbf{O_{11}}$). Many participants used Recolor a Few Points, where they recolored two to three points/bars using the same color (five did so with the scatterplot, seven with the bar chart). P1 and P6 used Recolor & Paint, where they colored one point/bar and dragged it over a few other points/bars. Similarly, P5 used Recolor & Dip, where he first colored one point/bar, then dragged a few other points/bars such that they overlaid the colored point/bar. Finally, six participants mentioned that if selection was available they would use Select & Recolor One, where they would first select all points/bars then color one of them.

Expand the range of a bin in a histogram ($\bm{O_{15}}$). Participants suggested five strategies to expand the range of a bin from one range to another range. Figure 11 shows three of these strategies.

Nine participants used Enwidth, where they increased the width of a bar to merge it with the bars next to it, as has been proposed in previous work [39]. P7 extended this strategy into Enwidth & Recolor, where she first increased the width of the bar and then colored the bar yellow to inform the system about the latest updates: “[…] I colored the bar to show that it has been updated” (P7). Three participants used Overlay bars, where they dragged and dropped the bins (bars) falling in the expected target range (e.g., the bars between 2008-2011) on top of each other such that they overlaid. P2 used Recolor a few points, where he colored in the same color the bins that fell in the expected target range. P3 used Enheight & Enwidth, where he first increased the height of the bar to exceed the $x$ axis then increased its width to merge the bar with the bars next to it.

5 Discussion

We structure our discussion based on Figure 3, that presents the 48 strategies (in columns) participants used to perform the 15 operations $\mathbf{O_{1}}$ to $\mathbf{O_{15}}$ (in rows). We first discuss the varying degrees of agreement for strategies as well as conflicting strategies (strategies that participants used to perform different operations). Then, we propose four high-level approaches for organizing the 48 strategies: exemplification, declaration, instrumentation and selection. Last, we discuss limitations of our study and future research directions.

5.1 Varying Degrees of Agreement

Some strategies were widely adopted by participants for a given operation, i.e. they have a relatively high degree of agreement. For two thirds of the operations (10/15), one strategy represents more than 50% of the distribution of all used strategies. For example, among the 12 strategies that were proposed to assign a data attribute to the color of all bars, 10 employed Recolor by Values (see $\mathbf{O_{12}}$, strategy 23). As another example, among the 15 strategies participants proposed to expand the range of a bin in a histogram, 9 employed Enwidth (see $\mathbf{O_{15}}$, strategy 41). On the other hand, some operations were performed using a wide range of strategies, with no consensus emerging. For example, to switch from a scatterplot to a bar chart (O2), participants used 8 different strategies. This suggests that it might be harder to identify common strategies for certain operations.

Findings from our empirical study show that users of visualizations can effectively employ direct manipulation of graphical encoding as a means of performing operations of varying complexity. However, the degree of agreement regarding which strategy to use varies from operation to operation. Going forward, visualization designers might consider incorporating strategies in consensus for performing operations using direct manipulation of graphical encodings. As such, our study provides a framework for collecting more empirical data and contributes to building a set of operations and corresponding consensual strategies.

5.2 Strategies in Conflict

Participants sometime employed the same strategy to perform different operations. For example, participants colored a few visual marks using the same color (Recolor A Few Points) to perform a variety of operations including: to group two bars in one bar ($\mathbf{O_{5}}$, strategy 24); to change the color of all points in both the scatterplot ($\mathbf{O_{9}}$, strategy 24) and the bar chart ($\mathbf{O_{11}}$, strategy 24); and to expand the range of a bin in a histogram ($\mathbf{O_{15}}$, strategy 24).

The fact that participants used the same strategy to perform different operations, in a relatively open environment, is revealing in several ways. First, it suggests that the space of strategies for performing operations using direct manipulation of encodings is not as vast as it might look. Second, it indicates that there is often no single strategy that will be unanimously used to perform an operation. Third, it shows that some strategies can be consensual for performing different operations. That is the case for Reheight to Target, for example, which was used 9 times both to navigate the point over time in a bar chart ($\mathbf{O_{13}}$, strategy 40) and to adjust the value of a bar ($\mathbf{O_{14}}$, strategy 40).

This many to many relationship between strategies and operations raises technical challenges in leveraging direct manipulation of graphical encoding for user interaction. Visualization designers should in the first stage avoid implementing support for strategies that different people would use for performing different operations. Future work should explore recommending to the user all possible operations in response to an employed strategy so that the user can select the most appropriate operation (similar to VisExemplar [37]). Not only would this enable designers to support multiple operations, this could also be used to collect data on people’s preferences towards developing an understanding of contextual strategies. In other words, collecting such data would enable us to analyze in which situations a given strategy is intended to trigger a particular operation.

5.3 Higher level Categorization of Strategies

By comparing the relations between strategies and operations, we identified four high-level approaches that participants used to manipulate encodings to invoke an operation (see Figure 3). Below we discuss these four high-level approaches: exemplification, declaration, instrumentation and selection.

Exemplification. With this approach, participants directly manipulated a small number of visual marks in order to illustrate by example the output they were trying to achieve. Participants widely used exemplification to invoke operations (e.g., see Rows 1-6 in Figure 3). Exemplification was mostly achieved through a repetitive set of actions to show the system how a part of the visual output should look like. For instance, one of the strategies that participants used to sort the bar chart is Reposition Bars by Height (strategy 12). With this strategy, they positioned a few bars one by one in an ascending order similar to how the bar chart should look like after sorting. In another example, one of the strategies that participants used to change the color of all data points is Recolor a Few Points (strategy 24). With this strategy, participants colored only a subset of points using the same color to demonstrate their higher-level goal of changing the color of all data points. The idea behind exemplification is similar to the visualization by demonstration paradigm [37]. In visualization by demonstration, one of the methods to provide visual demonstrations is to directly manipulate the graphical encodings used in the visualization to indicate a part of the expected visual output to the system.

Declaration. With this approach, participants manipulated a graphical encoding different from the primary graphical encoding used in the operation. For example, when we asked to assign a data attribute to an axis of a scatterplot ($\mathbf{O_{10}}$), we expected participants to mainly rely on positioning data points manually based on their values for that data attribute, since switching the axis results in changes in how points are positioned. However some participants manipulated encodings that were not directly linked to assigning a data attribute to the axis. One participant used Reposition Inline and Recolor (strategy 29), where they first rearranged a few points vertically/horizontally, then colored them. While repositioning a few points is directly related to mapping a data attribute to position, coloring the points is not directly related to creating such a mapping. In this case, participants used color as a way to communicate their intention to the system. Another participant used Enwidth & Recolor to expand the range of a bin in a histogram ($\mathbf{O_{15}}$). While color is not directly related to how ranges in histograms are represented, the participant colored the bar as a way to inform the system about the latest changes they had made. Another participant used Recolor by Values to sort the bar chart. Here, while the primary graphical encodings related to the sort operation are the height and the position of bars, the participant expressed the notion of order using an ordinal color scheme.

Instrumentation. With this approach, participants used a visual mark as an instrument (or tool). This resulted in relatively advanced and unexpected strategies. For example, two participants used Recolor and Paint to change the color of all points ($\mathbf{O_{9}}$) and bars ($\mathbf{O_{11}}$). They first colored a point/bar and then used this colored visual mark as a brush: they dragged it over other points/bars to color them. Similarly, two other participants used Recolor and Dip to perform these two same operations ($\mathbf{O_{9}}$ and $\mathbf{O_{11}}$), but the other way around. They first colored a point/bar and then used this colored visual mark as a bucket: they dragged a few other points/bars such that they overlaid the colored point/bar. To turn a visual mark into an instrument, participants used the graphical encodings color (strategies 34 and 35) and size (strategies 18, 38 and 46). The idea behind instrumentation is similar to what has been proposed with constructible interfaces [25], a paradigm that is strongly grounded into instrumental interaction [4]. In constructible interfaces, one of the methods to create or modify visual marks is to turn other visual marks into instruments that can be used to perform operations [25].

Selection. With this approach, participants expressed their interest in a selection technique (verbally because selection was not supported in the prototype). They explained that they would prefer having access to a selection option that they could use to select a subset of points. Then they would apply an operation to a single data point rather than having to apply the same operation to multiple points. For example, participants used Select & Recolor One to change the color of all points in both the scatterplot and the bar chart.

Differences in approaches in terms of visual marks. We can further differentiate these four approaches according to the number of visual marks they require one to manipulate. Participants sometimes thought of strategies that relied on direct manipulation of three or more graphical encodings. In such cases, participants thought of instrumentation and selection approaches because these approaches enabled them to perform repetitive actions rapidly (instead of manipulating different visual marks one by one). For example, when asked to change the size of all data points, some participants mentioned that resizing every single point might be tedious. Thus, they suggested Select & Resize where they would select a subset of points and resize them all at once rather than resizing each point individually. On the other hand, participants used instrumentation and selection less often when their strategies required them to change one or two visual marks. It suggests that the number of marks to directly manipulate to perform an operation informs which high-level approach to use.

5.4 Types of Encodings Used for Performing an Operation

We initially hypothesized that participants would manipulate multiple encodings while employing their strategies. For example, we thought participants would manipulate a variety of encodings such as color, size, and height of the bars while applying their strategies to sort a bar chart. However, our results show that a majority of strategies often rely on manipulating a single type of graphical encoding. For instance, for 13 out of the 15 strategies employed to sort a bar chart, participants only manipulated the position of the bars. Similarly, 13 out of the 14 strategies employed to map a data attribute to the size encoding, participants only manipulated the size of the data points.

This indicates that users of visualizations can effectively employ direct manipulation of graphical encoding as a means of performing operations, and that the mechanisms people might use to manipulate graphical encodings is not as complex and wide as one could expect. These findings can also help tool designers decide whether to implement direct manipulation of only one, or of multiple encodings in a tool according to the operations the tool must support.

5.5 Limitations

This qualitative study is the first attempt to understand how people manipulate graphical encodings to invoke different visualization operations. As such, it cannot answer all open questions related to this problem; here we discuss the limitations of our study and findings.

Prototype Functionality. Our findings must be interpreted in the context of the study prototype. To investigate how people directly manipulate graphical encodings to perform operations, we had to first design a tool that supports such functionality. Many participants explained strategies that were not supported by the prototype. This shows that participants thought about strategies beyond the implemented interactions, as we had hypothesized. However, it is likely that the limited functionality of our study software has impacted their strategies. Building on our findings, future studies should consider including some of the functionality suggested by our participants, such as the ability to select data points. This will allow participants to express more diverse strategies and broaden the palette of possible ways of performing visualization operations through direct manipulation of graphical encodings.

Operation Description. Although we kept the description of the operations as close as possible to previous work, the phrasing of the questions can influence the strategies employed by participants. Moreover, the nature of the operations differ from one another. For example, some operations are more direct in how they refer to the changes that need to be made on the visual marks (i.e., the questions are congruent [6] to the task to achieve). Despite this limitation, we observed a multiplicity of strategies for all operations.

5.6 Generalizability and Future Work

Our qualitative study provides rich and detailed results; however this study does not aim at generalization. Testing our research questions using other visualization techniques and operations might reveal nuances of direct manipulation of graphical encodings that were not tested in this study. One of the next steps will be to build on our findings to study how people perform and perceive a large set of operations and strategies via a crowdsourced study in order to gather more quantitative data and aim for more generalizable results.

In this study, we investigated how people perform different operations using direct manipulation of encodings. We aim to extend our study by investigating how people understand this method for user interaction. For example, we plan to conduct a study where we show videos of user interactions from this study to Amazon Mechanical Turk workers and ask them to list potential operations they think the person in the video is trying to perform. We expect that after collecting such data, we will be able to design machine learning models that predict people’s intended operations based on the strategies they employ.

The prototype used in our study only enabled participants to manipulate the visual marks and their graphical encodings – it did not react to participants’ actions (similar to a drawing interface like Adobe Illustrator). Enabling participants to convey their actions without the system reacting to those actions enabled us to observe participants’ unrevised behavior in isolation. However, a setup in which the system would react to participants’ actions for performing an operation might influence strategies employed by the participants. Specifically, if the system responds to each individual action then accurate predictions of operations may shorten strategies employed by the participants. We envision conducting a Wizard of Oz experiment in which we will provide different types of feedback as participants manipulate the encodings to perform an operation. This will help us understand how the reaction of the system influences the strategies employed by participants.

Today, visualization tools incorporate multimodal interactions to enhance user experience and system usability [2]. For instance, earlier work on natural language interfaces for visualization indicated potential value in combining direct manipulation and natural language as complementary interaction techniques [40]. Exploring how direct manipulation of graphical encodings combines and complement other input modalities such as natural language, body posture, and gestural interaction is an exciting avenue for future research.

Interactivity in many visualizations has been, and is, supported through WIMP widgets. These tools are successful in easing the processes of visualization construction, because they allow users to interactively construct visualizations instead of using programming and are fully discoverable. However, when other forms of interaction such as direct manipulation of graphical encodings are leveraged, it raises a number of important questions [36] including: How effective are different forms of interaction (e.g., WIMP vs. direct manipulation) for specific operations? How can we design tools that effectively leverage several forms of interaction? Understanding the differences and trade-offs between various forms of interactions and how they are used for specific operations will help designers and developers make informed interaction design decisions when creating visualization tools.

6 Conclusions

Designing visualizations that leverage the direct manipulation of graphical encodings as a means for user interaction is challenging because we lack a holistic understanding of which actions should be supported and what are their associated operations. To date, such systems rely on designer-generated heuristics to create these mappings. This paper provides the first list of strategies, sometimes consensual and sometimes conflicting, that people employ to perform operations using direct manipulation of graphical encodings. We organized the strategies into four high-level approaches: exemplification, declaration, instrumentation, and selection. This work provides a framework for collecting more empirical data and contribute towards developing an holistic understanding of the strategies people employ to perform certain operations.

Acknowledgements.

We are grateful to the study participants, reviewers, and members of the Georgia Tech Visualization Lab for their helpful insights and comments. This project is supported by NSF IIS-1750474 and the Natural Sciences and Engineering Research Council of Canada (NSERC).