Inline Co-Evolution between Users and Information Presentation for Data Exploration

TL;DR

This paper introduces a dynamic 3D virtual museum interface for data exploration that adapts in real-time to user interests, facilitating personalized discovery of complex database relationships.

Contribution

It proposes a novel co-evolution model of user interests and data presentation within a virtual museum environment for enhanced data exploration.

Findings

Real-time user interest evaluation improves data navigation.

The virtual museum model enables personalized and intuitive data discovery.

Application to cultural heritage demonstrates practical effectiveness.

Abstract

This paper presents an intelligent user interface model dedicated to the exploration of complex databases. This model is implemented on a 3D metaphor : a virtual museum. In this metaphor, the database elements are embodied as museum objects. The objects are grouped in rooms according to their semantic properties and relationships and the rooms organization forms the museum. Rooms organization is not predefi-ned but defined incrementally by taking into account not only the relationships between objects, but also the users centers of interest. The latter are evaluated in real-time through user interactions within the virtual museum. This interface allows for a personal reading and favors the discovery of unsuspec-ted links between data. In this paper, we present our model's formalization as well as its application to the context of cultural heritage.

Figures

Full text

\Arabicsection Introduction

This paper is a part of a project aiming at designing tools favoring the sensemaking of a user from his interactions with numerous information stored in a database through a virtual environment. Sensemaking is a concept that comes from the organizational theory [14] and inspired by the enactive field of cognitive science [13]. Enaction considers that mental representations of a living being emerge from his interactions with its environment. Moreover, since the living being acts on the environment, it evolves and then, representations result from a circular and dynamical process.

In the case where the living being is a human person and the environment is populated with data embodied with graphical objects (for instance, in a virtual environment), sensemaking involves an exploration with coming and going through the environment rather than a research of some precise data. The discovery of links between data during his exploration constitutes progressively a sense for a user.

However, there are different kinds of links between data as well as different users that will adopt different points of view on these data. In many cases, the number of potential links between data is very important. It is therefore necessary to present to the user only data linked together according to the user’s concern. There are as many representations of data organization as there are users. Moreover, these representations can modify user’s point of view during his exploration. Consequently, the choice of data presented to the user should be constructed in line as well as the organization of this presentation. To achieve this goal, the tool that allows the exploration of data should evaluate the user interests in real time. It should be able to choose new data to present to the user at any time. The result is a co-evolution between the user point of view and the data presented to him.

From these considerations, we propose a tool that constructs a virtual environment which presents data progressively, according to the real-time evaluation of user’s interests. In this paper, it is applied to cultural heritage objects. However, the model is generic and we will precise in which conditions it is possible to apply it to any domains.

This paper is divided into height parts: the first presents the related works, the second describes our approach, the third introduces our data, the fourth to the sixth present the dynamic evolution, the seventh highlights the taking into account of the memory maintenance in our system and the data organization in the museum, and the last parts present our future work and the conclusion.

\Arabicsection Related works

We identified two research domains related to our work: firstly, researches on data visualization aiming at enhancing user’s understanding, and secondly, researches on user’s models dedicated to adaptive human-machine interfaces.

\Arabicsection.1 Data visualization

Visualization is a key element to help a user to analyze data. It allows a global view of the data, organized according to criteria relative to a model elaborated from a point of view. One way to present data is to use a hierarchical structure. For instance, [4] propose to use a tree constituted by nodes and edges. The nodes represent data categories while the edges represent the hierarchical relationships between the categories. The display depends on the nodes list selected by the user during his navigation. He chooses some kinds of information before obtaining the corresponding representation. [16] propose a radial representation, for which a disc area represents the data set, and is divided into many sectors. Each sector is labeled by one description and one color which characterizes one category. The more a category is close the user’s interest, the more its corresponding sector is large. User’s interest is evaluated in terms of the selections and the zooms which are performed by the user.

For [5], data are represented by images and are positioned on a spiral. The image selected by the user is placed on the center of the spiral. There is a hierarchy between the data used to position child images on child spirals. The geometrical distance between the central image and each other images corresponds to a hierarchical distance between them. The more the distance of an image from the central image is large, the more its size is small. The user can select any image of interest with the computer mouse to bring a displacement of this image at the center of the screen. Then the position of the other images is re-organized as well as the different spirals.

All these approaches use quite abstract representations without a link with the daily environment of an individual. The appropriation of the system by the users is made delicate. For this reason, some approaches use metaphors reproducing a familiar environment, generally through visualization techniques in 3D. For instance, [1] use a virtual library where each book represents one data and each shelf represents one category. Wise et al. present the data on a topographic card [15]. Young [17] and Sparacino [12] use another metaphor: that of a city in which the data are represented by buildings, which are grouped into neighborhood. The relations between the data are represented by paths or bridges.

Other works as [2] focused on data categorization according to time, space and events, which allows the user to adapt the representation. Temporal information are presented on a timeline, spatial information on a card, and event information on a disc.

\Arabicsection.2 User adaptation

Many works in e-commerce and data exploration focus on finding ways to adapt information representation according to user’s center of interest. Almost of these approaches estimate this center of interest from informations consulted during navigation. [10] propose to classify users among three categories: expert, medium, beginner. Each category is associated with a level of information details to be presented to the user, as well as with a specific navigation mode. The system defines and changes the user category by analyzing the information that he consults. [8] present a web site dedicated to scheduling. It allows suggestions according to user’s center of interest. These centers of interest are determined from the sites that were previously visited. One similar approach is proposed by Cecilia di Sciascio in [3]. It consists to not only sort in real time the documents in terms of their relevance compared to the user’s centers of interest but also to suggest him some keywords. The user’s centers are estimated from the informations consulted during his navigation.

Lenzmann et al. [7] and [6] present an adaptive virtual interface using a virtual agent which accomplishes some tasks. According to the user’s satisfaction level which is evaluated from his gestural and his vocal commands, the agent improves progressively the quality of the service asked. Finally, other approaches as [9] propose adaptive interfaces based on machine learning and user feedback.

\Arabicsection Our approach

Our data are composed of the heritage objects pictures, text descriptions and keywords which are defined by linguists. We use a museum metaphor to implement the co-evolution principle. Each room presents pictures of some cultural heritage objects. Figure 1 shows the user’s view of the virtual museum. These objects may have numerous links together. For example, they have been discovered in the same place (topological links) or they were able to be created at the same time (chronological links). A lot of anthropological links are relative to their usage or signification during certain period of History. During a stroll through the museum, new rooms appear, according to the evaluation of user’s interest. This new rooms are populated with new objects. The user can guide the system as well as the system guides the user by proposing new rooms and new objects. Some objects are discovered by the user through links he is not aware of. The system is thus capable of improving user’s knowledge according to his focus and can introduce serendipity.

Three interactive tools allow the user to indicate which kind of information he wishes to focus on. These tools also give some feedback to the user about the system’s internal evaluation of the user’s interests. The Topos tool shows a colored map. When the user clicks on an area, it increases this area’s relevance within the system evaluation process. If one area is important to the user, its color turns green and over time progressively degrades to red. The Chronos tool allows to select a period of time and provides the same colored feedback as the Topos tools. The Thema tool is an interactive word-cloud. The user can click on words to indicate his interests. The words cloud is then self organized accordingly. A complete explanation of this self organization is provided in [11]. Semantic links between cultural objects and words and between words (together) exist and are provided through an ontology dedicated to anthropological reading of cultural heritage. As this ontology is developed by linguist partners of the project, it is a specific work in progress that we will not detail in this paper. For this particular work, we will only consider it as a graph which links concepts and keywords together.

\Arabicsection Data space

The underlying model of our proposition relies on the following elements:

• —

Objects:

«  Objects  » are the heritage objects from the database of our industrial partner. Each of them is associated with different elements as an image, a descriptive text, as well as dimensional entities (see below).

• —

Dimension:

The dimension are the time, the space, and the topic, and correspond to the categories which are manipulable by the Chronos, Topos, and Thema tools, presented above.

• —

Dimensional entities:

Each object is characterized by at least three dimensional entities which belong to the three dimensions: Chronos, Topos and Thema. The dimensional entities in the dimension Chronos define the period during which the object was created or used, Topos dimensional entities correspond to the place where the object was created or used, while dimensional entities of the dimension Thema is derived from the ontology and are concepts related to the object. This number of three dimensional entities by object is a minimum. For example, there may be several dimensional entities in the Chronos dimension which correspond to important dates for the object (destruction, renovation …).

• —

Distance:

Within each dimension, we have defined methods to calculate a distance between two dimensional entities. The temporal distance is defined by the smaller number of century which separates the two dimensional entities of the Chronos dimension. The geographical distance between two Topos dimensional entities is defined by the length (Territory count) of the shortest path that separate them. Finally, the semantic distance is determined using the ontology. A distance of 1 correspond to a direct semantic link between two concepts of the ontology.

Let:

• —

Ob={$ob_{i},i=1..N_{ob}$}: the set of heritage objects present in the environment where $N_{ob}$ is the size of this set

• —

D={$d_{i},i=1..N_{d}$}: the dimensions set where $N_{d}$ is the size of this set. In our case, we have three dimensions(Chronos, Topos, and Thema).

• —

$DE_{d_{j}}=\{de_{i},Dim(de_{i})=d_{j}\}$: the set of dimensional entities in the dimension $d_{j}$. $Dim(de)$ is the function which return the dimension of one dimensional entity $ed$.

All these definitions are not domain dependent and are available once the following conditions have been met: having a database associated with ontologies that are structured in term of dimensional entities. Each dimensional entity belong to a dimension.

\Arabicsection Virtual Museum Self-dynamic

\Arabicsection.1 Museum tree structure

The virtual museum is composed of a set of rooms which are dedicated to at least one topic. Topics are dimensional entities (from the three dimension set). The rooms are organized according to a tree structure. Each room have one «  parent  »room, from which the user has entered. Each room has three doors (closed in the beginning) leading to its «  children  » rooms, plus one open door leading to its «  parent  »room. These three closed doors will open dynamically upon dynamic creation of their respective child-rooms. Each room thus have at most three ”children” rooms. To avoid spatial collision of rooms on big museum structure, the rendering engine uses a non euclidean space mechanism that prevents rooms to be rendered simultaneously at the same location.

\Arabicsection.2 Estimation of user’s center of interest

The estimation of user’s center of interest is based on his interactions with the environment during his navigation. We defined 6 types of interactions:

to stand in front of an object 2. 2.

to consult text description of an object 3. 3.

to put an object in a virtual basket (this basket is used to keep within easy reach some objects that interest the user) 4. 4.

to remove an object from the virtual basket 5. 5.

to enter in a room 6. 6.

to use the tools

Let:

• —

$I=\{type_{I},DE_{I},W_{I}\}$ an interaction of $type_{I}$ with a weight $W_{I}$ on an object or tools concerned by the dimensional entities set $DE_{I}$

• —

$t_{I}$: the time when the interaction $I$ is realized

• —

$Trace=\{(I_{1},t_{I_{1}}),(I_{2},t_{t_{2}}),...(I_{n},t_{I_{n}})\}$ the temporal recording of the interactions that the user realizes during his exploration.

• —

$w(I,t)$: the weight associated with the interaction $I$ at time $t$ after it was performed. The temporal evolution of this weight is computed as follows:

$w(I,t)\leftarrow W_{I}*\exp(-\lambda*(t-t_{I}))$. With this formula, the value of this weight decreases in half after $\frac{ln(2)}{\lambda}$ seconds.

• —

$R(de)$ the real-time relevance of the dimensional entity $de$ for the user

Each second, the simulation is updated according to the following algorithm:

$\forall(I_{i},t_{I_{i}})\in Trace,$ and $\forall\ de\in DE_{I_{i}}$ :

$R(de)\leftarrow R(de)+w(I_{i},t_{I_{i}})*(R_{de\_max}-R(de))$

where $R_{de\_max}$ is a maximal relevance value defined for $de$.

Then, to favor serendipity and to help the user to find new objects, we decrease and propagate the relevance of dimensional entities to their neighbors (in their respective dimensions Topos, Chronos and Thema). For that, if the relevance of one dimensional entity is superior than a given threshold, this latter diffuses one part of its relevance to its direct neighbors. To avoid the cyclic diffusion, we prevent one dimensional entity to diffuse its relevance more than once for a certain time:

• —

Decreasing:

$\forall(I,t_{I})\in Trace,$ and $\forall de\in DE_{I},$

$R(de)=\tau*R_{de\_min}+(1-\tau)*R(de)$ where $\tau$ is the reduction rate and $P_{ed\_min}$ the minimal relevance of $de$.

• —

Propagation:

Let:

• —

$s_{d}$: the threshold of the relevance diffusion

• —

$\gamma$: the rate of the diffusion

• —

$de$: a dimensional entity

• —

$Ng(de)$: the set of neighbors of $de$. This neighboring is defined by a threshold distance.

If $R(de)>s_{d},then:$

$\forall de_{i}\in Ng(de),Ng(de_{i})=R(de_{i})+\frac{\gamma*R(de)}{|V(de)|}$

$R(de)\leftarrow(1-\gamma)*R(de)$

This formula is inspired by energy conservation. The relevance lost by one dimensional entity is equal to the sum of that it has diffused to its neighbors.

\Arabicsection The evolution of the museum structure

From the users’ center of interest, the system self-determines the moments at which a new room should be present to the user: 1) when one $de$ reaches a high pertinence threshold and it is not present in the adjoining rooms or 2) when the user approaches a closed door. When one room is created, we associated to it one topic which is formed by a set of one or two «  Dimensional entities   ».

\Arabicsection Objects self-organization

The objects presented in the room are determined semi-automatically. The system groups a set $G1$ of objects which correspond to the junction of the $de$ associated to the room. A second object group $G2$ is created and correspond to a set of objects having similar topic(distance 1 from the $de$ of room topic), and the group $G3$ formed by the $de$ to distance 2. The room content is formed by the selection of 12 objects belong them 40% of $G1$, 40% of $G2$, and 20% of $G3$. It allows the user discovering thematic related objects.

Once these objects are selected, their position in the room is self-organized by distance. All the objects are randomly positioned in the room and there is mass-spring system allowing to position the objects based on their similarity distances. The spring rest length between two objects is proportional to the sum of the distance between their $de$. Once the system stabilized, the position of the objects are normalized in order to enter all objects in the room while respecting the distance between them.

\Arabicsection Memory maintenance

In order to not disorient the user, no room are destructed during the simulation. This allows the user to have references when he wants to turn back onto his path. We note that we could have many rooms having the same themes (and objects). For this, the user can easily find back an object without the need to come back. Obviously this object will appears only if it belongs to the user’s center of interest (or if it is chosen by the algorithms randomly to favor serendipity).

\Arabicsection Conclusion and perspectives

We have presented an approach that allows for the co-evolution between a user knowledge on one domain and the presentation of the informations to him while preserving serendipity. The work presented in this paper illustrates this problem through a particular case: the reading of the heritage for the study of human societies. This reading is an intellectual construction that we have embodied through the mediation of virtual reality, through the adaptive creation of a virtual world, and according to a museum metaphor. The subjective character of this construction is took into account by the adoption of the enaction paradigm which is translated here by the sensori-motor coupling linking the user to the information system. This paradigm is implemented by a process which describe the relevance diffusion of heritage objects by taking into account both the conceptual description of heritage and the user’s intentions (inferred from user’s interaction within the virtual environment).

The implemented system leads to the creation of a virtual environment usable as interface between the user and a heritage database. This interface presents a 3D representation of the information contained in the database as well as the meta-information usable to guide the user in the environment.

We intend to continue this work in different directions: firstly, we will develop an evaluation methodology adapted to the developed approach. Several axis must be validated, such as the precision of users’ center of interest estimations, navigation, understanding of museum structure, and so on. Then, we will take into account of the user’s body. The goal is to give sense to the information according their position compared to the user’s body and rather than only using the 3D environment. Finally, we will apply our proposition in the context of social networks, publications of scientific papers,

Références

• [1]

Card, S., Robertson, G., and York, W.

The webbook and the web forager : an information workspace for the world-wide web.

Proceedings of the Conference on Human factors in Computing Systems (CHI’96) (1996).

• [2]

Cho, I., Dou, W., Wang, D. X., Sauda, E., and Ribarsky, W.

Vairoma: a visual analytics system for making sense of places, times, and events in roman history.

IEEE transactions on visualization and computer graphics 22, 1 (2016), 210–219.

• [3]

di Sciascio, C., Sabol, V., and Veas, E. E.

Rank as you go: User-driven exploration of search results.

In Proceedings of the 21st International Conference on Intelligent User Interfaces, ACM (2016), 118–129.

• [4]

Heer, J., and Card, S. K.

Doitrees revisited: scalable, space-constrained visualization of hierarchical data.

In Proceedings of the working conference on Advanced visual interfaces, ACM (2004), 421–424.

• [5]

Jankun-Kelly, T., and Ma, K.-L.

Moiregraphs: Radial focus+ context visualization and interaction for graphs with visual nodes.

In Information Visualization, 2003. INFOVIS 2003. IEEE Symposium On, IEEE (2003), 59–66.

• [6]

Lenzmann, B., and Wachsmuth, I.

A user-adaptive interface agency for interaction with a virtual environment.

In Adaption and Learning in Multi-Agent Systems, IJCAI’95 Workshop, Montréal, Canada, August 21, 1995, Proceedings (1995), 140–151.

• [7]

Lenzmann, B., Wachsmuth, I., and Cao, Y.

An intelligent interface for a virtual environment.

KI-NRW (Applications of Artificial Intelligence in North-Rine Westphalia) Report (1995), 95–01.

• [8]

Li, L., Yang, Z., Wang, B., and Kitsuregawa, M.

Dynamic adaptation strategies for long-term and short-term user profile to personalize search.

In Advances in Data and Web Management. Springer, 2007, 228–240.

• [9]

Mezhoudi, N.

User interface adaptation based on user feedback and machine learning.

In Proceedings of the companion publication of the 2013 international conference on Intelligent user interfaces companion, ACM (2013), 25–28.

• [10]

Paternò, F., and Mancini, C.

Effective levels of adaptation to different types of users in interactive museum systems.

Journal of the American Society for Information Science 51, 1 (2000), 5–13.

• [11]

Rajaonarivo, L., Maisel, E., and De Loor, P.

An enactive based realtime 3D self-organization system for the exploration of a cultural heritage data base.

In 20th International Conference Information Visualisation (Lisbonne, 2016), 100–105.

• [12]

Sparacino, F.

City of news.

ACM SIGGRAPH’99 Conference abstracts and applications (1999).

• [13]

Varela, F. J., Thompson, E., and Rosch, E.

The Embodied Mind.

MIT Press, 1993.

• [14]

Weick, K. E.

Sensemaking in Organizations.

Sage Publication, 1995.

• [15]

Wise, J.

The ecological approach to text visualization.

• [16]

Yang, J., Ward, M. O., and Rundensteiner, E. A.

Interring: An interactive tool for visually navigating and manipulating hierarchical structures.

In Information Visualization, 2002. INFOVIS 2002. IEEE Symposium On, IEEE (2002), 77–84.

• [17]

Young, P.

3d software visualization - visualization and representation.

internal Report, University of Durham, Visualization Research Group (1997).