A Model for Using Physiological Conditions for Proactive Tourist Recommendations

TL;DR

This paper presents a data model that leverages wearable sensor data to infer tourists' physiological states, enabling proactive and personalized activity recommendations based on health conditions.

Contribution

It introduces a novel data model linking wearable sensor parameters to physiological states for improved tourist activity recommendations.

Findings

Feasibility demonstrated with a self-quantification app

Relations established between sensor data and physiological conditions

Framework enables context-aware tourist recommendations

Abstract

Mobile proactive tourist recommender systems can support tourists by recommending the best choice depending on different contexts related to herself and the environment. In this paper, we propose to utilize wearable sensors to gather health information about a tourist and use them for recommending tourist activities. We discuss a range of wearable devices, sensors to infer physiological conditions of the users, and exemplify the feasibility using a popular self-quantification mobile app. Our main contribution then comprises a data model to derive relations between the parameters measured by the wearable sensors, such as heart rate, body temperature, blood pressure, and use them to infer the physiological condition of a user. This model can then be used to derive classes of tourist activities that determine which items should be recommended.

Equations

$$\begin{array}[]{l@{{}={}}c}\text{PC}&\left(\begin{array}[]{@{}cccccc@{}}c_{a},&c_{r},&c_{t},&c_{d},&c_{h},&c_{s}\\ \end{array}\right)\end{array}$$

$$\begin{array}[]{l@{{}={}}c}\text{W}&\left(\begin{array}[]{@{}cccccc@{}}w_{(or|a)}&w_{(ae|a)}&w_{(fd|a)}&w_{(rs|a)}&w_{(nl|a)}\\ w_{(or|r)}&w_{(ae|r)}&w_{(fd|r)}&w_{(rs|r)}&w_{(n|r)}\\ w_{(or|t)}&w_{(ae|t)}&w_{(fd|t)}&w_{(rs|t)}&w_{(nl|t)}\\ w_{(or|d)}&w_{(ae|d)}&w_{(fd|d)}&w_{(rs|d)}&w_{(nl|d)}\\ w_{(or|h)}&w_{(ae|h)}&w_{(fd|h)}&w_{(rs|h)}&w_{(nl|h)}\\ w_{(or|s)}&w_{(ae|s)}&w_{(fd|s)}&w_{(rs|s)}&w_{(n|s)}\\ \end{array}\right)\end{array}$$

$$\begin{array}[]{l@{{}={}}c}\text{ARI = PC . W}&\left(\begin{array}[]{@{}ccccc@{}}a_{or},&a_{ae},&a_{fd},&a_{rs},&a_{nl}\\ \end{array}\right)\end{array}$$

Full text

wearable devices, user modeling, proactive recommender systems, context modeling, machine learning

††copyright: none

1. Introduction

Suggesting points of interest or activities to tourists within a city is a popular and challenging recommender systems research problem. To provide interesting recommendation at the right time, context-awareness is very advantageous (Woerndl and Groh,, 2007). Currently, mobile phones are commonly used for eliciting its user’s context. However, in some situations tourists might be uncomfortable pulling out their smart phones due to the fear of them being stolen, or for social reasons as it might be inappropriate to check phones all the time when fellow-travellers are around. This gives rise to wearable devices, such as smart watches for context modeling. Wearable devices have embedded sensor technologies and can be used to collect various information about the wearer and the surrounding environment. Wallace and Press (Wallace and Press,, 2003) showed new perspectives of looking at wearable objects by involving computer technologies into them. However, wearable devices of the last ten years such as the Fitbit Zip111https://fitbit.com/au/Zip, Google Glass222https://developers.google.com/glass/develop/gdk/location-sensors, Apple Watch333https://apple.com/lae/watch/, and the Microsoft HoloLens444https://microsoft.com/en-IE/hololens had mixed success on the market and the smart phone is still the most prevalent mobile device.

As a tourist, it is useful to get personalized recommendations about where to visit, what to do next throughout a trip, given that the recommender has sufficient system information about the context of the tourist. Unlike classical recommender systems, mobile recommendations can also be proactive, meaning that the recommendations are pushed to the user without explicit request (Woerndl et al.,, 2011). This introduces the additional challenge to determine the appropriate timing of the recommendations. Wearable sensor devices help to gather various data about and around the wearer. This data can be used to acquire contextual information, and with further processing, the proper conditions can be derived to recommend tourist activities to the wearer. Our approach was thus guided by the following research questions:

1. RQ1:

How can different physiological conditions be inferred using wearable sensors? 2. RQ2:

How can these conditions be used to derive preferences for activities?

In the following section, we survey related work on tourist recommender systems based on context awareness, wearable devices, and sensor data. We present our data model in section 3, which shows how to select tourist activities based on data from typical sensors found in wearable devices. Finally, we draw the conclusion and present future work in section 4.

2. Foundations

We argue that to improve mobile, context-aware, and proactive recommender systems, sensors in wearable devices can be useful by inferring the physiological conditions of a user. If the sensor data can be interpreted accurately in the context modeling step, the overall recommendation accuracy of a mobile recommender system can be improved with recommendations that fit well to the context and condition of the user.

2.1. Context-Aware Recommender Systems

Throughout this paper, we follow Dey’s definition of context: “Context is any information that can be used to characterise the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves (Dey,, 2001).” It is generally believed that if the contextual information of a person, e.g., physical or emotional states, or her surroundings, e.g., location and time, is available, the accuracy of the recommendations increases. Adomavicius et al. (Adomavicius et al.,, 2011) presented a way how context can be used to develop intelligent recommender systems. Ashley-Dejo et al. (Ashley-Dejo et al.,, 2016) proposed a context-aware proactive recommender system for tourists and determined the accuracy of the recommendations comparing the benefits of using various different contextual information, such as location, weather, or time as opposed to not considering context at all. In their evaluation the traditional multi-criteria collaborative filtering approach, where no context information was used had the least performance, while the recommendations where all context information was used performed best.

For the recommendation of customized destinations and routes to travelers, the current location is one of the most prevalent context factors. Meehan et al. (Meehan et al.,, 2013) highlight the importance of considering more contextual information other than location, such as, weather, time, sentiment, and user preferences. But not all context factors are equally important as Baltrunas et al. (Baltrunas et al.,, 2012) found when they investigated the importance of a context factors for different activities. For example, TourRec, a recommender system for city trips (Laß et al.,, 2017) only uses time of the day, weather and previously visited points of interest. In the model proposed in our paper, we showcase the possibility of using physiological conditions for proactively recommending personalized tourist activities.

2.2. Wearable Devices

Wearable devices can support humans with the overload of information in communication and computation in appropriate contexts (Billinghurst and Starner,, 1999). A survey of commercially available wearables, research prototypes, and their classification was done by Seneviratne et al. (Seneviratne et al.,, 2017), and Godfrey et al. (Godfrey et al.,, 2018) provide an A-Z guide of the key terms required for understanding current wearable technology developments in health care.

Wearable devices are useful for measuring various physiological and biochemical parameters (Islam and Mukhopadhayay,, 2017), which can further be processed to benefit the healthcare domain. For example, a sensor-equipped ring has been used to provide physiological data of patients and further analyzing it with their profiles by using assistant systems showed that it can help to resolve wrong diagnoses by doctors (Shehab et al.,, 2017). Clothing or textiles are good collectors of body information as they stay in close contact with the skin (Stoppa and Chiolerio,, 2014). Smart textiles are also widely used to noninvasively gather health information (Coyle et al.,, 2010) and physical monitoring (Pandian et al.,, 2008). Kamisalic et al. (Kamišalić et al.,, 2018) surveyed how various physiological parameters and activities can be noninvasively measured by sensors incorporated into wrist-worn devices. For this reason these type of sensors are also useful to monitor athletes’ training and health (Düking et al.,, 2016).

Despite its potential, wearable devices have not yet become that predominant in the tourism sector. Tussyadiah (Tussyadiah,, 2013) identified personal motivations to use wearable devices for travel and tourism. Atembe (Atembe,, 2015) compiled the limited use of wearable devices in tourism by presenting some use cases and providing the usage of wearable devices there. One early work in this area is the one of Vlahakis et al. (Vlahakis et al.,, 2002), who designed a system that provides interactive, personalized, augmented reality tours in archaeological sites using mobile and wearable computers.

We work towards the usage of wearable devices in the tourism domain by showing a way to derive contextual information by using the data collected by wearable sensors and then utilizing the context to proactively recommend relevant activities during a trip suitable to her physiological conditions.

2.3. Data Ecosystems

The basic assumption is that sensors in wearable devices are capable of measuring different health parameters, such as the heart rate or blood pressure, and report this to a centralized application. Through constant monitoring, the sensor data can be used to determine different physiological conditions, e.g., to find out how active, tired, or hungry the user is. In a second step a recommender system can then utilize these conditions as context factors to make recommendations.

The feasibility of our approach is, thus, highly dependent on the real-time availability of accurate sensor data. Fortunately, there are commercial data ecosystems in place, such as Apple Health555https://apple.com/lae/ios/health/ or Samsung Health666https://www.samsung.com/global/galaxy/apps/samsung-health/. Using these apps on the mobile phone, a user can organize various health data at a single place, which can be visualized and analyzed. Figure 1 shows three screenshots of Apple Health. The first is the overview page, which mentions broad categories like ‘Activity’, ‘Mindfulness’, ‘Nutrition’, and ‘Sleep’, but also includes further points like heath-related data and even ‘Reproducible Health’, which is aimed exclusively for women to keep track of their menstrual cycle. The second screenshots depicts an aggregation of the activities recorded by the iPhone, while the third screenshot shows which heart-related data can be tracked. It can be seen as a manifestation of the ‘Quantified Self’ movement, where the users continuously record their own body data to analyze them with the motif of self improvement (Swan,, 2013). The data is collected via connected wearable devices, entered manually, or by granting read and write access to third party applications on the iPhone as can be seen in the third screenshot. This constitutes a data ecosystem, which can be used to develop novel applications leveraging this data given the user consents. Another example is the work of Pandian et al. (Pandian et al.,, 2008), a wearable sensor monitoring system that can measure and transmit physiological parameters to a remote monitoring station along with the geo-location of the wearer.

2.4. Inferring Physiological Conditions

Health parameters collected by the wrist-wearable devices have already been studied and used to deduce relevant physiological conditions from them (Kamišalić et al.,, 2018). For example, Alfeo et al. (Alfeo et al.,, 2018) collected information about heart rate and arm motion using wearable devices to assess per-night sleep quality of the wearer. Zhang et al. (Zhang et al.,, 2018) used wrist-worn wearable sensors to detect gestures and showed a correlation between the feeding gesture counts and caloric intake more of which lead to overeating. A wrist-worn sensing system was developed by Sugimoto et al. (Sugimoto et al.,, 2005) for calculating energy expenditure, by estimating oxygen uptake from correlation between heart rate and oxygen uptake. Defining some thresholds, we can find out if a person is active, relaxed or tired. We chose the following conditions that can be detected using wearable sensor data and are also relevant for travelers.

• •

Active (Sugimoto et al.,, 2005; Kamišalić et al.,, 2018)

• •

Relaxed (Sugimoto et al.,, 2005; Kamišalić et al.,, 2018)

• •

Tired (Sugimoto et al.,, 2005; Kamišalić et al.,, 2018; Alfeo et al.,, 2018)

• •

Drunk (Selvam et al.,, 2016)

• •

Hungry (Zhang et al.,, 2018; Sugimoto et al.,, 2005)

• •

Stressed (Gjoreski et al.,, 2017; Sandulescu et al.,, 2015; Kamišalić et al.,, 2018)

3. Deriving Activities from Physiological Conditions

The physiological condition of a tourist influences the suitability of the recommended activities. Literature lists various activity taxonomies (Mehmetoglu,, 2007), however, in this research we fall back to a subset of Foursquare’s venue categories777https://developer.foursquare.com/docs/resources/categories, as tourist activities for our data model. The main advantage is that they cover most touristic activities and provide a direct mapping to concrete items, which is useful for the final recommendation step. The five categories are

(1) Outdoors & Recreation,

(2) Arts & Entertainment,

(3) Food,

(4) Residence, and

(5) Nightlife.

Which activity should be suggested to the tourist depends on her current contextual and physiological conditions. For example:

• •

If the tourist is very active, she can be suggested to do sports or adventurous outdoor activities like horse-riding.

• •

If a traveler has not slept for a long time or is stressed, recommending a relaxing activity might be fitting.

• •

Contrary, if she is relaxed, we would suggest her an intellectual activity.

• •

Finally, if someone has been very active over a prolonged time without taking a break, we would recommend to refresh herself with food or drinks.

The above examples describe assumptions of one might expect tourists to behave in general under their particular physiological conditions. In reality, people might not follow that and act more peculiarly. Therefore, a context-aware recommender system need to learn the weights between the physiological conditions derived from wearable sensors and the respective activities. For this, a machine learning algorithm shall be employed to observe the behavior of the tourists given their physiological conditions. The upper part of Figure 2, depicts selected positive and negative influences from physiological conditions to tourist activities. Green arrows imply a positive correlation, i.e., if the condition has a higher value, the activity at the end of the arrow is recommended. If the arrow is orange, the score of corresponding activity is decreased. In our proposed model, the user model comprises a normalized value of each physiological conditions.

The associations between the sensor values and the physiological conditions in the lower part of Figure 2 can be derived as discussed in Section 2.4. Then, we can form a 1X6 physiological condition vector, $PC$ (see Equation 1), that denotes the current status of a tourist’s physiological conditions, each column having a value, $c_{a},c_{r},c_{t},c_{d},c_{h},c_{s}$ for each of the six physiological conditions: active, relaxed, tired, drunk, hungry and stressed.

[TABLE]

Next, we use the previously learned associative weights of our activity variables, $or$ (outdoors & recreation), $ae$ (arts & entertainment), $fd$ (food), $rs$ (residence), and $nl$ (nightlife) to compute the fitting activity. This is done by forming a $6X5$ weight matrix, $W$, as shown in Equation 2. Each column holds the values yielded from the machine learning for each activity, under each of the physiological condition in each row. For example, $w_{(or/a)}$ denotes the weight value of opting for activities from the group ‘outdoor and recreation’ having an ‘active’ physiological condition.

[TABLE]

$PC.W$ results in the $1X5$ Activity Recommendation Index (ARI) vector as shown in Equation 3.

[TABLE]

The tourist activity to be recommended is the one with the highest ARI value. Now the recommender system will choose the an item from this category based on the user preferences.

4. Conclusions and Future Work

This paper discusses a method to use wearable devices for deriving the physiological context of a tourist and by this to improve proactive recommendations of activities during a trip. The sensors equipped in the wearables monitor various physiological parameters of a tourist, and by further processing this data, it is possible to derive the physiological context of a traveler such as being hungry or stressed. The main contribution is a model to facilitate the already available data from health-related application such as Apple Health to recommend tourist activities.

In future, we plan to implement and evaluate the model in a mobile tourist recommender system. The challenge is to convince users to make their health-related data available for the recommender for potentially more fitting recommendations. Since data ecosystems like Apple Health make it easy to share it, we are optimistic that this is possible in general. We hope that the users’ trust can be achieved by open-sourcing the application and keeping the data on the device of the user.