SalienTrack: providing salient information for semi-automated self-tracking feedback with model explanations

TL;DR

SalienTrack is a framework designed to identify and explain salient events in self-tracking data, enhancing reflection and awareness through explainable AI in nutrition tracking.

Contribution

This paper introduces the SalienTrack framework, combining salience detection and explanation to improve self-tracking feedback, with a practical implementation and user insights.

Findings

SalienTrack effectively identifies salient nutrition events.

Participants found salient feedback helpful for reflection.

The framework can be adapted to other self-tracking domains.

Abstract

Self-tracking can improve people's awareness of their unhealthy behaviors and support reflection to inform behavior change. Increasingly, new technologies make tracking easier, leading to large amounts of tracked data. However, much of that information is not salient for reflection and self-awareness. To tackle this burden for reflection, we created the SalienTrack framework, which aims to 1) identify salient tracking events, 2) select the salient details of those events, 3) explain why they are informative, and 4) present the details as manually elicited or automatically shown feedback. We implemented SalienTrack in the context of nutrition tracking. To do this, we first conducted a field study to collect photo-based mobile food tracking over 1-5 weeks. We then report how we used this data to train an explainable-AI model of salience. Finally, we created interfaces to present salient information and conducted a formative user study to gain insights about how SalienTrack could be integrated into an interface for reflection. Our key contributions are the SalienTrack framework, a demonstration of its implementation for semi-automated feedback in an important and challenging self-tracking context and a discussion of the broader uses of the framework.