Evaluating Generative AI Tools for Personalized Offline Recommendations: A Comparative Study

TL;DR

This study compares five generative AI tools in recommending personalized offline activities for health interventions, assessing their accuracy and user satisfaction to identify the most effective solutions.

Contribution

It provides a systematic evaluation of generative AI tools for personalized offline recommendations in health contexts, an area previously underexplored.

Findings

Identified the most accurate AI tool for offline activity recommendations.

Analyzed how different AI tools impact user satisfaction.

Provided insights into the effectiveness of generative AI in health-related behavioral interventions.

Abstract

Background: Generative AI tools have become increasingly relevant in supporting personalized recommendations across various domains. However, their effectiveness in health-related behavioral interventions, especially those aiming to reduce the use of technology, remains underexplored. Aims: This study evaluates the performance and user satisfaction of the five most widely used generative AI tools when recommending non-digital activities tailored to individuals at risk of repetitive strain injury. Method: Following the Goal/Question/Metric (GQM) paradigm, this proposed experiment involves generative AI tools that suggest offline activities based on predefined user profiles and intervention scenarios. The evaluation is focused on quantitative performance (precision, recall, F1-score and MCC-score) and qualitative aspects (user satisfaction and perceived recommendation relevance). Two research questions were defined: RQ1 assessed which tool delivers the most accurate recommendations, and RQ2 evaluated how tool choice influences user satisfaction.