Alexa, in you, I trust! Fairness and Interpretability Issues in E-commerce Search through Smart Speakers

TL;DR

This study examines fairness and interpretability issues in voice-assisted e-commerce search, revealing discrepancies between user understanding, product relevance, and user preferences, raising concerns about trustworthiness of voice assistants like Alexa.

Contribution

It provides empirical evidence on fairness and interpretability challenges in voice-based e-commerce search, highlighting user perception gaps and relevance mismatches.

Findings

81% users interpret 'top result' differently from Alexa

68% of Alexa's chosen products are less relevant than top desktop results

73% of users prefer desktop top results over Alexa's choices

Abstract

In traditional (desktop) e-commerce search, a customer issues a specific query and the system returns a ranked list of products in order of relevance to the query. An increasingly popular alternative in e-commerce search is to issue a voice-query to a smart speaker (e.g., Amazon Echo) powered by a voice assistant (VA, e.g., Alexa). In this situation, the VA usually spells out the details of only one product, an explanation citing the reason for its selection, and a default action of adding the product to the customer's cart. This reduced autonomy of the customer in the choice of a product during voice-search makes it necessary for a VA to be far more responsible and trustworthy in its explanation and default action. In this paper, we ask whether the explanation presented for a product selection by the Alexa VA installed on an Amazon Echo device is consistent with human understanding as well as with the observations on other traditional mediums (e.g., desktop ecommerce search). Through a user survey, we find that in 81% cases the interpretation of 'a top result' by the users is different from that of Alexa. While investigating for the fairness of the default action, we observe that over a set of as many as 1000 queries, in nearly 68% cases, there exist one or more products which are more relevant (as per Amazon's own desktop search results) than the product chosen by Alexa. Finally, we conducted a survey over 30 queries for which the Alexa-selected product was different from the top desktop search result, and observed that in nearly 73% cases, the participants preferred the top desktop search result as opposed to the product chosen by Alexa. Our results raise several concerns and necessitates more discussions around the related fairness and interpretability issues of VAs for e-commerce search.