Seeker: Real-Time Interactive Search

TL;DR

Seeker is a real-time interactive search system enabling users to refine search results through simple feedback, improving search relevance by leveraging human input without requiring explicit item descriptions or representations.

Contribution

The paper introduces a novel interactive search algorithm that incorporates user feedback to adapt search results in real time, without needing explicit item representations.

Findings

Effective real-time search refinement demonstrated

Quantitative and qualitative evaluation confirms improved relevance

Human-in-the-loop experiments validate approach

Abstract

This paper introduces Seeker, a system that allows users to interactively refine search rankings in real time, through feedback in the form of likes and dislikes. When searching online, users may not know how to accurately describe their product of choice in words. An alternative approach is to search an embedding space, allowing the user to query using a representation of the item (like a tune for a song, or a picture for an object). However, this approach requires the user to possess an example representation of their desired item. Additionally, most current search systems do not allow the user to dynamically adapt the results with further feedback. On the other hand, users often have a mental picture of the desired item and are able to answer ordinal questions of the form: "Is this item similar to what you have in mind?" With this assumption, our algorithm allows for users to provide sequential feedback on search results to adapt the search feed. We show that our proposed approach works well both qualitatively and quantitatively. Unlike most previous representation-based search systems, we can quantify the quality of our algorithm by evaluating humans-in-the-loop experiments.

Equations

$$||x_i-x_t|{|}^2<||x_j-x_t|{|}^2.$$

$$\mathbb{P}\big{(}s_{ij}|t,i,j\big{)}=\frac{1}{1+\exp\big{\{}-\alpha\big{(}||x_{j}-x_{t}||^{2}-||x_{i}-x_{t}||^{2}\big{)}\big{\}}},$$

$$\mathbb{P}(S|t,A,B)=\prod_{s_{ij}\in S}\mathbb{P}\big{(}s_{ij}|t,i,j\big{)},$$

$$\log\mathbb{P}(S|t,A,B)=\sum_{s_{ij}\in S}\log\mathbb{P}\big{(}s_{ij}|t,i,j\big{)}.$$

$$\mathbb{P}(t|S)\propto\mathbb{P}\big{(}S|t\big{)}\mathbb{P}(t),$$

$$\begin{split}\log\mathbb{P}(t|S)&\propto\log\mathbb{P}\big{(}S|t\big{)}+\log\mathbb{P}(t)\\ &=\sum_{s_{ij}\in S}\log\mathbb{P}\big{(}s_{ij}|t,i,j\big{)}+\log\mathbb{P}(t).\end{split}$$

$$p_j=\frac{e^{g_j}}{{\sum }_{i=1}^N{e}^{g_i}}.$$

$$\underset{j}{\mathrm{argmax}}\{g_j+Gumbel(0,1)\}\sim \frac{e^{g_j}}{{\sum }_{i=1}^N{e}^{g_i}}=p_j.$$

$$f_{i}(z)=e^{-\big{(}z-g_{i}+e^{-(z-g_{i})}\big{)}},$$

$$F_i(z):=\mathbb{P}(z_i\le z)=e^{-e^{-(z-g_i)}}.$$

$$\mathbb{P}(j^{\ast })$$

$$\displaystyle=\int_{z_{j}=-\infty}^{+\infty}\,e^{-\big{(}z_{j}-g_{j}+e^{-(z_{j}-g_{j})}\big{)}}\prod_{i\neq j}e^{-e^{-(z_{j}-g_{i})}}\,dz_{j}$$

$$={\int }_{z_j=-\infty }^{+\infty }{e}^{-z_j+g_j-e^{-z_j}{\sum }_{i=1}^N{e}^{g_i}}d{z}_j$$

$$\displaystyle=\frac{e^{g_{j}-e^{-z_{j}}\sum_{i=1}^{N}e^{g_{i}}}}{\sum_{i=1}^{N}e^{g_{i}}}\Biggr{|}_{-\infty}^{+\infty}=\frac{e^{g_{j}}}{\sum_{i=1}^{N}e^{g_{i}}}=p_{j}.$$

$$p_j=\frac{e^{{\eta }_k{g}_j}}{{\sum }_{i=1}^N{e}^{{\eta }_k{g}_i}}.$$

$$\underset{j}{\mathrm{argmax}}\{{\eta }_k{g}_j+Gumbel(0,1)\}$$

$$\sim \frac{e^{{\eta }_k{g}_j}}{{\sum }_{i=1}^N{e}^{{\eta }_k{g}_i}}.$$

$$\operatorname*{argmax}_{j}\{\eta_{k}\,g_{j}+Gumbel(0,1)\}\sim\operatorname*{argmax}_{j}\Big{\{}g_{i}+\frac{Gumbel(0,1)}{\eta_{k}}\Big{\}}.$$

$$\operatorname*{argmax}_{j}\Big{\{}g_{j}+\sqrt{C^{2}/n_{j}}\;Gumbel(0,1)\Big{\}}.$$

$$\mathbb{P}(S|t)=\prod _{i=1}^p\mathbb{P}(a_i|t)\prod _{j=1}^q\mathbb{P}(b_j|a_1,\cdots ,a_p,t).$$

$$\mathbb{P}(a_{i}|t)=\frac{1}{\exp\big{\{}\alpha_{1}||a_{i}-x_{t}||^{2}\big{\}}},$$

$$\mathbb{P}(b_{j}|a_{1},\cdots,a_{p},t)=\\ \frac{1}{1+\exp\Big{\{}-\alpha_{2}\Big{(}||b_{j}-x_{t}||^{2}-\min\limits_{i\in\{1,\cdots,p\}}||a_{i}-x_{t}||^{2}\Big{)}\Big{\}}},$$

$$\mathbb{P}(t|S,\mathbf{r})\propto \mathbb{P}(S|t,\mathbf{r})\mathbb{P}(t|\mathbf{r}),$$

$$\mathbb{P}(t|\mathbf{r})=\frac{1}{1+\exp\Big{\{}-\beta\sum\limits_{k}r_{k}^{T}x_{t}\Big{\}}}.$$

$$\mathbb{P}\big{(}s_{ij}|t,i,j,\mathbf{r}\big{)}=\\ \frac{1}{1+\exp\Big{\{}-\alpha\Big{(}||x_{j}-x_{t}||^{2}-||x_{i}-x_{t}||^{2}+\sum\limits_{k}r_{k}^{T}(x_{i}-x_{j})\Big{)}\Big{\}}}.$$

$$\varphi (c,x)=c\odot x.$$

$$D_{n}:=\sum_{x_{k}\in X_{n}}\big{|}\mathbb{P}(x_{k}|\text{ Method 1})-\mathbb{P}(x_{k}|\text{ Method 2})\big{|}.$$

$$i=\underset{k\in \{1,...,n\}}{\mathrm{argmin}}d(x,x_k).$$

$$\mathbb{P}(x_k|\text{ Method 1})=\mathbb{P}(x\in J_k)={\int }_{J_k}f(x)dx=mf(x_k^{\ast })\text{ for all }k.$$

$$1={\int }_{T}f(x)dx=\sum _{i=1}^n{\int }_{J_i}f(x)dx=\sum _{i=1}^{n}mf(x_i^{+}),$$

$$S=\min \left(\frac{1}{2m_T(1+U{m}_T+m_T)},1,\frac{1}{m_T}\right).$$

$$|f(x_i^{+})-f(x_i)|<Sϵ\text{ and }|f(x_k^{\ast })-f(x_k)|<Sϵ\text{ for all }i,k\in \{1,...,n\},$$

Full text

Interactive Search, Real Time Recommendation, Online Learning, Active Learning, Multi-Armed Bandit

††journalyear: 2019††copyright: rightsretained††conference: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 4–8, 2019; Anchorage, AK, USA††booktitle: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’19), August 4–8, 2019, Anchorage, AK, USA††doi: 10.1145/3292500.3330733††isbn: 978-1-4503-6201-6/19/08††ccs: Information systems Search interfaces††ccs: Information systems Probabilistic retrieval models††ccs: Information systems Information retrieval diversity††ccs: Information systems Test collections††ccs: Information systems Relevance assessment††ccs: Computing methodologies Online learning settings††ccs: Computing methodologies Active learning settings††ccs: Computing methodologies Discrete space search††ccs: Computing methodologies Search with partial observations††ccs: Computing methodologies Sequential decision making

1. Introduction

Search engines and online shopping websites maintain indices with millions of items. Often, it is difficult for a user to accurately describe in words what they are looking for (Teo et al., 2016). Even if the user is able to describe their target item effectively, large index and catalog sizes mean it is difficult to sift through similar items efficiently.

Consider the situation in which a user is searching for a new movie to watch. They have a mental representation of the characteristics of the movie they would enjoy but are not acquainted with the genre keywords, latest movies, actors or directors. Being unfamiliar with current movie jargon, they are unable to accurately describe their preferred movie with a traditional keyword interface, nor do they have an example photograph. However, if we show the same user another movie they have seen and ask them “Is this movie similar to the one they have in mind? Yes or no?”, people can answer such ordinal questions with less noise than absolute judgments – i.e. finding the exact words to describe their choice (Stewart et al., 2005).

The above scenario is not restricted to movies only. In the case of browsing for a song on a media platform, searching for a news article on a news website, or a dress on an online platform, the user may not be able to accurately describe the desired item in a traditional keyword interface. But users could provide relative judgments based on what they have experienced before. For example, answers to queries like “Songs similar to Heroes by David Bowie: Yes or no?” or “News similar to that of the Queen’s involvement with Brexit: Yes or no?” are easier to provide.

In addition, traditional search engines (Ledford, 2015; Smith and Linden, 2017) and the newer representation search systems (described in Section 2) are temporally static. The engines use text or imagery as the query and respond with a ranked list of results. This ranking is based on an estimate of relevance to the user in their current context – location, historical searches etc. They do not provide the user the opportunity to adapt and fine-tune the resulting page with additional feedback. In traditional engines, for a given user in a given session, each query is independent of each other. Figure 1 illustrates the difference between traditional engines and our setting.

In this paper, we describe our system, Seeker, that dynamically refines search results based on real-time interactions with the user (in the form of likes and dislikes) within a single search session. From a customer perspective, this system adds the feeling of an ”in-store” shopping discovery experience, with a personal curator. In our setting, the user scrolls through a page of items and may ”like” or ”dislike” any item at any time. The data gathered from these preferences is used to update the list of results shown in real-time, thereby iteratively closing in on what they are looking for. To our knowledge, Seeker is the first interactive and dynamic search experience which enables the user to seamlessly zoom in, zoom out, and pivot by scrolling up and down and selecting items to like and dislike in an adaptive manner.

In this work we make the following contributions:

• •

Introduce Seeker, an interactive recommendation algorithm deployed at scale, which adapts to customer inputs in real time.

• •

Propose a novel evaluation metric with humans in the loop that allow us to quantify the quality of our proposed algorithm and evaluate it against other methods. Most embedding-based representational search engines in the past have evaluated their systems only qualitatively rather than quantitatively. In our experiments, we simulate the tasks of searching for a particular item, and quantifiably measure progress.

The paper is organized as follows. In Section 2, we review related papers and search engines. In Section 3, we describe how we model human preferences expressed in likes and dislikes and translate those preferences into probability distributions over our catalog. In Section 4, we present our adaptive algorithm for making real time recommendations. In Section 5, we evaluate Seeker’s results. In Section 6, we discuss directions for future research. In Section 7, we summarize our work.

2. Related Work

Over the last few years, there has been a growing trend of exploring new interfaces beyond traditional keyword search, and in particular, visual-based search (Datta et al., 2008). In (Hadi Kiapour et al., 2015), users query relevant items by uploading real world photographs of clothing. The engine then displays results that are visually similar to the query photograph. Pinterest built a system which allowed users to hover over pins and find visually similar items in the catalog (Jing et al., 2015a). An advantage of these systems is that they help people find things using an understanding they might not be able to put into words.

Many lines of research focused on learning the relative similarities of images. They accomplish this by mapping each image to a numerical vector, so they can capture the visual similarities in Euclidean space (Lai et al., 2015; Babenko et al., 2014; Li et al., 2016; Xia et al., 2014; Zhu et al., 2016). Using the similar approach but in a scalable manner, companies have also rolled out their visual search platforms, from Google Goggles, Google Similar Images, and Amazon Flow to Microsoft (Bing) (Hu et al., 2018), Pinterest (Jing et al., 2015b), eBay (Yang et al., 2017), and Alibaba (Zhang et al., 2018).

All these methods, however, require the user to provide a photograph of the targeted item. They fail when users do not have an actual visual representation of the desired item, but instead a mental picture of it. The users themselves may not know how to properly describe their mental visualization in words. Our algorithm addresses this issue, as Seeker is able to work with any embedding representation, including visual, textual and audio.

Moreover, Seeker dynamically adjusts the search results based on interactive user feedback; all mentioned projects do not allow users to fine-tune their current-session search with additional feedback. While we use proprietary embeddings in the examples of this paper, the underlying engine can operate upon features derived from other domains (or combinations of domains) as well – customer behavior, language understanding, audio, etc.

3. Problem Formulation

3.1. The Setting

Figure 1 illustrates how Seeker is different from traditional search. The user starts with a ranked list of results and provides feedback in the form of likes or dislikes; the search engine then generates a new set of ranked results, updating the page in real-time. In this section, we define the notation to formally describe the above search process.

Assume that we have a catalog of $N$ items, out of which $M\ll N$ can be displayed. We model user feedback as a sequence of likes and dislikes over discretized timesteps $t_{0},t_{1},\cdots,t_{k}$. The user starts with an initial ranking of items at timestep $t_{0}$. This initial $t_{0}$ ranking can be thought of as Seeker’s prior belief on what the user desires, can be generated from a traditional search or recommendation engine, and may incorporate diversity or business requirements.

The user interacts with the page by liking or disliking items. At each timestep, $t_{k}$, Seeker produces a new ranked list of results, based on the feedback from $t_{0},\cdots,t_{k-1}$. It does so by constructing a discrete probability distribution over the catalog of $N$ items at each timestep. The probability distribution represents the likelihood of an item being the user’s desired item.

We featurize each catalog item $i$ by embedding it into a vector space $x_{i}\in\mathbb{R}^{d}$. Seeker requires a high correlation between human perception of similarity and distance metric in the embedded vector space. Based on the properties of the items displayed, embedding strategies described in (Le and Mikolov, 2014; Peters et al., 2018; Devlin et al., 2018; Szegedy et al., 2016) have been shown to correlate with human perception.

Seeker can be divided into three major components, as seen in Figure 2. Section 3.2 describes how we convert likes and dislikes to preference pairs and probability distributions. Section 4.1 details how we use preference pairs to estimate a target’s likelihood. Section 4.4 shows how we use probabilistic sampling to recommend items to users at each timestep.

3.2. Pairwise Comparison

Let $a_{i}^{k}\in\mathbb{R}^{d}$ for $i=1,...,p$ be the vector representations of the liked items, and $b_{j}^{k}\in\mathbb{R}^{d}$ for $j=1,...,q$ the vector representations of the disliked items. We will drop the superscript when the context is clear. Let $A:=\{a_{1},...,a_{p}\}$ and $B:=\{b_{1},...,b_{q}\}$ be the non-empty subsets of $\{x_{1},...,x_{N}\}$. We define $s_{ij}$ as the preference pair which consists of a liked item $a_{i}$ and a disliked item $b_{j}$ from sets $A$ and $B$ respectively. We create $pq$ preference pairs from all cross-pairings between the $p$ likes and $q$ dislikes.

The intuition behind preference pairs follows from our assumption that the user has some ideal item $t$ (referred to as the target) in their mind that they wish to find. Then $s_{ij}\in S$ represents the preference that the user thinks item $i$ is more similar to their desired item $t$ than item $j$, i.e they prefer i over j given t:

[TABLE]

Equation 3.1 resolves to item $t$ being spatially closer to item $i$ than it is to item $j$. In this paper we use the Euclidean distance to measure vector similarity, but Seeker is agnostic to the metric used.

We use preference pairs to model the probability of a catalog item being the hidden target item $t$, featurized as $x_{t}$. If we were to present a user with item $x_{i}$ and item $x_{j}$, what is the probability that they chose $i$ over $j$? Questions of this form are known as triplets in the Machine Learning literature (Jain et al., 2016; Schroff et al., 2015). Equation 3.2 mathematically models our question:

[TABLE]

where $\alpha\geq 0$.

Intuitively, the answer to the above triplet question should depend on how similar items $x_{i}$ and $x_{j}$ are to $x_{t}$. As similarity and distance are equivalent in our world, the probability of preferring $i$ over $j$ becomes a function of how close $x_{i}$ and $x_{j}$ are to $x_{t}$. According to this model (and Equation 3.2), if items $x_{i}$ and $x_{j}$ are equidistant from the target $x_{t}$, then they are equally preferred, and the probability of choosing $i$ over $j$ is $0.5$. If $x_{i}$ is the target $x_{t}$ while $x_{j}$ is infinitely far away, the probability of choosing $i$ becomes $1$. For items in the middle we get a smooth noise model that accounts for the stochasticity in human decisions.

Our model includes a preference hyperparameter $\alpha$, which represents our confidence in the vector space representation:

• •

When $\alpha=0$, then \mathbb{P}\big{(}S_{ij}|t\big{)}=0.5 for all combinations of targets, likes and dislikes. This means our embeddings have no correlation with human judgment of similarity, and preferring $i$ over $j$ is as good as a fair coin flip.

• •

When $\alpha=\infty$, then \mathbb{P}\big{(}S_{ij}|t\big{)}=1. This removes randomness from the decision process, perfectly aligns our representation of human judgment with the metric distance, and deterministically picks the closer item.

We use $\alpha=1$ for the results discussed in Section 5.

4. Item Ranking

In this section we describe how we go from preference pairs and a noise model to a ranked list of items to be displayed to the user.

4.1. Target Estimation

To keep our notation consistent, we always assume that the user prefers item $x_{i}$ to $x_{j}$ when we write $s_{ij}\in S$. We make the further assumption that each preference pair is independent from each other. This is a simplifying assumption which serves as a good baseline  (Chapelle and Li, 2011; Hill et al., 2017). In Section 6.1, we investigate ways to drop the independence assumption. Equation 4.1 represents the joint distribution likelihood of observing preferences $S$, given target item $t$ and likes and dislikes sets $A$ and $B$:

[TABLE]

where $\mathbb{P}(s_{ij}|t,i,j)$ is defined as in Equation 3.2. The log-likelihood becomes:

[TABLE]

We do not know a priori what is the hidden target $t$. Our goal is to find $t$ or approximate it. We note that $t$ may not be present in our catalog, and in this case our goal is to find an item as similar to $t$ as possible. In order to build a probability distribution over our catalog, we borrow ideas from (Tamuz et al., 2011). We use the same noise model, but apply it to recommend items to the user, instead of learning a metric space. For each catalog item, we compute the log-likelihood mass of that item being the target, given the user’s likes and dislikes, as shown in Algorithm 1.

4.2. Posterior Construction

Instead of presenting items according to their likelihood of being the target, we allow for the inclusion of priors into our model. Let $\mathbb{P}(i)$ be the prior probability of item $i$ being the actual desired target. One can compute such priors using traditional search engines, and personalize them using the user’s browsing or purchase history (Teo et al., 2016).

Given priors $\mathbb{P}(i)$, the posterior probability of an item being the target is:

[TABLE]

and the log-posterior becomes:

[TABLE]

At each time step the user provides feedback causing the size of $S$ to grow. Therefore the log likelihood will eventually dominate the posterior density score. In the early stages when we have fewer likes and dislikes, our posterior belief on the target is dominated by a well founded prior. This prevents us from having to wait a long time before showing meaningful results.

4.3. Items Recommendation

We consider four different ways to display $M\ll N$ items to the user:

4.3.1. Pure Exploitation/Noiseless

The simplest approach is to sort the posteriors and recommend the top $M$ items. Theoretically, this prevents us from exploring the search space. Practically, this leads to a poor user experience with limited product diversity.

4.3.2. Pure Exploration/Random

The other extreme solution is to show random results all the time, completely ignoring the posterior densities.

4.3.3. Epsilon-greedy

Another approach is to randomize some of the results while leaving the others untouched, as in Algorithm 2. We rank items by their posterior densities, and replace each item with a random item with probability $\epsilon$. See (Bubeck et al., 2012) for a detailed study of $\epsilon$-greedy algorithms.

4.4. Boltzmann Exploration

The fourth method to recommend items involves sampling without replacement according to the item’s posterior densities. Let $g_{j}$ be a score associated with item $j$. A popular way to generate a discrete distribution over the items is by using the exponential weighing scheme, known as the softmax or Boltzmann equation:

[TABLE]

Here, $p_{j}$ is our belief probability that item $j$ is the true target. Even though $g$ is unconstrained in $\mathbb{R}$, common values are $g_{j}=\mathbb{P}(x_{j}|S)$ and $g_{j}=\log\mathbb{P}(x_{j}|S)$, the latter resulting in polynomial weighing (Szepesvári, 2010; Jang et al., 2017). Note that if the items were equally spaced, sampling from the discrete distribution $p_{j}$ is asymptotically equivalent to sampling from the hidden continuous distribution, as we show in Appendix A.

Sampling without replacement when $N$ is large can prove to be very slow. When $N$ and $d$ are large, normalizing our posterior densities can lead to precision issues with sampling. We can overcome this problem by using the Gumbel-Max trick (Maddison et al., 2014), which shows that adding standard Gumbel noise to $g_{i}$ and taking the $\max$ is equivalent to sampling according to Boltzmann (Equation 4.5):

[TABLE]

We sketch the proof for completeness. Let $z_{i}=g_{i}+Gumbel(0,1)$. By the additive property, $z_{i}\sim Gumbel(g_{i},1)$, with probability density function (PDF):

[TABLE]

and cumulative distribution function (CDF):

[TABLE]

Proof.

Define by $\mathbb{P}(j^{*})$ the probability that $z_{j}$ is the largest among all $z_{i}$. We have:

[TABLE]

∎

Since the added Gumbel noises are independent, showing the $M$ items with the highest $z_{i}$ scores is equivalent to sampling $M$ items without replacement from Equation 4.5.

To balance exploration and exploitation, one resorts to annealing (Aarts and Korst, 1988), with an appropriately tuned sequence of learning rate parameters (aka inverse temperature) $\eta_{k}\geq 0$ for each timestep $t_{k}$:

[TABLE]

Note that $\eta_{k}=0$ recovers the pure exploration mode, and $\eta_{k}=+\infty$ recovers the pure exploitation mode. Varying $\eta_{k}$ allows us to trade-off exploitation and exploration.

On the other hand, similarly to the proof above, we have

[TABLE]

Note that, by dividing by $\eta_{k}$, we establish:

[TABLE]

Sampling from $Gumbel(\eta_{k}\,g_{j},1)$ and taking the maximum, as in Equation 4.4, is similar to Thompson Sampling in a bandit setting (Thompson, 1933; Russo et al., 2018). The crucial difference (and drawback) is that the Gumbel method doesn’t take into account the uncertainty of the reward estimates.

Finding the right schedule for $\eta_{k}$ can be very difficult in practice (Vermorel and Mohri, 2005). In (Cesa-Bianchi et al., 2017), the authors provide an annealing schedule for $\eta_{k}$ in a standard stochastic multi-armed bandit setting, guaranteeing sublinear regret. Let $n_{j}$ be the number of times arm $j$ has been played up to timestep $t_{k-1}$. For some constant $C>0$, they set $\eta=\sqrt{n_{j}/C^{2}}$, and sample according to:

[TABLE]

Equation 4.13 decouples the learning rates of the individual items, and factors-in the uncertainty of the reward estimates. We now have a proper way to sample from a Boltzmann, with convergence guarantees. Even though our setting is not exactly the same as (Cesa-Bianchi et al., 2017), we borrow parts of their sampling strategy to recommend items to the user. As detailed in the theoretical justifications of Appendix B, we recommend setting $C^{2}=1/8$ for $g_{j}=\mathbb{P}(x_{j}|S)$.

As a user can repeatedly interact with the same item, we treat $n_{j}$ as the number of times a user interacts with item $j$. It starts with $n_{j}=1$ and is incremented with every like or dislike to item $j$. Putting it all together, we obtain our final Boltzmann sampling algorithm (Algorithm 3).

5. Evaluation

As our experiments require human judgments, there exist no such ground truth datasets for validation. Instead we propose an experimental framework with a human in the loop that simulates the Seeker experience and generates quantifiable metrics. The evaluation study serves as a benchmark for future sequential search algorithms.

5.1. Experimental Setup

Seeker assumes that the user has a mental image of a target item they cannot easily express in words. When accessing Seeker, the user is presented with a subset of $M$ items to interact with, using like or dislike clicks. At any moment, the user can expand the catalog listing view by clicking on “Explore More”. Our experimental setting mimics this initial user experience.

A single experimental session involves the following: A user is presented with a target item $x_{t}$. This target item is an explicit simulation of the user’s hidden target. At each timestep, we present the user with a grid of $M$ items. The user’s goal is to find the target item through a series of feedbacks. At each timestep, they may like, dislike, or remove a previously liked/disliked item. Upon receiving user feedback, we recommend $M$ new items to view in the next timestep. The session goes on for $K$ timesteps. If the user can find the target within the $M$ items, they may stop playing. Otherwise, they try to get as close to the target item as possible based on their perception of similarity. For our experiments, we set $M=12$, $K=15$, $d=2048$, $N=2228$, $g_{j}=\log\mathbb{P}(x_{j}|S)$, and used an uninformative prior.

We enlisted volunteers to participate in the experiment defined above, and collected 358 (roughly 90 per sampling algorithm) unique sessions. The target and exploration algorithm for each session was selected uniformly at random. Users were instructed to like and dislike items assuming that they wanted to purchase the target item. Users had no prior knowledge of the selected catalog or algorithm. At each timestep, we invoke Seeker to generate a posterior distribution over the catalog, according to Equation 4.4. This distribution enforces a natural ranking on the items. We monitor the normalized rank of the target item at each timestep. The normalized rank $\rho$ is defined as the rank of the target item divided by the size of the catalog. A target $x_{t}$ with a normalized rank of $\rho=0.1$ means it has a final rank of \big{\lceil}0.1*N\big{\rceil}.

5.2. Experimental Results

Seeker aims at helping the user quickly zoom-in on the desired target item. A typical metric for such recommender systems is recall at k (Herlocker et al., 2004). As we have only one target of interest, we measure how close our recommendations are to target $t$. We can do that using the target’s normalized rank. For a given session $i$, let $\rho_{i}$ be the lowest normalized rank attained by $t$ in all $K$ timesteps. We define recall @$\rho_{j}$ as the percentage of sessions with $\rho_{i}\leq\rho_{j}$. For example, a recall of $0.4$ @$0.02$ means that $40\%$ of sessions achieved a normalized ranking of $\rho=0.02$ or less.

Figure 3 plots recall @$\rho$ for our sampling strategies. We plot $\rho$ up to $0.1$, as the user is unlikely to scroll past higher percentiles. Boltzmann exploration achieves the highest recall, dominating all other strategies. Noiseless and Greedy perform similarly, outperforming random at lower recalls. Random improves at higher recalls due to its higher degree of exploration, where the target gets ranked high by pure chance.

Figure 4 plots the convergence time of our sampling strategies. From the user’s perspective, this reflects how long it takes to find a reasonably close approximation of the target item. We report the mean number of steps it takes for the rank of the target item to drop below a given recall cutoff $\rho$. Boltzmann exploration consistently outperforms the other strategies. Greedy and Noiseless surpass Random, but their advantage diminishes at higher rank cutoffs.

5.3. Discussion

We would like to point out that the experimental setup described above is not restrictive. Although we do present a window of $M$ items with which the user interacts, the user can expand the window size $M$ by explicitly clicking on an “Explore More” option. Once in the expanded view, scrolling down past the last displayed item triggers the display of additional items in an infinite scroll mode covering the whole $N$ catalog items. Since we maintain an explicit ranking on all items, this mode of experimentation merges naturally with our algorithm.

Infinite scroll, such as home feeds on social media websites like Facebook and twitter, may create a better user experience and allow for the user to browse quickly. But when the target is explicit, such an infinite scroll feature makes our experimental framework trivial – the user can just scroll until they find the target. This prevents us from gaining insight about convergence, which explains why we limited our study to the windowed-version of the application. We consider our experimental setup a restrictive experience in terms of user experience.

Additionally, the catalog contains multiple similar items. This leads to a large number of identical feature-vector representations, making it challenging to surface the target item among $M=12$ items in just $K=15$ timesteps. Hence, it is likely that the Section 5.2 experimental results are pessimistic, as users are constrained from browsing the search space efficiently. Nevertheless, as the top-most items get the most visibility, Seeker’s ability to quickly zoom-in to the item of interest remains crucial.

On occasions, the Seeker interface produces pages with very similar items ranked closely, leading to lack of exploration. Two items which look mostly identical are likely to have similar vector representation and hence may appear adjacent to each other (Nassif et al., 2016). In a deterministic setting, this would have resulted in a page full of very similar items, and prevented the user from pivoting to other parts of the catalog. Although Boltzmann exploration offers a principled remedy, depending on the use case, one may want to model additional uncertainty into the user actions in the early stages of the interactions. As a remedy, we can modify the posteriors by adding noise, using submodular functions (Chen et al., 2017), or determinental point processes (Affandi et al., 2012).

The constant $C$ in Algorithm 3 is borrowed from the work in (Cesa-Bianchi et al., 2017) which uses the non-contextual stochastic multi-armed bandit setting. Under their setting $C$ is a reasonable estimate to bound variance. However, items in our search space have features that are shared and correlated. Our sampling strategy currently does not take into account this covariance properly when making recommendations. We leave augmenting our sampling algorithm with a new variance bound for future work.

6. Future Work

We are considering improving this work on multiple fronts.

6.1. Bipartite Preference Model

When constructing the preferences from user feedback in Section 3.2, we treat the likes and dislikes independently, valuing them equally. However, intuitively, if we put more emphasis on likes, we may be able to find the target faster. Likes are less ambiguous than dislikes: likes have a clearer implication when treated in isolation, while dislikes usually require context to be a useful learning signal. In fact, we empirically observe that the likes are more clustered with one another than the dislikes to themselves.

We can mathematically model the emphasis of likes over dislikes by assuming the likes are independent of each other but the dislikes are conditioned on the likes. Conditional dependencies can be expressed in the form of a bipartite graph as shown in Figure 5.

Using Bayes rule, we can represent Figure 5 as:

[TABLE]

Here we assume that:

[TABLE]

and

[TABLE]

with $\alpha_{1},\alpha_{2}\geq 0$.

We interpret the model in the following way. Equation 6.2 conveys that the probability of liking item $i$ is proportional to how similar $i$ is to target $t$. Equation 6.3 conveys that the probability of disliking item $j$ is proportional to its relative distance to the target as compared to the relative distance between the target and liked items $A$. One can quantify the distance between $t$ and $A$ in different ways. Here we propose $min$ to reflect the customer’s gradual approach towards the target. We leave the evaluation of this model to future work.

6.2. Incorporating Additional Feedback

So far, the only form of feedback that the user provides is in the form of likes and dislikes. Consider the situation where the user provides feedback in the form of a text or utterance. This transitions us into the guided conversational search paradigm and we could incorporate some of the strategies described in (Huang et al., 2018; Wen et al., 2016; Shah et al., 2018).

Assume we have a technique (like LSTM to create word embeddings) to map a spoken feedback into a vector $r_{i}\in R^{d}$. We want to incorporate this feedback into the model. Equation 4.3 becomes:

[TABLE]

where

[TABLE]

To estimate $\mathbb{P}(S|t,\mathbf{r})$, we change Equation 3.2 to:

[TABLE]

Although we used text/speech as an example, the additional feedback embedding $r$ can originate from an arbitrary source. Similarly, we can incorporate extra feedback into the bipartite preference model of Section 6.1.

6.3. Personalized Recommendations

Another possible direction is to personalize Seeker. Let $c$ be an embedding vector for each user. The dataset now comes in the form of quadruplets $(c,t,a,b)$, where each user $c$ has a target $t$ and pairs $(a,b)$ of likes and dislikes.

To personalize, we define a synthetic embedding kernel $\phi(c,x)$, where $x$ denotes an item. For example, we can use element-wise product:

[TABLE]

Now, we can substitute this kernel into our modeling formulas, replacing any item $x$ with personalized embedding $\phi(c,x)$.

7. Conclusion

This paper presents Seeker, an interactive, real-time search system. Seeker allows users to search for products even when it is difficult to describe them in words. Unlike embedding-based search engines, this method does not require a preknown representation of the desired item. With interactive binary feedback, our system learns to dynamically refine search results from the user’s preferences in real time. Our evaluation results show that our Boltzmann exploration method allows users to find their products more quickly and with greater regularity compared to alternative exploration strategies.

Acknowledgements.

We gratefully acknowledge Kevin Jamieson and Lalit Jain for sharing their experience of designing adaptive algorithms. We would like to thank Miguel Jimenez Gomez, Xiaopeng Zhang and Andrea Matsunaga for their software development expertise. In addition, we thank the volunteers for their help in performing the user study. Finally, we thank Amazon for the opportunity to conduct this research project.

Appendix A Asymptotic Sampling Equivalence

Given $T$ as a compact (i.e. closed and bounded) subset of $\mathbb{R}^{d}$. Let $f:T\to\mathbb{R}_{+}$ be a continuous probability density function. Let the set $X_{n}=\{x_{1},...,x_{n}\}$ consist of points $x_{i}$’s that are equally spaced on $T$ in the grid-like manner such that $\sum_{i=1}^{n}f(x_{i})>0$. Consider the following two ways of sampling over $X_{n}$:

1. (1)

Each time, sample $x$ from $f$ on $T$, and choose $x_{k}\in X_{n}$ if and only if $x_{k}=\operatorname*{argmin}_{x_{i}\in X_{n}}d(x,x_{i})$, where the metric $d(\cdot,\cdot)$ is usually the Euclidean metric. We assume argmin is unique. 2. (2)

Each time, sample each $x_{k}\in X_{n}$ from the discrete distribution on $X_{n}$ so that $x_{k}$ is chosen with probability $f(x_{k})/\sum\limits_{i=1}^{n}f(x_{i})$.

Define:

[TABLE]

Prove that $\displaystyle\lim_{n\to\infty}D_{n}\to 0$.

Proof.

Partition $T$ into $n$ disjoint regions $(J_{1},...,J_{n})$ in the grid-like manner such that for each $i\in\{1,...,n\}$ for each $x\in J_{i}$,

[TABLE]

Since the points $x_{i}$’s are equally spaced on $T$, the regions $J_{i}$’s all have the same measure: $m(J_{i})=m=\frac{m_{T}}{n}$, where $m_{T}$ is the (fixed) Lebesgue measure of $T$. So

[TABLE]

Here the first equation holds by the definition of $J_{i}$’s, and the second by the Mean Value Theorem (MVT) for some $x_{k}^{*}\in J_{k}$. On the other hand,

[TABLE]

where the second equation holds by the additivity of integral, and the last equation holds by the MVT for some $x_{i}^{+}\in J_{i}$.

Since $f$ is continuous on the compact subset $T$ of $\mathbb{R}^{d}$, there is an upper bound $U$ such that $f(x)<U\text{ for all }x\in T$. Moreover by the Heine - Cantor theorem, $f$ is uniformly continuous on $T$.

Now fix $\epsilon>0$, $\epsilon<1/2$. By uniform continuity of $f$ on $T$, there exists $\delta>0$ such that for all $x_{1},x_{2}\in T$ with $d(x_{1},x_{2})<\delta$, we have $|f(x_{1})-f(x_{2})|<S\epsilon$ where

[TABLE]

Because the regions $J_{i}$’s are partitioned in the grid-like manner, there exists $N_{0}\in\mathbb{Z}_{+}$ such that for all $n>N_{0}$, the diameter of each $J_{i}$ is smaller than $\delta$. This implies $d(x_{i}^{+},x_{i})<\delta$ for all $i$ and $d(x_{k}^{*},x_{k})<\delta$ for all $k$. Hence for all $n>N_{0}$, we have

[TABLE]

which implies:

[TABLE]

Therefore for each $k\in\{1,2,...,n\}$,

[TABLE]

This implies that for all $n>N_{0}$,

[TABLE]

This ends the proof. ∎

Appendix B Setting Parameter $C$

Given an item, a user can like or dislike it. Our rewards are thus binary, making the reward distribution $1/2$-subgaussian with variance factor $\sigma^{2}=1/4$. We follow (Cesa-Bianchi et al., 2017)’s Theorem 3 computations with a standard Gumbel noise $Gumbel(0,1)$ (see Equations 4.7 and 4.8). We do not introduce extra variable $c$ in the proof of Lemma 3, setting $L=\frac{9C^{2}\log^{2}_{+}(T\Delta_{i}^{2})}{\Delta_{i}^{2}}$. We thus bind the regret $R_{T}$ as:

[TABLE]

Here the finite horizon $T$ is the final timestep, and the gap $\Delta_{i}$ is the difference between the mean reward of the optimal item, and the mean reward of item $i$.

Although $T$ may potentially be specified, $\Delta_{i}$ is unknown. To obtain a small regret, the authors recommend setting $C=\sigma$. But one can easily see that choosing $C=\sigma/\sqrt{2}$ leads to an even smaller regret. We therefor set $C^{2}=\sigma^{2}/2=1/8$.