Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking

TL;DR

This paper introduces a method to interpret tree-based ensemble models by providing actionable feature adjustments to change negative predictions into positive ones, demonstrated on online advertising data.

Contribution

The paper presents a novel technique leveraging tree ensemble internals to recommend feature modifications for altering model predictions, enhancing interpretability and actionable insights.

Findings

Effective transformation of negative ad instances into positive ones.

Validated approach on Yahoo Gemini ad dataset.

Improved understanding of feature influence on predictions.

Abstract

Machine-learned models are often described as "black boxes". In many real-world applications however, models may have to sacrifice predictive power in favour of human-interpretability. When this is the case, feature engineering becomes a crucial task, which requires significant and time-consuming human effort. Whilst some features are inherently static, representing properties that cannot be influenced (e.g., the age of an individual), others capture characteristics that could be adjusted (e.g., the daily amount of carbohydrates taken). Nonetheless, once a model is learned from the data, each prediction it makes on new instances is irreversible - assuming every instance to be a static point located in the chosen feature space. There are many circumstances however where it is important to understand (i) why a model outputs a certain prediction on a given instance, (ii) which adjustable features of that instance should be modified, and finally (iii) how to alter such a prediction when the mutated instance is input back to the model. In this paper, we present a technique that exploits the internals of a tree-based ensemble classifier to offer recommendations for transforming true negative instances into positively predicted ones. We demonstrate the validity of our approach using an online advertising application. First, we design a Random Forest classifier that effectively separates between two types of ads: low (negative) and high (positive) quality ads (instances). Then, we introduce an algorithm that provides recommendations that aim to transform a low quality ad (negative instance) into a high quality one (positive instance). Finally, we evaluate our approach on a subset of the active inventory of a large ad network, Yahoo Gemini.

Tables

Table 1. Table 1. Set of features used to characterise ad creative ( CR ), ad landing page ( LP ), or both ( CR-LP ).

o — X[2,c,m] — X[1,c,m] — X[5,c,m] — Category	Source	Description
Language†	CR	This set of features capture the extent to which the text of the ad creative may include adult, violent, or spam content (e.g., ADULT_SCORE, HATE_SCORE, and SPAM_SCORE)
DOM	LP	This set of features are derived from the elements extracted from the HTML DOM of the ad landing page, such as the main textual content (LANDING_MAIN_TEXT_LENGTH), the total number of internal and external hyperlinks (LINKS_TOTAL_COUNT), the ratio of main text length to the total number of hyperlinks on the page (LINKS_MAIN_LENGTH_TOTAL_RATIO), etc.
Readability	CR-LP	These features range from a simple count of tokens (words) in the text of the ad creative and landing page to well-known scores for measuring the summarisability/readability of a text (e.g., READABILITY_SUMMARY_SCORE), etc.
Mobile Optimising	LP	This set of features describe the degree of mobile optimisation of the ad landing page by measuring the ability of it to be tuned to different screen sizes (VIEW_PORT), testing for the presence of a click-to-call button (CLICK_TO_CALL), etc.
Media	LP	These features refer to any media content displayed within the ad landing page, such as the number of images (NUM_IMAGES), etc.
Input	LP	This set of features represent all the possible input types available on the ad landing page, such as the number of checkboxes, drop-down menus, and radio buttons (NUM_INPUT_CHECKBOX, NUM_INPUT_DROPDOWN, NUM_INPUT_RADIO), etc.
Content & Similarity	CR-LP	These features extract the set of Wikipedia entities from the ad creative and landing page (NUM_CONCEPT_ANNOTATION), and measure the Jaccard similarity between those two sets (SIMILARITY_WIKI_IDS), etc.
History	LP	These features measure historical indicators, such as the median dwell time as computed from the last 28 days of observed ad clicks (HISTORICAL_DWELLTIME), and the bounce rate – i.e., the proportion of ad clicks whose dwell time is below 5 seconds (HISTORICAL_BOUNCE_RATE), etc.

Table 2. Table 2. Best cross validation hyperparameter settings for each learning model (“ − - ” if the hyperparameter is not considered).

Model	$s$	$\alpha$	$d$	$K$	ROC AUC
RF	$-$	$-$	$16$	$1$,$000$	0.93
GBDT	$-$	$0.1$	$-$	$100$	0.92
DT	entropy	$-$	$3$	$-$	0.84

Table 3. Table 3. Evaluation of best performing models on the test set 𝒟 test subscript 𝒟 test \mathcal{D}_{\text{test}} .

	RF	GBDT	DT
${\text{F}}_{\text{1}}$	0.84	$0.81$	$0.75$
MCC	0.66	$0.63$	$0.49$

Table 4. Table 4. The impact of tolerance threshold ϵ italic-ϵ \epsilon on ad coverage.

$ϵ$	0.01	0.05	0.10	0.50	1.00
ad coverage (%)	58.5	64.2	72.3	77.4	63.2

Equations

$$\mathbf{x^{\prime}}=\operatorname*{arg\,min}_{\mathbf{x^{*}}}\Big{\{}\delta(\mathbf{x},\mathbf{x^{*}})~{}|~{}\hat{f}(\mathbf{x})=-1\wedge\hat{f}(\mathbf{x^{*}})=+1\Big{\}}$$

$$p_{k,j}=\{(x_1⋚{\theta }_1),(x_2⋚{\theta }_2),\dots ,(x_n⋚{\theta }_n)\}$$

$$\hat{f}(\mathbf{x})=-1⟺\left(\sum _{k=1}^K{\hat{h}}_k(\mathbf{x})\right)\le 0$$

$${\hat{h}}_{k^{-}}(\mathbf{x})=-1,\forall k^{-}\in K^{-}$$

$$\mathbf{x}_{j(\epsilon)}^{+}[i]=\left\{\begin{array}[]{l l}\theta_{i}-\epsilon&\quad\text{if the $i$-th condition is $(x_{i}\leq\theta_{i})$}\\ \theta_{i}+\epsilon&\quad\text{if the $i$-th condition is $(x_{i}>\theta_{i})$}\end{array}\right.$$

$$\mathbf{x^{\prime}}=\operatorname*{arg\,min}_{\mathbf{x}_{j(\epsilon)}^{+}\in\Gamma~{}|~{}\hat{f}(\mathbf{x}_{j(\epsilon)}^{+})=+1}\Big{\{}\delta(\mathbf{x},\mathbf{x}_{j(\epsilon)}^{+})\Big{\}}$$

$$\text{helpfulness}(i)=\frac{|\text{helpful}(i)|}{|\text{helpful}(i)|+|\neg \text{helpful}(i)|}$$

Full text

In this paper, we present a technique that exploits the internals of a tree-based ensemble classifier to offer recommendations for transforming true negative instances into positively predicted ones. We demonstrate the validity of our approach using an online advertising application. First, we design a Random Forest classifier that effectively separates between two types of ads: low (negative) and high (positive) quality ads (instances). Then, we introduce an algorithm that provides recommendations that aim to transform a low quality ad (negative instance) into a high quality one (positive instance). Finally, we evaluate our approach on a subset of the active inventory of a large ad network, Yahoo Gemini.

Model interpretability; Actionable feature tweaking; Recommending feature changes; Altering model predictions; Random forest

††copyright: acmcopyright††doi: 10.1145/3097983.3098039††isbn: 978-1-4503-4887-4/17/08††conference: KDD ’17; August 13-17, 2017; Halifax, NS, Canada††journalyear: 2017††price: 15.00††ccs: Theory of computation Machine learning theory††ccs: Computing methodologies Ensemble methods††ccs: Information systems Online advertising

1. Introduction

An increasing number of organisations and governments rely on Machine Learning (ML) techniques to extract knowledge from the large volumes of data they collect every day to optimise their operational effectiveness.

ML solutions are usually considered as “black boxes”; they take some inputs and produce desired outputs, especially when their ultimate goal is prediction rather than inference. As long as ML models work properly, “everybody” is happy and little attention is devoted to understand why such surprisingly good results are obtained. Still, it is beneficial to have available techniques supporting humans in interpreting and “debugging” these models, particularly when they fail (Szegedy et al., 2013) or lead to some oddities.111http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit/

Excluding recent trends in ML such as Deep Learning (LeCun et al., 2015), typically the initial effort when designing an ML solution consists in modelling the objects of a given domain of interest, i.e., feature engineering. This step aims to describe each object in the domain using an appropriate set of properties (features), which define a so-called feature space. For a given dataset, each object can be considered as a static point located in the feature space since each feature value is deemed to be fixed; once a model is learned from the data, each prediction it makes on new objects is irreversible.

Let us assume that we disagree with a prediction that the model returns for a given object or that we would like to enforce switching such a prediction. The research question we ask in this work is how can we understand what can be changed in the feature vector in order to modify the prediction accordingly?

To better understand this challenge with an example, consider an ML application in the healthcare domain, where patients (objects) are mapped to a vector of clinical indicators (features), such as age, blood pressure, daily carbohydrates taken, etc. Assume next that an ML model has been designed to accurately predict from these features whether a patient is at risk of a heart attack or not. If for a given patient our model predicts that there is a high risk of a heart attack it would be of great advantage for medical physicians to also have a tool that suggests the most appropriate clinical treatment by offering targeted adjustments to specific indicators (e.g., reducing the daily amount of carbohydrates). In other words, to recommend the clinical treatment to switch a patient from being of high risk (negative instance) to low risk (positive instance).

In this work, we propose an algorithm for tweaking input features so as to change the output predicted by an existing machine-learned model. Our method is designed to operate on top of any tree-based ensemble binary classifier, although it can be extended to multi-class classification. Our proposed algorithm exploits the internals of the model to generate recommendations for transforming true negative instances into positively predicted ones (or vice versa).

We describe the theoretical framework along with experiments designed to validate the proposed algorithm. Our approach is then evaluated in the commercial and more implementable setting of online advertisement recommendations to illustrate the generic nature of our framework and the many and varied domains it can be applied to.

After presenting an effective Random Forest classifier that is able to separate between low and high quality advertisements (Lalmas et al., 2015), we show how our algorithm can be used to automatically generate “interpretable” and “actionable” suggestions on how to convert a low quality ad (negative instance) into a high quality one (positive instance). Such insights can be provided to advertisers who may turn them into actual changes to their ad campaigns with the aim of improving their return on investment. Finally, we assess the quality of recommendations that our algorithm generates out of a dataset of advertisements served by the Yahoo Gemini ad network.

2. Problem Statement

We start by considering the typical binary classification problem and focus on an ensemble of tree-based classifiers as an effective solution to the above problem. Additionally, we define how the internals of an existing ensemble of trees can be used to derive a feedback loop for recommending how true negative instances can be turned into positively predicted ones (or vice versa).

The approach we propose can be easily extended to the more general multi-class classification problem. We plan to present this result in future extended work.

2.1. Notation

Let $\mathcal{X}\subseteq\mathbb{R}^{n}$ be an $n$-dimensional vector space of real-valued features. Any $\mathbf{x}\in\mathcal{X}$ is an $n$-dimensional feature vector, i.e., $\mathbf{x}=(x_{1},x_{2},\ldots,x_{n})^{T}$, representing an object in the vector space $\mathcal{X}$. Suppose that each $\mathbf{x}$ is associated with a binary class label – either neg (negative) or pos (positive) – and let $\mathcal{Y}=\{-1,+1\}$ be the set encoding all such possible class labels.

We assume there exists an unknown target function $f:\mathcal{X}\longmapsto\mathcal{Y}$ that maps any feature vector to its corresponding class label. In addition, we let $\hat{f}\approx f$ which is learned from a labelled dataset of $m$ instances $\mathcal{D}=\{(\mathbf{x_{1}},y_{1}),(\mathbf{x_{2}},y_{2}),\ldots,(\mathbf{x_{m}},y_{m})\}$. More specifically, $\hat{f}$ is the estimate that best approximates $f$ on $\mathcal{D}$, according to a specific loss function $\ell$. Such a function measures the “cost” of prediction errors we would make if we replaced the true target $f$ with the estimate $\hat{f}$.

The flexibility vs. interpretability of $\hat{f}$ depends on the hypothesis space which $\hat{f}$ has been picked from by the learning algorithm. In this work, we focus on $\hat{f}$ represented as an ensemble of $K$ tree-based classifiers, $\hat{f}=\phi(\hat{h}_{1},\ldots,\hat{h}_{K})$. Each $\hat{h}_{k}:\mathcal{X}\longmapsto\mathcal{Y}$ is a base estimate, and $\phi$ is the function responsible for combining the output of all the individual base classifiers into a single prediction.

A possible implementation of $\phi$ could use a majority voting strategy. In this setting, a given instance $\mathbf{x}$ would obtain a predicted class label $\hat{f}(\mathbf{x})$ based on the result of the majority of the base classifiers; this is the mode of the base predictions. Although other strategies may be used, this does not impact our proposed approach.

2.2. Enforcing Positive Prediction

In any ensemble of tree-based classifiers, each base estimate $\hat{h}_{k}$ is encoded by a decision tree $T_{k}$, and the ensemble is represented as a forest $\mathcal{T}=\{T_{1},\ldots,T_{K}\}$.

Our aim is to identify how to transform a true negative instance into a positively predicted one. Let $\mathbf{x}\in\mathcal{X}$ be a true negative instance such that $f(\mathbf{x})=\hat{f}(\mathbf{x})=-1$. The task can now be defined as transforming the original input feature vector $\mathbf{x}$ into a new feature vector $\mathbf{x^{\prime}}$ ($\mathbf{x}\leadsto\mathbf{x^{\prime}}$) such that $\hat{f}(\mathbf{x^{\prime}})=+1$. Moreover, we accomplish an optimised form of the problem by choosing $\mathbf{x^{\prime}}$ as the best transformation among all the possible transformations $\mathbf{x^{*}}$, according to a cost function $\delta:\mathcal{X}\times\mathcal{X}\longmapsto\mathbb{R}$. This is defined as follows:

[TABLE]

The cost function measures the “effort” of transforming $\mathbf{x}$ into $\mathbf{x^{\prime}}$. A possible choice of such a function is the number of features affected by the transformation or the Euclidean distance between the original and the transformed vector.

2.3. Positive and Negative Paths

Any root-to-leaf path of a single decision tree can be interpreted as a cascade of if-then-else statements, where every internal (non-leaf) node is a boolean test on a specific feature value against a threshold. We restrict the tree decisions to be binary representations as any multiway decision can be represented in a binary form and there is little performance benefit in n-ary splits. An instance’s feature value is then evaluated at each node to determine which branch to traverse. This is repeated until the leaves are reached whereby the pos/neg classification labels are defined and assigned.

Given a forest of $K$ decision trees $\mathcal{T}=\{T_{1},\ldots,T_{K}\}$, we denote by $p_{k,j}$ the $j$-th path of the $k$-th tree $T_{k}$. We refer to $p^{+}_{k,j}$ (or $p^{-}_{k,j}$) as the $j$-th path of $T_{k}$ that leads to a leaf node labelled as pos (or neg) – a positive (or negative) path. For simplicity, we assume that each path of a decision tree contains at most $n$ non-leaf nodes, which correspond to $n$ boolean conditions, one for each distinct feature.222In general, there can be multiple boolean conditions associated with a single feature. We thus represent a root-to-leaf path as follows:

[TABLE]

Let $P^{+}_{k}=\bigcup_{j\in T_{k}}p^{+}_{k,j}$ describe the set of all positive paths, and $P^{-}_{k}=\bigcup_{j\in T_{k}}p^{-}_{k,j}$ the set of all the negative paths in $T_{k}$. Also, let $P_{k}=P^{+}_{k}\cup P^{-}_{k}$ be the set of all the paths in $T_{k}$.

We thus enumerate the possible paths in a single decision tree. Even under the assumption that each $p_{k,j}\in P_{k}$ is at most a length-$n$ path then $T_{k}$ is a depth-$n$ binary tree, whose number of leaves is therefore bounded to $2^{n}$. As the total number of leaves coincides with the total number of possible paths, we obtain $|P_{k}|\leq 2^{n}$.

In general, we cannot ensure a bound on each $|P_{k}|$, as there might exist some paths $p_{k,j}\in P_{k}$ whose length is greater than $n$. In practice though, we can specify the maximum number of paths at training time by bounding the depth of the generated trees to the number $n$ of features. Even with such a relaxed condition, the total number of possible paths encoded by the forest $\mathcal{T}$ is equal to $\sum_{k=1}^{K}|P_{k}|\leq K2^{n}$, therefore still exponential in $n$. We see later how this does not disrupt computational efficacy in practice, as our algorithm operates on a subset of those paths.

2.4. Tweaking Input Features

Given our input feature vector $\mathbf{x}$, we know from our hypothesis that $f\left(\mathbf{x}\right)=\hat{f}(\mathbf{x})=-1$. If the overall prediction is obtained using a majority voting strategy, it follows:

[TABLE]

Furthermore, there must be at least $\left\lceil\frac{K}{2}\right\rceil$ decision trees (base classifiers) of the forest $\mathcal{T}$ whose output is $-1$. That is, there exists $K^{-}\subseteq\{1,\ldots,K\}$ with $|K^{-}|\geq\left\lceil\frac{K}{2}\right\rceil$, such that:

[TABLE]

As we are operating in a binary classification setting, there must also exist $K^{+}=\{1,\ldots,K\}\setminus K^{-}$, which denote the set of classifier indices that output a positive label when input with $\mathbf{x}$, i.e., $\hat{h}_{k^{+}}(\mathbf{x})=+1,~{}\forall k^{+}\in K^{+}$.

Our goal is to tweak the original input feature vector $\mathbf{x}$ so as to adjust the prediction made by the ensemble from negative ($-1$) to positive ($+1$). We can skip all the trees indexed by $K^{+}$, as these are already encoding the (positive) prediction we ultimately want. We therefore focus on each tree $T_{k}$ where $k\in K^{-}$, and consider the set $P^{+}_{k}$ of all its positive paths. With each $p^{+}_{k,j}\in P^{+}_{k}$ we associate an instance $\mathbf{x}^{+}_{j}\in\mathcal{X}$ that satisfies that path – i.e., an instance whose adjusted feature values meet the boolean conditions encoded in $p^{+}_{k,j}$ to finish on a pos-labelled leaf, and therefore $\hat{h}_{k}(\mathbf{x}^{+}_{j})=+1$.

Among all the possibly infinite instances satisfying $p^{+}_{k,j}$, we restrict to $\mathbf{x}_{j(\epsilon)}^{+}$ to be feature value changes with a “tolerance” of at most $\epsilon$. We call it the $\epsilon$-satisfactory instance of $p^{+}_{k,j}$. We consider $p^{+}_{k,j}$ containing at most $n$ boolean conditions, as specified by Equation 1. Therefore, for any (small) fixed $\epsilon>0$, we build a feature vector $\mathbf{x}_{j(\epsilon)}^{+}$ as follows:

[TABLE]

Having a single global tolerance $\epsilon$ for all the features works as long as we standardise every feature (e.g., using z-score or min-max). If we use standard z-score for each feature, the actual magnitude of the change cannot just depend on $\epsilon$ but instead needs to be considered a multiple of a unit of standard deviation from the feature mean. Let $\theta_{i}=\frac{t_{i}-\mu_{i}}{\sigma_{i}}$ be the z-score of the threshold on the $i$-th feature value, where $t_{i}$ is the non-standardised value, $\mu_{i}$ and $\sigma_{i}$ are the mean and standard deviation of the $i$-th feature respectively.333In practice, $\mu_{i}$ and $\sigma_{i}$ are often unknown, so the sample mean and the sample standard deviation are used instead. Now, suppose that $x_{i}=\theta_{i}\pm\epsilon$ is the tweaked value of the $i$-th feature according to the ongoing transformation of the input vector $\mathbf{x}$. Therefore, $x_{i}=\frac{t_{i}-\mu_{i}}{\sigma_{i}}\pm\epsilon$. Returning to the original (i.e., non-standardised) feature scale, we obtain $x_{i}=t_{i}-\mu_{i}\pm\epsilon\sigma_{i}$. Depending on the sign, the tweaked feature is either moving closer to or farther away from the original feature mean $\mu_{i}$, pivoting around $t_{i}$.

For each $p^{+}_{k,j}\in P^{+}_{k}$ we transform our input feature vector $\mathbf{x}$ into the $\epsilon$-satisfactory instance $\mathbf{x}_{j(\epsilon)}^{+}$ that validates $p^{+}_{k,j}$. This leads us to a set of transformations $\Gamma_{k}=\bigcup_{j\in P^{+}_{k}}\mathbf{x}_{j(\epsilon)}^{+}$, associated with the $k$-th tree $T_{k}$.

Each resulting transformation in $\Gamma_{k}$ may have an impact on other trees of the forest. There might exist $l\in K^{+}$ whose corresponding tree already provides the correct prediction when this is input with $\mathbf{x}$, $\hat{h}_{l}(\mathbf{x})=+1$. It may also happen that by changing $\mathbf{x}$ into $\mathbf{x}^{\prime}$ the prediction of the $l$-th tree is incorrect and now $\hat{h}_{l}(\mathbf{x}^{\prime})=-1$. In other words, by changing $\mathbf{x}$ into another instance $\mathbf{x}^{\prime}\in\Gamma_{k}$ we are only guaranteed that the prediction of the $k$-th base classifier is correctly fixed, i.e., from $\hat{h}_{k}(\mathbf{x})=-1$ to $\hat{h}_{k}(\mathbf{x}^{\prime})=+1$. The overall prediction for $\mathbf{x}^{\prime}$ may or may not be fixed, where $\hat{f}(\mathbf{x}^{\prime})$ may still output $-1$, exactly as $\hat{f}(\mathbf{x})$ did.

If the change from $\mathbf{x}$ to $\mathbf{x}^{\prime}$ also leads to $\hat{f}(\mathbf{x}^{\prime})=+1$, then $\mathbf{x}^{\prime}$ will be a candidate transformation for $\mathbf{x}$. More formally, let $\Gamma=\bigcup_{k=1}^{K}\Gamma_{k}$ be the set of all the $\epsilon$-satisfactory transformations of the original $\mathbf{x}$ from the positive paths of all the trees in the forest. Our feature tweaking problem can then be generally defined as follows:

[TABLE]

In (Cui et al., 2015), it has already been proven that a problem similar to the one we define above is NP-hard as it reduces to DNF-MAXSAT. Our version, in fact, introduces an additional constraint ($\epsilon$) on the possible way features can be tweaked and thus it is itself NP-hard. This problem definition is still valid for the base case when $K=1$. There, the additional condition requiring $\hat{f}(\mathbf{x}_{j(\epsilon)}^{+})=+1$ is not necessary because it is implicitly true by definition. In that scenario, the ensemble is composed of a single base classifier – i.e., the forest contains a single decision tree and tweaking its prediction also results in changing the overall prediction. Note that when there is only one decision tree, our problem can be solved optimally: We can enumerate all the positive paths, choose the one with the minimum cost, and check if the threshold of tolerance $\epsilon$ is satisfied on each feature. Because base trees are interconnected through the features they share, simply enumerating positive paths does not work for an ensemble of trees since the output of a base tree may affect outputs of its sibling trees. The presence of the condition $\hat{f}(\mathbf{x}_{j(\epsilon)}^{+})=+1$ circumvents this issue by querying the model itself on the correctness of the $\epsilon$-transformation of an instance.

2.5. The Feature Tweaking Algorithm

Our approach takes as input 4 key components: (i) The trained ensemble model $\hat{f}$; (ii) A feature vector $\mathbf{x}$ that represents a true negative instance; (iii) A cost function $\delta$ measuring the “effort” required to transform the true negative instance into a positive one; and (iv) A positive threshold $\epsilon$ that bounds the tweaking of each single feature to pass every boolean test on a positive path of each tree. The result being the transformation $\mathbf{x^{\prime}}$ of the original $\mathbf{x}$ that exhibits the minimum cost according to $\delta$. The detailed description is presented in Algorithm 1.

In the worst case, our algorithm examines all $K$ trees of the forest, although it investigates only those trees whose base predictions are negatives (neg-labelled). Then, all the positive paths of each tree are considered, and for each of those a potential candidate $\epsilon$-transformation is built according to the scheme proposed in Equation 2. As such, the number of steps depends on the number of positive paths on each tree, which in turn is related to the number of leaves.

We stated in Section 2.3 that we cannot provide apriori any limit to the depth of a decision tree, and therefore to its number of leaves. However, in practice these can be bounded whilst training the model. We therefore set the maximum length of each root-to-leaf path, i.e., the depth of each tree, to be at most equal to the number of input features $n$. Considering our goal, this is not a limitation as each transformation should at most affect each feature exactly once. The total number of positive paths to be examined will then be limited by $K2^{n}$, and so the worst case complexity is $O(2^{n})$. It follows that our method might be unsuitable for dealing with high-dimensional datasets, such as text, images, or videos. In reality such a setting would not really make sense when transforming input instances in the original feature space (e.g., change a few words on a document or a few pixels of an image).

Although the search space can be exponential in the number of features, in practice the number of positive paths of each tree is significantly smaller. This makes our method feasible on average input sizes (i.e., below 100 features). For example, in our experiments we found that the maximum depth of each tree leading to the best classification results is significantly smaller than the total number $n=45$ of features (see Table 2). In addition, many positive paths may share several boolean conditions, especially when extracted from the same decision tree. This allows us to avoid tweaking the same input feature multiple times according to the same condition by using some caching mechanism.

Finally, our algorithm can be easily parallelised since each tree can be explored independently from the others for any given instance –i.e., we can adjust the $k$-th tree while keeping the remaining $K\setminus k$ trees simultaneously fixed for all $k\in\{1,\ldots,K\}$.

3. Use Case: Improving Ad Quality

We demonstrate the utility of our method when applied to a real-world use case in online advertising. We investigate how our algorithm can be used to improve the quality of advertisements served by the Yahoo Gemini ad network.

3.1. Why Ad Quality?

A main source of monetisation for online web services comes in the form of advertisements (ads for short) impressed in dedicated real estate units of rendered web pages. Online publishers operating these web services typically reserve predefined slots within their streams, utilising a third party ad network to deliver ad inventory to impress within them. Ad networks free publishers from running their own ad servers as they decide for them which ads should be placed at which slots, when and to whom. Advertisers rely on ad networks to optimise their return on investment – for example through targeting the right audience according to the advertiser budget and marketing strategy.

A trustworthy relationship between advertisers, publishers and ad networks is instrumental to the success of online advertising. On the advertiser side, ad networks provide advertisers with tools for monitoring key performance indicators (e.g., number of ad impressions, click-through rate or CTR, bounce rate) as well as effective mechanisms to overcome fraudulent activities such as click spam (Daswani and Stoppelman, 2007; Stone-Gross et al., 2011). On the publisher side, ad networks provide mechanisms that allow users to hide ads they dislike and indicate the reasons for doing so. This information can be used by ad networks to ensure that the ads they serve do not negatively affect user engagement on a publisher website, as well as reporting to advertisers on the quality of their ads.

As with any self served content delivery platform, an ad network has a varying distribution of quality with many ads being of low quality. Not serving them may not be an option when supply is exhausted. Therefore, another approach to positively shift the inventory quality distribution is to leverage the interpretability of the internal machinery of existing ad quality prediction models – i.e., binary classifiers – so as to offer actionable recommendations. The intention of these programmatically-computed recommendations is to provide advertisers with guidance on how they can improve their ad quality at scale. Such a system yields value for all beneficiaries in this advertising ecosystem ultimately culminating with a better user experience.

Returning to our work, by applying our feature tweaking algorithm introduced in Section 2.5, we show in the rest of this paper how we can transform a low quality ad into a set of new “proposed” high quality ads by shifting their position in an ad quality feature space. The algorithm employs the internals of a learned binary classifier to tweak the feature-based representation of a low quality ad so that the new “proposed” ads are promoted to high quality ones when re-input to the classifier. Each transformation is associated with a cost, allowing us to generate actionable suggestions from the “proposed” instances with the least cost, so as to improve low quality ads. We validate our approach on a dataset of mobile native ads served by the Yahoo Gemini444https://gemini.yahoo.com ad network.

3.2. A Definition of Ad Quality

Many factors can affect the quality of an ad: its relevance, i.e., whether the ad matches the user interest (Raghavan and Hillard, 2009); the pre-click experience, i.e., whether the ad annoys a user (Zhou et al., 2016); and finally the post-click experience, i.e., whether the ad landing page555We refer to ad landing page as the web page of the advertiser that a user is redirected to after clicking on an ad. meets the user click intent that brought them to the landing page (Lalmas et al., 2015). We focus on the latter, the post-click experience, following from  (Barbieri et al., 2016; Lalmas et al., 2015).

Inspired by these studies, we define and measure the quality of ads using the time spent on their landing pages as a proxy, referred to as dwell time. We know from (Lalmas et al., 2015) that ad landing pages exhibiting long dwell times promote a positive long-term post-click experience. Based on this definition of ad quality, we design a binary classifier that effectively separates between low and high quality ads, i.e., ad landing pages whose dwell time is below or above a threshold $\tau$, respectively. We compute $\tau$ as the median of all the sample means of dwell times observed for a large set of ad landing pages. Intuitively, an ad landing page is of high quality if its average dwell time is greater than $\tau$ – i.e., if the average time users spent on the page is greater than the average time users spent on at least 50% of any other ad landing page. Although more sophisticated approaches can be designed (Barbieri et al., 2016), this is not the main goal of this research, and we leave it for future work.

3.3. Predicting Ad Quality

To apply our algorithm to our use case, we first need to learn a binary classifier that predicts whether an ad is of high quality or not, given a feature-based representation of the ad creative and the landing page.

3.3.1. Ad Feature Engineering

A sample of the set of features used in this work is listed in Table 3.3.1. This is based on the same set used in (Barbieri et al., 2016), with the additional “Language” category (marked with “$\dagger$” in the Table). Due to space constraints we do not show all the features, and invite the reader to refer to (Barbieri et al., 2016) for a complete description of them.

Each feature in the table is associated with a category and a source. The former indicates the type of features whilst the latter specifies whether the feature is computed from the ad landing page (LP), the ad creative (CR), or a combination of the two (CR-LP). Although our focus is on the post-click experience (i.e., ad landing page), we aim to obtain the best performing model for predicting the quality of the ads, and hence include both pre-click (i.e., ad creative) and historical features; the latter not participating in the tweaking process as – by definition – they cannot be altered.