Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

TL;DR

Octopus is an AI framework that optimally balances quality, cost, and time in crowdsourcing by using a hierarchical POMDP approach, outperforming existing methods in real-world experiments.

Contribution

It introduces a novel hierarchical POMDP-based framework to jointly optimize quality, cost, and time in crowdsourcing, addressing a previously intractable multi-objective problem.

Findings

Octopus outperforms state-of-the-art approaches in experiments.

It effectively manages crowdsourcing tasks in real-world settings.

Demonstrates practical deployment on Amazon Mechanical Turk.

Abstract

We present Octopus, an AI agent to jointly balance three conflicting task objectives on a micro-crowdsourcing marketplace - the quality of work, total cost incurred, and time to completion. Previous control agents have mostly focused on cost-quality, or cost-time tradeoffs, but not on directly controlling all three in concert. A naive formulation of three-objective optimization is intractable; Octopus takes a hierarchical POMDP approach, with three different components responsible for setting the pay per task, selecting the next task, and controlling task-level quality. We demonstrate that Octopus significantly outperforms existing state-of-the-art approaches on real experiments. We also deploy Octopus on Amazon Mechanical Turk, showing its ability to manage tasks in a real-world dynamic setting.

Tables

Table 1. Table 1: Notation used in the paper.

Symbol	Meaning
$𝒰$	Overall utility (requester defined function)
$U_q$	Per-task utility
$c$	Pay per ballot
$C$	Total cost incurred for all tasks
$C_q$	Total cost incurred for task $q$
${\pi }_q$	Policy for task $q$’s QualityManager
$\gamma$, $\overline{\gamma }$	Worker error parameter, average error
$d_q$	Difficulty of task $q$
$t_q$	True binary answer for task $q$
${𝐛}_{𝐪}$	Belief state for task $q$
$v_q$	Probability of POMDP’s most likely true answer for task $q$
${\nu }_q$	Task quality (${\nu }_q=2v_q-1$)
$\overline{\nu }$	Batch quality ($\overline{\nu }=\frac{1}{n}{\sum }_q{\nu }_q$)
${\theta }_q$	Estimated number of ballots required to complete task $q$
$\theta$	Estimated number of ballots required for completing the batch of tasks ($\theta ={\sum }_q{\theta }_q$)
$\tilde{\theta }({\nu }_q,c)$	Mapping from $({\nu }_q,c)$ to ${\theta }_q$ for task $q$ under $\pi$
${\varphi }_q$	Priority of task $q$
${\tau }_{max}$	Deadline
$\tau$	Current Time
$\Delta$	Granularity (e.g. ${\Delta }_{\tau }$)

Table 2. Table 2: Online performance comparison of Octopus  and Gao ( r = 1 𝑟 1 r=1 ) for a single trial. 𝒰 𝒰 {\cal U} is shown in thousands.

Method	Octopus			Gao $\left(r=1\right)$
Deadline	Cost	Acc.	$𝒰$	Cost	Acc.	$𝒰$
1 hr	$2.63	74.4%	-25.9	$2.68	71.4%	-28.9
2 hrs	$3.17	77.2%	-23.1	$1.72	73.8%	-26.4
4 hrs	$6.23	77.4%	-23.2	$0.50	74.4%	-25.6

Full text

Introduction

Task control of workflows over micro-task crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), has received significant attention in AI literature (?). Typically, a requester needs to balance three competing objectives – (1) total cost, owing to payments made to workers for their responses (or ballots), (2) overall quality, usually evaluated as accuracy of the final output, and (3) the total time for completing the task. These criteria are inter-related: increasing the pay per task attracts more workers to the task, thereby reducing completion time. However, it also exhausts the budget sooner, so requesters can afford fewer ballots per task, likely reducing the overall quality.

Most prior work on crowd controllers has focused on the tradeoff between cost (or no. of ballots) and quality (?; ?; ?; ?; ?). A common approach is to define a Partially Observable Markov Decision Process (POMDP) per task, which decides on whether to get another ballot or submit the best answer for that task. However, this work is time-agnostic, and assumes that pay per ballot is given as input.

Recent work has also studied the tradeoff between cost and completion time for a batch of tasks (?). They model the problem as a Markov Decision Process (MDP) that changes the pay per ballot, so that all ballots can be obtained by the given deadline in a cost-efficient manner. However, this work assumes that the number of ballots needed to complete the whole batch is a constant known to the requester in advance.

There is limited research on simultaneously addressing tradeoffs between cost, quality and latency. We know of only one work that studies this for the specific workflow for finding max of a set of items (?). This work assumes that latency is pay-independent – an assumption well-known to be incorrect (?; ?). Under a fixed latency-per-response assumption, they speed up the task and save cost by taking fewer responses. Our three-way optimization is for the broader case of answering a batch of tasks, and uses variable pricing to alter the latency of task completion, in line with crowdsourced marketplace dynamics.

Building upon these strands of research, we present Octopus, an AI agent that can balance all three objectives (cost, quality, time) in concert on real crowdsourced marketplaces, by optimizing a requester-specified, joint utility function for a batch of tasks. It achieves this by controlling both the pay per ballot and the (predicted) accuracy of each individual task.

We could model the whole problem as a single POMDP, however, that is unlikely to scale. An alternative could be to use multi-objective MDPs, but they are also less tractable, because they produce a pareto-optimal set of solution policies (?). Octopus uses a three-component architecture – one to set the pay per ballot (CostSetter), another to choose the next available task (TaskSelector) and a third to control each task’s quality (QualityManager). A key technical novelty is in the careful modeling of the CostSetter’s state space in order to circumvent intractability – the state space contains aggregate statistics regarding completion levels of all tasks, so that it can decide the next best pay to set.

We perform extensive experiments using both simulated and real data, as well as online experiments on AMT. Since no existing system performs direct 3-way optimization in crowdsourced marketplaces, our experiments compare against existing state-of-the art approaches that optimize 2 of the 3 objectives. We find that in most settings, Octopus simultaneously outperforms, or is at par with multiple variants of these baselines. Our contributions are:

We describe Octopus, a novel framework to address cost-quality-time optimization for a batch of tasks in a crowdsourced marketplace setting. It contains three components that set the pay per ballot, select the next task and control each task’s quality. A key technical novelty is the use of aggregate statistics of all tasks in the state space design for the CostSetter, ensuring tractability for real-time deployment. 2. 2.

Octopus consistently performs at par with or better than state-of-the-art baselines, yielding up to 37% reward improvements on real data. 3. 3.

We deploy Octopus on AMT and demonstrate that it is able to optimize utility effectively in a live, online experiment.111Code can be found at https://github.com/krandiash/octopus.

Related Work

Cost-Quality Optimization. There is significant work on getting more quality out of a fixed budget. One branch of this research focuses on collective classification, which develops aggregation mechanisms to infer the best output per task, given a static set of ballots (?; ?; ?; ?). The other branch studies intelligent control, which dynamically decides whether to ask for a new ballot on a task, or stop and submit the answer. These include control of binary or multiple choice tasks (?; ?; ?), multi-label tasks (?), and tasks beyond multiple choice answers (?; ?). All these works design agents to control a single task and assume a constant pay per ballot. Our work closely follows the POMDP formulation laid down in Dai et al. (?) for binary tasks.

Cost-Time Optimization. Increasing pay per ballot can reduce completion times. Faradani et al. (?) develop models to find upfront, the static price per ballot so that a desired deadline can be met. Gao & Parameswaran (?) extend this by varying pay at discrete time-steps using an MDP. Both approaches assume a fixed number of ballots known a-priori, without dynamic quality control of tasks. There is also some work on price-independent latency reduction (?).

Cost-Quality-Time Optimization. There is limited work in this area. The only paper we are aware of is Venetis et al. (?), which addresses cost-quality-time optimization but in a restrictive setting with important distinctions from our work: (i) they look at max-finding for a set of items, while our task-type is classification; (ii) they consider latency to be pay-independent and fixed per task, while we study the more realistic setting in which changing pay directly impacts workers’ desire to work on our tasks; (iii) unlike us, they don’t change pay per task directly, and instead, change the number of responses sought per task to control both cost and latency. Qualitatively, our work thus also highlights how workers perceive pay changes in a crowdsourcing marketplace and its overall effect on task completion.

Worker Retention. Previous work deals with incentivizing workers to perform more tasks via bonuses or diversification (?; ?; ?; ?). Recently, Kobren et al. (?) model the process of worker retention. We present empirical results that suggest worker retention plays a dominant role in determining task completion rates.

Task Routing. Prior work deals with two issues; deciding which task from the batch to solve next, or which worker to route a task to. Ambati et al. (?) rank tasks based on user preferences using a max-entropy classifier. Other work uses low-rank matrix approximations (?) for equal difficulty tasks. Rajpal et al. (?) decide which worker pool to route a task to. Other papers study task routing on volunteer platforms (?; ?).

Like AMT, we assume no control on which worker picks a ballot job, but we select the best next task to assign to an incoming worker. Following (?; ?), we assume that worker quality is independent of the pay per ballot. We re-verify this for our data in our experiments.

Decentralized Approaches. There is related work in decentralized Wald stopping problems (?) which considers how to optimize a common utility function given a set of agents who each make independent observations. However, these approaches do not scale well with the number of agents (tasks in our setting), which can be quite large. There is also work in decentralized metareasoning (?) to decide when to stop optimizing a utility function. Metareasoning approaches typically assume that utility is monotonically increasing over time, which is not true in our setting since for instance, conflicting ballots on a single task would decrease utility.

Problem Definition

A requester provides a batch of $n$ binary tasks $q\in 1\dots n$, each having a $0/1$ response. They also provide a utility function ${\cal U}$, which describes how to tradeoff cost, time and quality. The agent can dynamically change pay per ballot $c$, and choose a variable number of ballots per task to optimize the final objective. We study the setting where ${\cal U}$ is expressed as a sum of task-level utilities ($U$) minus cost, i.e. ${\cal U}=\sum_{q=1}^{n}(U_{q}-C_{q})$. Here, $C_{q}$ is total money spent on $q$. We assume that answers to all tasks are to be returned to the requester as one single batch.

MDP/POMDP background. An MDP models the long-term reward optimization problem under full observability and is defined by a five tuple $\langle S,A,T,R,\gamma\rangle$. Here, $S$ is a set of states, $A$ a set of actions, $T(s^{\prime}|s,a)$ denotes the probability of transitioning to state $s^{\prime}$ after taking action $a$ in state $s$, and $R(s,a)$ maps a state-action pair to a real-valued reward. $\gamma$ is the discount factor for making infinite-horizon MDPs well formed. A POMDP extends an MDP into a partially-observable setting, where the state is not fully observable and only a belief (probability distribution) over possible states can be maintained using observations from the model. A POMDP is represented as $\langle S,A,T,R,O,\gamma\rangle$ tuple, where a new function $O(o|s^{\prime},a)$ denotes the distribution over observations on taking an action $a$ and arriving in a new state $s^{\prime}$. Lack of space precludes a long discussion of the subject – there are existing solvers for solving MDPs and POMDPs of reasonable sizes, e.g. (?), which we use in our work.

To formulate the problem optimally, we would need to define a single, centralized POMDP over a state containing answers and difficulty estimates of all individual tasks as well as the current pay per ballot and the current time. The actions will include requesting a ballot on a task $q$, changing the pay, and a terminal submit action. Solving this POMDP would yield the optimal policy, which would decide which task to get ballots on next, when to change pay and when to submit. Since the number of tasks in a batch can be huge, this POMDP is unlikely to scale due to a large state and action space. Naive extensions to Dai et al.’s or Gao et al.’s state-of-the-art models for cost-quality and cost-time optimization respectively are not possible either – Dai et al.’s model is solved per task, whereas pay must be set based on progress of the whole batch of tasks; Gao et al.’s model assumes a fixed number of ballots per task, and has no natural way to optimize quality by taking a variable number of ballots based on each task’s difficulty.

Octopus for Three-Way Optimization

We propose a three-component architecture (see Figure 1). In Octopus  each task has its own QualityManager that decides, based on the current pay, whether it is worth taking another ballot for this task (light edge) or not (dark edge). This information is conveyed to the TaskSelector, which selects an available (light) task to route to an incoming worker. Based on the current progress of the whole batch, the CostSetter decides what pay per ballot to set; this action is taken at periodic intervals. We design Octopus such that it can allocate tasks as and when workers arrive and works instantaneously in practice, therefore utilizing crowdsourcing marketplaces with full parallelism.

QualityManager

Background. QualityManagers follow the worker response model and POMDP formulation of Dai et al. (?). Each worker is assumed to have an error parameter $\gamma\in(0,\infty)$ ($\gamma=0$ is error-free). An average worker has error parameter $\bar{\gamma}$. Each task $q$ has an unknown true Boolean answer $t_{q}$, and an associated difficulty $d_{q}\in[0,1]$ ($d_{q}=0$ is easy) – these are estimated using an Expectation-Maximization algorithm (?) as data is received. Each task has a prior difficulty distribution $p(d_{q})$. A worker’s ballot for $q$ depends on their $\gamma$, $t_{q}$ and $d_{q}$.

In Dai et al. (?) the POMDP for $q$ maintains a belief state $\mathbf{b}_{q}$ over $(d_{q},t_{q})$ state tuples. For instance $\mathbf{b}_{q}(0.5,1)=0.4$ indicates a 40% belief that $d_{q}=0.5$ and the answer to $q$ is $1$. The POMDP has two actions; (1) request another ballot, or (2) mark $q$ as completed. The POMDP policy $\pi_{q}$ maps every belief state to an action. Each time the POMDP receives another ballot, a Bayesian update is performed to re-estimate a new belief. For Octopus, the POMDP optimizes $U_{q}-C_{q}$. We note that $\pi_{q}$ depends on the pay per ballot $c$: if $c$ is smaller, the POMDP can afford more ballots. Dai et al.’s original model keeps $c$ constant, but in our case the CostSetter can change it, triggering a change in $\pi_{q}$.

We define $v_{q}$ as the confidence in the most probable answer for task $q$; $v_{q}=\mathrm{max}(v_{q}^{0},v_{q}^{1})\in[0.5,1]$, where $v_{q}^{0},v_{q}^{1}$ are the current probabilities that $q$’s answer is 0 or 1 respectively. $v_{q}$ can be computed from $\mathbf{b}_{q}$ by summing out $d_{q}$.

Computation of Aggregate Statistics. We now define two batch-level statistics, which will be a key part of the state space representation for the CostSetter.

We first define a normalized estimate of task quality, $\nu_{q}=2v_{q}-1$; $\nu_{q}$ normalizes $v_{q}$ so that it lies in $[0,1]$. A high value of $\nu_{q}$ (near 1) indicates the POMDP’s high confidence in its estimated answer for $q$, and vice versa for a low (near 0) value. We also define a related notion of batch quality, $\bar{\nu}=\frac{1}{n}\sum_{q=1}^{n}\nu_{q}$, which is an aggregate statistic estimating the current quality for the entire batch of $n$ tasks. Finally, we construct the batch quality histogram – a histogram built by binning tasks into equally sized bins based on their $\nu_{q}$ values. The bin width is denoted as $\Delta_{\nu}$.

The CostSetter also needs an estimate of the number of ballots remaining. We define $\theta_{q}(\pi_{q})$ as an estimate of the expected number of ballots that will be needed (starting from the current time) until task $q$ will be marked completed. For notational ease we write $\theta_{q}$ to denote $\theta_{q}(\pi_{q})$. Recall that $\pi_{q}$ can change with a change in $c$. Hence $\theta_{q}$ also depends on $c$.

How can we compute $\theta_{q}$? We use a trajectory-tree approach (similar to (?)) called FrontierFinding. We construct a binary tree of future observations, rooted at the current time step, where each edge corresponds to an observation (a worker response of $0/1$). The node below any edge contains the belief state generated by updating the POMDP, using the observation associated with that edge. Each trajectory is a path from the root to a leaf, and is generated with an associated path probability (using Dai et al.’s generative model assuming an average worker). A leaf is created whenever the policy takes the ‘mark as completed’ action, or when the path probability drops below a threshold. $\theta_{q}$ is simply the expected length of a trajectory in this tree.

We also estimate the expected ballots to completion for the batch: $\theta=\sum_{q=1}^{n}\theta_{q}$. $\theta$’s role is similar to that of $\bar{\nu}$ – it is an aggregate statistic that describes how far the batch is from completion. It also helps in quantifying the expected cost of completion: if $\theta=1000$ and $c=3$, we would expect to spend $c\cdot\theta=3000$ units of money to complete the batch.

In summary, we described the design of a per-task QualityManager. Collectively, $n$ of these help us in estimating two aggregated quantities, $\bar{\nu}$ and $\theta$, which measure the overall quality, and degree of completion of the batch, respectively. All notation for this and future sections is summarized in Table 1.

TaskSelector

The TaskSelector decides which incomplete task to assign to the next incoming worker. It must have an ‘anytime’ behavior, i.e. it must increase utility ${\cal U}$ quickly. This is because the time of final submission is not in its control, and the batch might be submitted at any time by the CostSetter.

To be prepared for any contingency, the TaskSelector uses a 1-step greedy policy over expected utility gain. We define each task’s priority ($\phi_{q}$) as the difference between the current utility ($U_{q}$) of $q$ and the expected utility after receiving 1 ballot ($U^{\prime}_{q}$) from an average worker (error rate $\bar{\gamma}$), given the current belief state $\mathbf{b}_{q}$ of $q$’s QualityManager. Thus, $\phi_{q}=\mathbf{E}[U_{q}^{\prime}|\bar{\gamma},\mathbf{b}_{q}]-U_{q}$. TaskSelector assigns the task with the maximum $\phi_{q}$ value to the next available worker.

Unfortunately, $U_{q}$ (and therefore ${\cal U}$) is neither monotonic (conflicting ballots decrease utility) nor submodular (a skilled worker could arrive after an error-prone one), so we cannot utilize prior work on adaptive submodularity (?; ?) to guarantee solution quality. Providing quality bounds is left for future work.

Lastly, note that the task allocation process is instantaneous, as well as completely parallelized, since we don’t wait for a task to be returned before allocating another task. Given enough workers, we could get ballots on every single task in parallel. This is important, since it allows us to take full advantage of micro-task marketplaces.

CostSetter

The CostSetter is an MDP that changes $c$ (pay per ballot) in order to maximize ${\cal U}$. It uses information about the completion level of each task to assess whether the batch of tasks is completing on schedule or needs to be sped up or slowed down. To influence the rate of completion of the batch, it sets $c$ at discrete time steps $\tau\in\{0,\Delta_{\tau},2\Delta_{\tau},\dots\}$.

The key challenge for the CostSetter is in defining the state space. Ideally, as stated earlier, each task’s belief $\mathbf{b}_{q}$ should be part of the state, but that would make computations intractable. Instead, we approximate by using aggregate statistics over the whole batch of tasks. We describe the state space, actions, transition functions, and rewards of this MDP below.

State Space. The choice of the best pay per ballot $c$ depends on its current value, the current time, and the aggregate degree of completion of the batch. We choose the state to be a 4-tuple $(\bar{\nu},\theta,\tau,c)$. Both $\bar{\nu}$ and $\theta$ are important for this decision; $\bar{\nu}$ estimates the expected accuracy on the batch, while $\theta$ gives us an idea of how much more improvement in $\bar{\nu}$ is possible (at the current $c$). For a fixed $\bar{\nu}$ and $c$, a high $\theta$ would indicate the presence of several unsolved tasks and the possibility of improving $\bar{\nu}$. On the other hand, a low $\theta$ would indicate that most tasks are solved and there is little improvement possible. $\theta$ therefore captures the spread of the distribution of task qualities $\nu_{q}$, while $\bar{\nu}$ is the mean of this distribution. We use this intuition later to construct the transition function for the CostSetter.

As another example consider a case where both $\theta$ and $\bar{\nu}$ are high. This indicates that the QualityManagers consider there be to scope for quality improvement despite the batch quality being high already, possibly due to very low $c$. If we did not have $\theta$ in the state, we would instead base our decision on the high $\bar{\nu}$ value, and believe that further improvement in utility was not possible.

All state variables are continuous, and for tractability we discretize them. $\theta$ is discretized with a granularity $\Delta_{\theta}$, $\bar{\nu}$ with a granularity $\Delta_{\nu}$, and $\tau$ with a granularity $\Delta_{\tau}$. We assume that $c$ can take values $\{c_{1},c_{2}\dots,c_{k}\}$. These values are defined by the requester, and in practice would respect marketplace constraints, such as minimum wages. Interestingly, the model is robust in that if a requester provides a very poor starting wage, workers will likely not pick up the task, and the model will subsequently respond by increasing the wage.

Actions and Rewards. Every state has access to two pay-change actions: $\uparrow$ and $\downarrow$. $\uparrow$ increases $c_{i}$ to $c_{i+1}$ while $\downarrow$ does the opposite. The $\uparrow$ and $\downarrow$ actions incur no cost to the system. However, in practice we assign a small cost to these actions to prevent frequent cyclical pay-changes in a policy. Changing $c$ has no impact on $\tau$ and $\bar{\nu}$, but it does change $\theta$, since the number of ballots remaining per task depends on the current pay per ballot (via the policy $\pi_{q}$). We discuss how to compute this when defining the transition function.

We also have a no-change action, which increments $\tau$ by $\Delta_{\tau}$, along with asking workers for more ballots at $c$, which remains unchanged (for the next $\Delta_{\tau}$ duration). This is essentially a marketplace action, where we post tasks to the marketplace with a pay per ballot equaling $c$. At the end of $\Delta_{\tau}$ minutes, we would then arrive in a new state. The cost of this transition (to the nearest $\Delta_{\theta}$ value) is just the amount paid to workers during this duration on the marketplace, equaling the number of ballots received during this time, multiplied by $c$.

The final action is a ‘terminate’ action that submits all answers to the requester. Its reward should be ${\cal U}$ based on batch quality and current time (cost is not needed, since that was already accounted in the no-change action). Unfortunately, the MDP has access to only the aggregate statistics, and not the full batch quality histogram, which is needed for computing ${\cal U}$. We now describe a novel $\beta$-reconstruction procedure that allows us to extrapolate the full histogram from aggregate statistics, useful for computing this reward as well as the transition function.

$\mathbf{\beta}$-Reconstruction. The goal is to reconstruct an approximate batch quality histogram given the aggregate statistics, $\bar{\nu}$ and $\theta$, and the current $c$. The procedure assumes that the histogram can be approximated with a two parameter Beta distribution, $\beta_{\lambda_{1},\lambda_{2}}$. Also assume that we are provided a function $\tilde{\theta}$ that maps a $(\nu_{q},c)$ pair to $\theta_{q}$; it returns the expected number of ballots needed for a task $q$ given its current quality and pay per ballot. Note also that $\tilde{\theta}$ will be a non-increasing function of $\nu$. We now show that we can find suitable $\lambda_{1}$ and $\lambda_{2}$ given $\bar{\nu}$, $\theta$ and $\tilde{\theta}$.

To compute the best fit $\lambda_{1},\lambda_{2}$ values, we solve two equations. The $1^{\mathrm{st}}$ equation enforces that the mean of the reconstructed distribution is $\bar{\nu}$: $\frac{\lambda_{1}}{\lambda_{1}+\lambda_{2}}=\bar{\nu}$.

We can reparameterize $\beta_{\lambda_{1},\lambda_{2}}$ using $\lambda_{1}=\lambda\bar{\nu}$; $\lambda_{2}=\lambda(1-\bar{\nu})$ and write it as $\beta_{\lambda}(\bar{\nu})$. Our task now reduces to finding the best fit $\lambda$. A $2^{\mathrm{nd}}$ equation imposes $\theta$ as an expectation over the batch quality distribution: $\hat{\theta}(\lambda)=n\int_{0}^{1}\tilde{\theta}(\nu,c)\beta_{\lambda}(\nu)d\nu\approx\theta$.

Since $\tilde{\theta}$ is computed using a POMDP policy, it will rarely be available in closed form. The integral above is approximated using a numerical algorithm. The best $\lambda$ is found via $\mathrm{argmin}_{\lambda}|\hat{\theta}(\lambda)-\theta|$ using a linear search, and works instantaneously in practice. Having found a suitable $\lambda$ value, we now bin all $n$ tasks into the $\beta$ distribution to recover the task quality histogram, as desired.

Finally, we describe the procedure for estimation of $\tilde{\theta}(\nu_{q},c)$. First we calculate the corresponding $v_{q}=0.5(\nu_{q}+1)$. Intuitively, $\tilde{\theta}(\nu_{q},c)$ assesses the number of ballots taken by the QualityManager when its belief $\mathbf{b}_{q}$ in the current answer is $v_{q}$ and it has difficulty $d_{q}$. However, we haven’t reconstructed $d_{q}$ – we use the prior distribution $p(d)$ as its belief on $d_{q}$. To compute $\tilde{\theta}(\nu_{q},c)$ we initiate the QualityManager’s POMDP from such a belief state, and compute $\theta_{q}$ under policy $\pi_{q}$ using FrontierFinding.

$\uparrow/\downarrow$** Transitions.** We now describe the transitions for $\uparrow$ action in a given state $(\bar{\nu},\theta,\tau,c_{i})$ to reach $(\bar{\nu},\theta^{\prime},\tau,c_{i+1})$. As we change $c$, the main change is in $\theta$. We quantify this change by a simple observation: regardless of pay, the number of ballots taken until this point is fixed. Suppose that at $\tau=0$ we compute the estimated number of ballots for the batch as $\theta_{0}(c_{i})$. If we have taken $x$ ballots till now, our current estimate of $\theta$ will be simply $\theta_{0}(c_{i})-x$. At pay $c_{i+1}$, our next state’s estimate should be $\theta_{0}(c_{i+1})-x$. Thus, when changing pay from $c_{i}$ to $c_{i+1}$ we can simply add $\theta_{0}(c_{i+1})-\theta_{0}(c_{i})$ to compute $\theta^{\prime}$. A similar analysis works for the $\downarrow$ action.

No-Change Transitions. The no-change action emulates the setting that the tasks are posted on the platform for $\Delta_{\tau}$ time at pay $c$. Let the next state be $(\bar{\nu}^{\prime},\theta^{\prime},\tau+\Delta_{\tau},c)$. Estimation of $\bar{\nu}^{\prime}$ and $\theta^{\prime}$ requires a model of task completion. Similar to Gao & Parameswaran (?), we maintain a pay-dependent ballot completion model, $\mathrm{Pr}(n_{b}|\Delta_{\tau},c)$, as the probability that Octopus will receive $n_{b}$ ballots in duration $\Delta_{\tau}$ at pay $c$. However, different from their work, the probability model combines the effects of worker arrival, retention and time taken per task (and not just arrival). Thus, $\theta$ will reduce by $n_{b}$ with probability $\mathrm{Pr}(n_{b}|\Delta_{\tau},c)$. The cost of the transition will be $-c\cdot n_{b}$. Since $n_{b}$ is discretized upto $\Delta_{\theta}$ granularity, the cost will be rounded off to the nearest bucket.

For updating $\bar{\nu}$, we $\beta$-reconstruct the batch quality histogram using the current state. We then bin $n$ tasks into this histogram, and simulate the TaskSelector on this reconstructed batch. To do this, we first create a POMDP belief state for each reconstructed task by choosing $\nu$ from the histogram to recover $v$, and using the prior difficulty distribution. We then select a task, simulate a ballot using an average worker ($\bar{\gamma}$) and compute the posterior belief. We continue until all $n_{b}$ ballots are used up. At the end we compute the $\bar{\nu}^{\prime}$ based on the updated state of the batch. For robustness, we repeat this entire procedure multiple times and average the $\bar{\nu}$ values from different runs.

Implementation Details. We construct the whole MDP with all transitions and rewards using simulations and $\beta$-reconstructions as described above. Since time can be unbounded, we keep a max time $\tau_{\max}$ when defining the total state space. We also recognize that the transition from $\bar{\nu}$ to $\bar{\nu}^{\prime}$ depends on $n_{b}$ but not on any other part of the state. By caching a table of $(\bar{\nu},n_{b},\bar{\nu}^{\prime})$ values once, we can save on a lot of simulations when computing the transition function of the no-change action in various states.

We use Value Iteration to learn the policy with $(\bar{\nu}=0,\theta=n\cdot\max\tilde{\theta}(\nu=0,c=c_{1}),\tau=0,c=c_{1})$ as our start state. Intuitively, we are starting at 0 quality, the maximum possible number of ballots to completion, and the lowest price.

In a real execution environment, it is possible that over time, the CostSetter’s aggregates diverge from the set of QualityManager’s beliefs in the batch. To improve performance, we synchronize the CostSetter’s $\bar{\nu}$ and $\theta$ to those of the real batch after every time interval. Experimentally Octopus performs well even without synchronization, indicating that the learned transitions are effective.

In summary, we described the CostSetter, an MDP that keeps track of the aggregate statistics $\bar{\nu}$ and $\theta$ for a batch of $n$ tasks, and changes pay to optimize utility ${\cal U}$. A key contribution is a novel $\beta$-reconstruction procedure that approximates the batch quality histogram for a state.

Experiments

We conduct three sets of experiments: (i) experiments on simulated data; (ii) offline evaluation on real data; (iii) live experiments on Amazon MTurk (AMT).

Model Parameters. For experiments, we initialize Octopus with a hard deadline utility function: $U_{q}=-\infty$ if $\tau<\tau_{\max}$; otherwise, $U_{q}=-{\cal P}$ for every incorrect answer and zero for a correct answer. This joint utility combines the utilities from Dai et al.’s and Gao et al.’s models. Here, penalty ${\cal P}$ represents how important quality is to the requester. Notice that since a POMDP doesn’t know whether it is submitting the correct answer, it cannot compute the utility exactly. It uses its belief to estimate expected utility as $-{\cal P}(1-v_{q})$. This linear utility makes the reward computations for CostSetter’s ‘terminate’ action simple. Given the $\bar{\nu}$ of the state, the reward is calculated as $-0.5{\cal P}(1-\bar{\nu})$ on termination.

Having a hard deadline makes the CostSetter state space finite, since we only consider states with $\tau\leq\tau_{\max}$. We use a time-independent Poisson process for the ballot completion model $\Pr(n_{b}|\Delta_{\tau},c)$. These parameters are illustrative – Octopus can accommodate other utilities/distributions.

Baselines. Our main goal is to compare Octopus’s performance with state-of-the-art methods. However, we know of no algorithms that performs direct three-way optimization in a crowdsourced marketplace setting.222We don’t compare to Venetis et al. (?) since they run a tournament for max-finding, different from our task type. They also don’t change pay directly or model latency as done by us. So we compare against state-of-the-art methods that optimize 2 of the 3 objectives, while giving them the added benefit of our TaskSelector. We compare Octopus to Dai et al.’s and Gao et al.’s models, which are related to our work. Code was provided by the authors.

All comparisons are on the requester’s utility function, ${\cal U}$ computed against gold labels with ${\cal P}=200$. We normalize ${\cal U}$ so that Octopus always has 1.0 utility, i.e. the performance of baselines is represented as a proportion of Octopus.

Data collection. We collect data on AMT at 6 pay points using a Twitter Sentiment dataset (?). Workers are asked to classify the sentiments of tweets into either positive or negative sentiment. At each price point (\0.001,$0.002,\dots,$0.006$ per ballot), we post 400 tweets, and seek 20 ballots/tweet. A single HIT contains 10 tweets for a worker to solve. 40 tweets are common to all prices for a total of 2200 tweets. Each price is posted on a different weekday at the same time, and remains active for 24 hours. This ensures consistency in data collection across prices and minimizes interaction between different prices.

Figure 2a (task completion rates vs. pay) verifies that higher pricing results in faster task completion: at 0.1 cent, only 5500 ballots are received even after 24 hours, whereas at 0.6 cents, all 8000 ballots are received within 12 hours. We estimate the Poisson parameters using this data.

Worker Retention vs. Arrival. Contrary to prior work (?; ?), we observe that the increase in task completion rate with pay is predominantly due to higher worker retention, rather than a higher rate or number of worker arrivals (possibly due to the large number of HITs that we sought). Figure 3a shows that both retention and task completion rates are highly correlated, doubling as price goes from 0.1 to 0.6 cents. However, Figure 3b shows that the worker arrival rate does not rise much. To the best of our knowledge, there is no prior marketplace model that handles both retention and arrivals.

We also verify that worker quality is independent of pay. We run a K-S test for every pair of costs. We could not reject the null hypothesis (error rates drawn from cost-independent distributions) at $p<0.05$ (Figure 2b).

Simulation Experiments

Our main aim through simulations is to assess the quality of $\beta$-reconstruction. We use a variety of parameter settings and simulate Octopus with ballot arrivals simulated according to a pay-dependent Poisson process, and answers generated based on worker models. Note that this experiment is without any synchronization between the CostSetter state and the real batch. Figure 5a compares the $\theta$ values between the system and the batch (for one such setting333$500$ tasks with $p(d)=\beta(2.0,2.0)$, worker errors sampled from $\Gamma(2.0,0.5)$, ${\cal P}=200$, $\Delta_{\tau}=15$ mins, $\Delta_{\theta}=10$, $\Delta_{\bar{\nu}}=100$.). Even after 10 time intervals, and across multiple cost changes (the sharp drops in the curve), our tracking is extremely accurate. Figure 5b shows that our $\bar{\nu}$ tracking is extremely effective as well, with very little divergence from the true value. Even when the values do diverge in both cases, they are highly correlated.

High quality tracking of $\theta$ and $\bar{\nu}$ are essential. If we diverge too far from the real values, we would likely optimize the long-term expected reward poorly. Overall, this reaffirms the hypothesis that the CostSetter is able to capture the global state of the whole batch using just the aggregate statistics.

Our other goal is to compare Octopus’s performance with baselines in simulation, which we do below.

**Comparison to **Dai. Comparing Octopus with Dai highlights the benefit of changing cost on real-world utility. Ideally, our method should be able to vary pay to match or exceed the utility of the static cost baselines. We run both Octopus and Dai with deadlines ranging from 60 to 360 minutes. We run Dai for different static pays, ranging from 1 to 6 (measured in a tenth of a cent), allowing it to take ballots until the deadline. Statistical significance is indicated on the plots.

Figure 4 shows the comparison. Octopus simultaneously outperforms (or is at par with) all static cost baselines for every deadline, whereas each static cost baseline has a ‘sweet spot’ range of deadlines where it does best. Octopus plans robustly, where despite making pricing decisions 24 times without synchronizing for the $6$ hr deadline, it still outperforms Dai. Figure 4b depicts the utility averaged across all deadlines as a function of task-completion rates. As shown, Octopus is robust to different task-completion rates, and continues to outperform Dai on changing them. Figure 4d & 4e explain why Octopus achieves better utility scores than Dai – it maintains very high accuracy while keeping costs reasonable. Qualitatively, for every deadline, we observe that Octopus tends to have an average cost that is close to the best static cost.

**Comparison to **Gao. Comparing with Gao allows us to delineate the effect of the QualityManagers while optimizing for batch quality using aggregate statistics.

For fairness, we augment Gao’s framework; fixing $r$ ballots/task up-front, and using the same worker response model as Octopus for ballot aggregation. Figure 4c demonstrates Octopus’s performance against Gao for different values of $r$. Octopus consistently outperforms Gao for all values of $r$ and across all deadlines. Octopus’s performance improves for longer deadlines, due to the higher quality achieved by the QualityManager. For instance, Octopus is around $100\%$ better than Gao for the 6 hr deadline. We see that Gao-3 and Octopus incur nearly the same cost (Figure 4g) but Octopus is $\sim$6% better in terms of accuracy (Figure 4f). Examining Figure 6b reveals that this is in part due to Octopus collecting $\times\frac{1}{3}$ more ballots than Gao-3 by changing pay/ballot more intelligently.

It is interesting to note that increasing $r$ does not significantly improve baseline performance. This is due to the fact that the QualityManager only takes more ballots on tasks that really need them. For Gao the extra cost spent on every task is not offset by a corresponding increase in quality, especially for tasks that are very easy (don’t require $r$ ballots) or too hard (unsolved even with $r$ ballots). In Figure 4f & 4g, we notice that Octopus once again has near-highest accuracy, but spends much less than Gao variants that have higher accuracy than Octopus.

Lastly, Figure 6 demonstrates that Octopus has intuitive behavior – as the deadline length increases, Octopus spends less pay/ballot on average since more time can be taken to finish the batch of tasks.

These experiments demonstrate that Octopus simultaneously achieves higher utility than every variant of baseline methods, and does so without sacrificing accuracy or incurring high costs.

Offline Experiments on Real Data

We run extensive offline experiments on our collected data against the state-of-the-art baselines described earlier.

When comparing algorithms we sample different execution trajectories from the real data by choosing a random ballot for each task (since quality is cost-independent), while keeping ballot arrivals as per the real data. Multiple trajectories help compute statistical significance over algorithms’ performances. These experiments synchronize the CostSetter’s state with the real batch after every $\Delta_{\tau}$ time. We use an uninformed uniform distribution for the difficulty prior $p(d)$.

Figure 7a shows Octopus when compared to Dai at different static price points, with the $x$-axis being different $\tau_{\max}$ values. Statistical significance is marked on the plots. Octopus outperforms most Dai costs across all deadlines, with upto $37\%$ increase in real utility. No single static cost is able to match Octopus across all deadlines. This underscores the benefits of changing pay dynamically based on current task completion. In Figure 7d & 7e, we see that Octopus achieves high accuracy at relatively low cost.

Figure 7b compares Octopus with Gao-$r$, with $r$ denoting the static number of ballots per task. We find that Octopus outperforms Gao for most deadlines. Further analysis reveals that our batch exhibits a bi-modal task difficulty distribution – no algorithm exceeds around $76\%$ accuracy, while getting $70\%$ is easy with $\sim$1 ballot per task (in Figure 7f, we see a sharp jump in accuracy only when the deadline is long enough to take a lot of ballots). For shorter deadlines, where solving difficult tasks is infeasible, Gao outperforms Octopus slightly, since Octopus optimizes with respect to a uniform prior. For longer deadlines, estimates of task difficulties are refined by the QualityManagers, and Octopus  gives large gains.

Lastly, Octopus outperforms TaskSelector baselines where Greedy task selection is replaced by a random policy (Random), or a single round robin followed by random selection (Random-Robin), by large margins (Figure 7c). Note also that the baselines start to converge over time, as the advantage of doing intelligent task selection diminishes when nearly all tasks are run till completion.

Overall, we find that Octopus learns robust policies, consistently outperforming all baselines.

Live Online Experiments

Lastly, we deploy Octopus on AMT to test performance in a dynamic, online setting, as well as gain qualitative insight into worker behavior. In this, we keep exactly 3 HITS (of 10 tasks each) on AMT at a time. If a worker accepts a HIT, another one is posted immediately – this enables greedy task routing while ensuring full power of worker parallelism. After every $\Delta_{\tau}$ mins, all available HITs are taken down and reposted with the new price output by Octopus. Octopus’s policies are learned using task completion rates estimated from real data earlier. At runtime, querying Octopus is instantaneous. We solve a batch of 500 tweets for 3 deadlines – 1, 2, and 4 hours. We compare against Gao-$1$, the best baseline in offline expts in Table 2.

For the 1 hr deadline, Octopus maintains pay at 0.5 cent/ballot, before decreasing it to 0.1 cent/ballot in the last 15 minutes. This allows Octopus to receive around 1 ballot/task; more ballots stagnate ${\cal U}$ for short deadlines due to conflicting workers, and Octopus prefers to save money to optimize utility. On the other hand, Gao maintains pay at 0.6 cent throughout to also ensure it receives 1 ballot/task, but is unlucky in the responses it receives. In terms of decision making, the superiority of Octopus is clear, since it recognizes the danger posed by disagreement on task quality estimates.

In the 2 hr deadline, Octopus once again increases pay initially, but gets more ballot arrivals than expected. In response, after 45 minutes Octopus decreases pay so that it can take a higher number of ballots for difficult tasks, getting a substantial accuracy and utility improvement over Gao.

For the 4 hr deadline, Octopus is aggressive in trying to solve all tasks till completion. Due to the bi-modal difficulty of the tasks, workers provide several conflicting ballots on harder tasks, which the TaskSelector prefers to re-route for utility gain. The overall accuracy increases marginally over the 2 hour deadline.

Qualitatively, workers respond as predicted – flocking to the tasks when pay was set at 0.5 cent or more, and staying away at very low pay. Workers respond naturally to the price changing algorithm; dropping out immediately if the pay is suddenly lowered, and coming back if it is increased once again. No worker complained about the fluctuating pay.

Discussion

In this work, we focused on experiments with Octopus in a fixed time deadline setting to compare with past work. Extension to the popular fixed budget setting (where $\sum_{q}C_{q}$ is constrained) is simple: (i) never synchronize $\theta$ so that its value in any state is simply the start value of $\theta$ (which is fixed and known) minus the ballots received, making it a proxy for cost incurred so far; (ii) modify the CostSetter’s reward to give a $-\infty$ reward for total cost (now computable using $\theta$) exceeding the budget. Another extension involves using a different formulation of the QualityManager; any algorithm that defines an appropriate policy and from which we can compute $\theta$ and $\bar{\nu}$ is suitable.

Conclusion

We present Octopus, one of the first AI agents for a 3-way optimization of total cost, work quality and completion time in crowdsourcing. The agent combines three different subagents that control quality per task, select the best next task and set pay for the whole batch. A key technical contribution is the computation of aggregate statistics of the quality and completeness of the whole batch – this is used as the state for best setting the next pay.

Octopus outperforms state-of-the-art baselines in a variety of simulated and real world settings, demonstrating the superiority of our approach. We also showcase Octopus’s real world applicability by deploying it directly on AMT. In the future, we hope to develop general purpose formulations of Octopus as a plug-and-play architecture for practitioners.

Acknowledgments

This work is supported by Google language understanding and knowledge discovery focused research grants, a Bloomberg award, a Microsoft Azure sponsorship, and a Visvesvaraya faculty award by Govt. of India to the third author. We thank Chris Lin, Yihan Gao and Aditya Parameswaran for sharing code, and all the AMT workers who participated in our experiments.