Scalable Panel Fusion Using Distributed Min Cost Flow

TL;DR

This paper introduces a scalable, network flow-based method for panel data fusion in digital audience measurement, efficiently handling large datasets through distributed algorithms and providing conditions for optimality.

Contribution

It formalizes the panel fusion problem as a network flow model and proposes a distributed algorithm to solve large-scale instances efficiently.

Findings

Successfully fused two real-world panel datasets

Achieved scalable performance with tens of millions of observations

Provided theoretical conditions for solution optimality

Abstract

Modern audience measurement requires combining observations from disparate panel datasets. Connecting and relating such panel datasets is a process termed panel fusion. This paper formalizes the panel fusion problem and presents a novel approach to solve it. We cast the panel fusion as a network flow problem, allowing the application of a rich body of research. In the context of digital audience measurement, where panel sizes can grow into the tens of millions, we propose an efficient algorithm to partition the network into sub-problems. While the algorithm solves a relaxed version of the original problem, we provide conditions under which it guarantees optimality. We demonstrate our approach by fusing two real-world panel datasets in a distributed computing environment.

Tables

Table 1. Table 1. Comparative results

	Core fusion algorithm	Optimized iterative fusion algorithm
Compute resources	Single node with 150 GB memory, 88 cores	Spark cluster with 50 executors - 10 GB and 3 cores per executor
End to end execution time	2.5 Hour	20 minutes
Cost	x	2.37x
Total assignments	92,600	92,543
Assignments within same demo categories	56,506 (61.02%)	58,521 (63.23%)
Assignments across different demo categories	36,094 (38.97%)	34,022 (36.76%)
Flow assigned with same demo categories	56.27%	58.60%
Flow assigned across different demo categories	43.72%	41.39%

Table 2. Table 2. Clustering (Partitioning) parameters relaxation

Iteration	clustering (partitioning) parameters
Iteration 1	age, gender, ethnicity, household income, race, household size, children
Iteration 2	age, gender, ethnicity, household income, race, household size
Iteration 3	age, gender, ethnicity, household income, race
Iteration 4	age, gender, ethnicity, household income
Iteration 5	age, gender, ethnicity
Iteration 6	age, gender
Iteration 7	age
Iteration 8	No partitioning

Table 3. Table 3. Number of matched panelists per iteration

	Census panel dataset	Traditional panel dataset
Iteration 1	7,044,759	306,504
Iteration 2	910,327	68,191
Iteration 3	128,173	14,003
Iteration 4	378,567	42,428
Iteration 5	202,914	11,241
Iteration 6	56,411	9,985
Iteration 7	43,643	5,665
Iteration 8	0	0
Total	8,764,794	458,017

Table 4. Table 4. Quality of matches results

	Traditional panel dataset (A)	Traditional panel dataset (B)
Number of panelists	458,017	458,017
Same panelist matches	99.37% (455,127)	99.37% (455,127)
Different panelist matches	0.63% (2,890)	0.63% (2,890)

Equations

$$\sum _{i\in U}{w}_i=\sum _{j\in V}{w}_j$$

$$\underset{x}{\min }$$

$$\sum _{j\in V}{x}_{i,j}=w_i\text{mbox}foralli\in U$$

$$\sum _{i\in U}{x}_{i,j}=w_j\text{mbox}forallj\in V$$

$$w_d=\sum w_i-\sum w_j$$

$$d(z_{i},z_{j})=\sum_{k\in R}\left(z_{i}^{(k)}-z_{j}^{(k)}\right)^{2}+\begin{cases}\infty&\mbox{if }z^{(k)}_{i}\neq z^{(k)}_{j}\mbox{ for any $k\in C$ }\\ 0&\mbox{otherwise}\end{cases}.$$

Full text

Internet measurement, panel fusion, network flow, distributed minimum cost flow

1. Introduction

Audience measurement – the estimation of the size and characteristics of an audience – plays a fundamental role in the advertising ecosystem: advertisers pay content producers (for example – newspapers, websites, and television networks) based on the number of people exposed to content and advertisements.

Audience measurement has traditionally relied on audience panels. An audience panel is a group of participants that agree to logging of their exposure to media, and demographic and household information. Some media – such as over the air television – require traditional panels for measurement, as there is no return communication path. Measurement of media delivered to internet connected devices follows a distinct paradigm – information can be collected electronically, through a variety of means – monitoring software, web page, video, and mobile application tracking tags, and internet connected televisions. The result is extremely large scale panels with narrow purview. Instead of measuring all behavior of a person, only a single medium is measured. This is the challenge of cross-media audience measurement.

In practice, multiple large scale disparate panels are required to provide accurate statistics on cross-media audiences. Consider the following example. Imagine a first panel measuring mobile application consumption, and a second panel measuring website consumption, with no way to directly link the panelists. How can one measure the size of the audience shared by an app and a website? Panel fusion addresses this question by relating panelists from two or more panels, and combining their observed behaviors to create accurate cross-media statistics.

Panel datasets represent selected groups of people or households with demographic attributes, e.g., age, gender, household size, income, race, and ethnicity. More advanced demographic attributes like supermarket preferences, automobile ownership, product purchase interests can also be associated with panelists. Each panelist is assigned a projection weight. These weights are used to make the overall panel represent the demographic composition and behavior of the audience universe, using techniques such as Raking (Deville et al., 1993). Different panels are comprised of different panelists, each with a different bias and different associated projection weight. A key component to a successful fusion is matching the panelists in the different panels in a way that maintains the composition and behavior present in both panels in the combined dataset in an optimal way. The alignment of projection weights between two disparate panels can be cast as a network flow problem with the weights of the first panel representing the source, and the weights of the second panel representing the sink. The cost of associating the behavior of a panelist from the first panel with the behavior of a panelist from the second panel is defined by a subjective measure of similarity between their behavior and demographic information.

In this paper, we address the problem of combining two disjoint panel datasets by casting the problem as a min-cost-circulation network problem. The formulation guarantees projection weight alignment between the panels, and optimizes to prefer associations that align demographic profiles and observed behavior similarities. The approach and solution can be applied to any panel fusion problems that need to be optimally solved at a large scale. We provide a computationally efficient algorithm designed for distributed compute platforms.

We demonstrate our methodology at internet scale with large datasets from Comscore, an internet and TV audience measurement and analytics company. Comscore’s digital network consists of web pages-, advertisement- and application tags deployed on websites and advertisements across the internet. The scale of the dataset is immense, consisting of more than 50 billion measured events each day. These measurements are organized to define a single large scale (census) inferred panel of over 30 million panelists. The breadth of the information in this large scale panel is limited in that only the activity on tagged websites, apps and advertisements is observed. Comscore also maintains complementary traditional panel datasets. These panels are comprised of panelists who have agreed to install monitoring software on their desktop, mobile or other digital devices. The monitoring software reports on the online behavior of the panelist, creating a complete picture of their internet activity. These panel datasets consists of approximately one million panelists with demographic and behavioral information. After casting the panel fusion problem as a minimum cost circulation problem, we fuse the two datasets, creating a single source panel, with scale derived from the first panel, and breadth derived from the second.

In summary, this paper makes the following contributions. First, we formalize the panel fusion problem, and cast it as a minimum cost circulation problem, following the work of (Soong and de Montigny, 2001). Next, we propose a scalable solution to solve the fusion problem, and more general minimum cost circulation problems. We demonstrate our results at scale, and report on the characteristics of the fused panels.

2. Problem definition

In this section, we define the panel fusion problem. We cast the problem as a transportation problem, which is a special case of the more general network flow problem.

Let $U$ represent a set of panelists with $|U|=n_{1}$, and $V$ be a second set of panelists with $|V|=n_{2}$. Each panelist $i$ is a node in a bipartite graph, $G=(U,V,E)$, with $i\in\{U\cup V\}$. The two sets of panelists are disjoint, $\{U\cap V\}=\{\}$. Each panelist $i$ has an associated projection weight, $w_{i}\in\mathbb{R}$, $w_{i}>0$, and a feature vector, $z_{i}$ of length $m$. Edges between nodes $i$ and $j$, denoted $e_{i,j}\in E$ have an associated cost per unit flow $c_{i,j}$, which depends on the dissimilarities of the features of the panelist: $c_{i,j}=d(z_{i},z_{j})$, where $d(\cdot,\cdot)$ is a measure of distance between the attributes between the panelist.

In our instantiation of the problem, the feature vector contains categorical values corresponding to demographic categories (i.e, Male, age 20-24, etc.), and real valued features representing minutes associated with types of behavior (i.e, one hour spent on social media). Additionally, the panelist weights, specified a priori, are used to scale the panelist behavior to match the measurement universe 111For example, if the panel is comprised of a random sample from the general population at a rate of 1/1000, each panelist is given a weight of 1000.. Both panels represent the same measurement universe, hence

[TABLE]

is given.

To provide intuition, a match between panelist $i\in U$ and panelist $j\in V$ is considered appropriate if the following qualitative constraints are met:

1. (I)

the projection weights of matched panelists are the same, and 2. (II)

feature vectors $z_{i}$ and $z_{j}$ are as similar as possible.

We make these qualities precise in Section 2.1. Assignment constraints as defined above imply a one to one assignment between panelists. In practice, our approach requires fractional assignment of panelists, as in general, $n_{1}\neq n_{2}$.

2.1. Panel fusion as transportation problem

The aim of the minimum cost flow problem is to determine a path with the least cost through a network, while simultaneously satisfying supply and demand constraints of the nodes. In the panel fusion problem, a dense bipartite graph where the projection weights represent the source – the supply – and sink – the demand – of the network. Let edge $e_{i,j}$ be have upper capacity bound = $\infty$ and lower capacity bound = 0. The upper bound is considered $\infty$ for simplicity of explanation but can be changed to influence flow where needed. In many practical applications, the lower bound is set to 0 to allow discarding certain edges completely if needed. Let the flow on an $e_{i,j}$ be given by $x_{i,j}$. The minimum cost flow problem is written as follows:

[TABLE]

The minimum cost circulation problem aims to minimize the total cost by adjusting the flow between the nodes, $x_{i,j}$. In other words, the sum of the costs (the product of the cost per unit flow, and the flow), across all edges in the bipartite graph, is minimized.

The constraints state that, for every panelist $i\in U$, the fractional weights of the panelists in $V$ that flow to/from $i$ must sum to $w_{i}$. The same constraint applies for every panelist $j$ in $V$. The constraints guarantee alignment of the panelist weights as proposed in constraint \Romannum1.

3. Algorithm

We present the core framework for solving panel matching and then work through natural modifications which lead to the practical implementation of a solution with large scale distributed platforms. Section 5 explains more about minimum circulation flow problem and related algorithms.

Algorithm 1 outlines the core fusion algorithm with a single bipartite graph. Line [2-3] describes the normalization of features and calculation of distances between panelists of the two disjoint panel sets. Normalization and distance methods are not explained for the sake of simplicity but any effective method can be applied.

Line 4 describes generation of bipartite graph as defined in Sec 2.1. $\mathit{MCFSolver}$ on line 6 contains the cost scaling successive approximation algorithm to solve with minimum cost flow in bipartite graph. $\mathit{generateAssignedPairs}$ on line 7 transforms the flow solution into assigned pairs of ($i$, $j$, $x_{(i,j)}$) where $i$ is matched with $j$ with flow of $x_{(i,j)}$.

As defined in (Ahuja et al., 1995), the mass balance constraint and the flow bound constraints of network flow problem asserts our Constraint \Romannum1, which enforces that the weights of matched pair weights are perfectly aligned with outgoing/incoming flows. Cost minimization in network flow solution where cost is a function of the distance between panelists optimizes the results against Constraint \Romannum2.

3.1. Node splitting

It is worth mentioning that the solution includes fractional assignments. Therefore, a given node can be assigned to multiple nodes while making sure that split or fractional flow is bounded by supply or demand. We refer this case as $\mathit{node\ splitting}$. In the panel fusion problem, this can be interpreted as one panelist being matched with multiple panelists by distributing its projection weight while following Equation 2.

This is a side effect of solving panel fusion with a single bipartite graph. We will provide an optimization where this can be completely avoided if needed. This optional constraint can be enforced depending on the requirements of the fusion problem at hand.

3.2. Algorithm optimization

For many applications in modern audience measurement, the above algorithm requires solving a large bipartite graph. In our particular application, it is computationally challenging if not entirely impractical to solve. Finding the minimum cost flow in a bipartite graph with $\sim$30 million nodes and $\sim$15 trillions edges is time and resource intensive solution. As such, the practicality of the above algorithm in real world problems is limited.

In Algorithm 1, cost function represents the distance between panelists from two different panels. Two panelists with very high distance are very unlikely to be assigned to each other even when solved with one single bipartite graph. Assuming this, if we disconnect such distant panelists then the large bipartite graph can be clustered (partitioned) into smaller sub-graphs. A subset of distance features can be used as clustering (partitioning) parameters which is an effective heuristic to localize panelists into smaller clusters. Any residual panelists and their weights can be re-clustered (re-partitioned) by $\mathit{relaxed}$ clustering (partitioning) parameters into sub-graphs to be solved again.

These clustering (partitioning) parameters can be relaxed iteratively to give better diversity but with less specificity. It gives any residual panelist a wider pool of panelists to be matched compared to the previous iteration. This approach resembles with $\mathit{search\ query\ relaxation}$ - widely used in information retrieval research area where query parameters are relaxed iteratively to give ample query results in order of specificity.

Intuitively, this $\mathit{iterative\ relaxed\ clustering\ (partitioning)}$ would give similar assignments as Algorithm 1 with a single large bipartite graph. Clustered (partitioned) smaller sub-graphs can be solved independently on multiple processors, which is ideal for distributed platforms. It makes massive scale panel fusion an embarrassingly parallel problem on any distributed platform with commodity hardware. An obvious additional benefit of splitting the problem into smaller sub-graphs is that the aggregated computing time for the sub-graphs will be significantly less than the time for solving the single dense graph because of the polynomial complexity of the algorithm: $O(n^{2}m\log(Cn))$.

$\mathit{Node\ splitting}$ can be also avoided by ignoring any split or fractional panelist assignments and leaving that panelist for the next iteration to be matched with the wider pool of panelists due to relaxed clustering (partitioning).

We observed that features with higher subjective matching priority (e.g, gender versus present of children in household) work very well as clustering (partitioning) parameters and remaining features can be used for distance in the cost function within smaller sub-graphs. Depending on problem definition, geographical location boundaries like zip codes, market areas, states can also be used as a clustering (partitioning) parameters.

3.2.1. Pseudocode:

Algorithm 2 describes the optimization discussed above. Feature vector $z$ is normalized as before and divided into two feature sets for clustering (partitioning) and distance calculations on Line 2. Lines [3-17] describe iterative relaxation of clustering parameter $p$ which performs a loop until all panelists from either side are completely assigned. Inside this loop, line 4 shows generating clusters (partitions) using features $p$. Lines [6-16] show routine that can run on separate processors in parallel. Line 7 calculates the distance between panelists in same cluster using distance features $d$ and build bipartite graph on line 8.

Line 10 explains an optional step to prune edges in the graph if needed. We note that (Lee and Orlin, 1994) mentions $2n\log n$ edges are sufficient for making optimized assignment so pruning the edges can be helpful to speed up computation. Edge pruning is an optional step and is not to be considered the only optimization. We evaluated pruning edges at random. In particular, edges below a certain threshold were discarded, disconnecting any node from bipartite graph. Pruning is also helpful to remove unwanted assignments in the solution and can be a reasonable adaptation depending on the domain of a problem.

Equation 1 states that total weights in both sets of panelists is exactly the same as they represent the same universe. This does not hold true when we divide panelists into smaller clusters to build sub-graphs. Therefore, we add a dummy balancing node on the side of bipartite graph with less supply/demand. The balancing node gets supply/demand equal to the absolute difference between supplyTotal and demandTotal. This process is defined at line 11.

Line 12 solves the graph in the same manner as Algorithm 1. Line 14 describes the residual process that generates assignment pairs and removes/updates matched panelists and their weights for the next iteration. Balancing nodes, if added, should be removed from assignment solution set. Line 15 relaxes the clustering parameter $p$ so the next iteration generates more diverse but less specific clusters to create a wider panelists pool for assignment.

3.2.2. Graph balancing:

There will always be an optimal flow with minimal cost in the graph provided there is enough supply and demand to match and edges with the capacity to transport. We need to make sure that every sub-graph will have an optimal solution by setting the total supply as same as the total demand. If not, we should add a dummy balancing node just to solve the sub-graph optimally. This dummy balancing node $d$ has proxy supply/demand as,

[TABLE]

if $w_{d}$ ¡ 0, then node d is $\mathit{supply\ node}$; if $w_{d}$ ¿ 0, then node d is $\mathit{demand\ node}$;

This dummy balancing node and flows assigned from/to it are removed from the assignment pair solution set. Residual panelists and weights are adjusted after removing the balancing node.

4. Evaluation

We evaluated our algorithm on a large scale panel fusion problem from Comscore. Census-level inferred panel dataset are derived from measurement data. The Comscore tagging network is referred as census panel dataset. It contains panelist demo identifiers and minutes spent on internet categories across PC, Phone and Tablet platforms. Comscore also maintains traditional panel dataset through recruitment, which contains similar information. The census panel dataset has the breadth to cover niche audience measurement demands while the traditional panel datasets contain in-depth insights. Combining these two independent panel datasets addresses complex measurement problems.

Let $z_{i}=[z_{i}^{(1)},z_{i}^{(2)},\dots]^{T}$ be the feature vector, with both real valued elements $z_{i}^{(k)}$ for $k\in R$ and categorical valued elements $z_{i}^{(k)}$, $k\in C$ otherwise. Then, the distance function can be generalized as -

[TABLE]

In our instantiation, demographic features such as gender, age-group and income are categorical features.

Claim 1: The relaxed version of the network flow problem solved by Alg. 2 is optimal if: i) the cost per unit flow is defined by Equation 4 and ii) node re-balancing is not required.

We compare our optimized algorithm with the core fusion algorithm containing single graph. We demonstrate results with PC panel datasets only but the same experiments can be performed with Phone and Tablet panel sets as well. The Census panel dataset for PC platform contains approximately 8.7 million panelists while the traditional panel dataset has over 450,000 panelists with their demo profiles and online behavior. For simplicity’s sake, demo profiles can be categorized as demo categories for age, gender, household income, ethnicity, race, household size, presence of children in household. We use the Google OR tool (Goo, 2019) library for cost scaling successive approximation algorithm to solve assignment problem. This is an open source library with JVM support which makes it easier to use in modern distributed systems like Apache Spark (Apa, 2019), (Zaharia et al., 2016).

A single bipartite graph for the above panel datasets would generate around 4 trillion edges, which is computationally expensive to build and solve for the minimum circulation problem. For experiment purpose, we sampled these datasets to 1% and adjusted projection weights accordingly to represent entire universe. This reduced sample sizes of the census panel dataset and the traditional panel dataset for PC platform to 87,576 and 4,605 respectively.

4.0.1. Core fusion algorithm setup

: We generated a single bipartite graph for these sampled datasets with 403,287,480 edges with the cost between the edges as distance using Equation 4. All features are used for L2 distance calculation. The cost of edges between panelists with different demo categories, however, was penalized with scalar of 1000 to discourage flow on such edges unless absolutely necessary. This helps to enforce Constraint \Romannum2 as explained before. We followed Algorithm 1 to generate assignments on a standalone machine.

4.0.2. Optimized iterative relaxed fusion algorithm setup

: We followed the optimization described in Algorithm 2. Only real valued features (minutes spent per internet category) are used for L2 distances while demo identifiers are used as clustering (partitioning) parameters. Equation 4 is used for distance calculations. In iteration 1, we generated 2858 clusters (partitions) for each unique combination of demo categories and built independent bipartite graphs to solve in parallel. We leveraged the Apache Spark distributed system where each task was given a graph to solve. For simplicity of explanation, we completely relaxed clustering parameter in iteration 2 and combined all demo categories together to create one single graph. Again, the cost of edges between panelists in iteration 2 graph were penalized with scalar of 1000 to discourage flow unless necessary.

Table 1 shows comparative results from test runs with the above experimental setup. The core fusion algorithm consumes considerable physical memory on a single node even with just 1% of sampled data. This proves serious scalability concerns for the core fusion algorithm in practical applications. On the hand, the optimized algorithm is ideal for distributed systems with reasonable compute resources. The optimized version also outperforms the core algorithm for end to end execution time due to the obvious parallelism in the solution. This makes optimized relaxed fusion algorithm easy to adapt on commodity hardware with general purpose distributed systems.

The optimal cost of the iterative relaxed fusion algorithm is 2.37 times higher than the core fusion algorithm. We observed that cost with an optimized algorithm decreases as clustering (partitioning) parameters are slowly relaxed with multiple iterations. Comparing the matching results, however, looks very promising. The core fusion algorithm solution consists of 61.02% and 38.97% assignments within same demo categories and across different categories respectively. The optimized algorithm solution has slightly better results with 63.23% and 36.76% assignments within same demo categories and across different demo categories. This is due to the stricter partitioning of graph based on demo categories in iteration 1. Overall, an optimized iterative relaxed algorithm comes with an additional cost of a circulation but it is highly practical with general purpose hardware to compute and provides similar assignment results in the end.

Furthermore, we provide results from the full scale PC panel fusion problem described above. Panel fusion of approximately 8.7M panelists from the census panel dataset with approximately 450,000 panelists from the traditional panel dataset was done with 8 iterations where clustering (partitioning) demo categories are slowly relaxed as shown in Table 2. Table 3 shows number of panelists matched from both census panel dataset and traditional panel dataset in every iteration. It demonstrates that panelists from both datasets are completely matched at iteration 7. The number of matched panelists decreases as we relax the clustering (partitioning) parameter. This proves Constraint \Romannum2 is met, as more panelists are matched with strict demo profiles.

Figure 1 and 2 show the decrease in execution time and the number of clusters (partitions) as the iteration increases. Figure 3 shows decay in unmatched panelists as we advances with iterations. Iteration 7 has 0 unmatched panelists on both sides.

4.1. Matching effectiveness

Validation of panelist matching quality in this problem is very subjective and difficult due to lack of ground truth. We set up an experiment where same panel dataset is used on both sides of bipartite graph for fusion. Any panelist will obviously have the same demographic profile and online behavior when compared to itself. The most ideal match in this case is when same panelist is matched to itself because it will be guaranteed to meet constraints \Romannum1, \Romannum2. We used the traditional panel dataset on both sides of bipartite graph and run our optimized algorithm as explained above. Table 4 shows the results for the same. It shows that more than 99% panelists were matched with themselves which proves that our optimized algorithm generates optimal matches in real world scenarios as well.

This strengthens our belief that using relaxed distributed minimum cost flow algorithm guarantees generation of optimal assignments (matches) in real world panel fusion at scale. Comscore runs several independent analysis of using panel assignments from this algorithm for cross platform campaign evaluations. Results are very much aligned with actual panelists observed across multiple platforms while increasing the scale of measurement for better accuracy. Many niche cross platform campaigns which were difficult to measure with limited observable panelists are now very much possible due to the scale of the of the disparate panel datasets that we can now fuse.

5. Related Work

While there is limited academic literature on large scale panel fusion problems, network optimization and its applications in real world problems has a very rich history. Network flow problems were first studied by Russian mathematician A.N. Tolstoi in the 1930s to build a railway network in Russia. (Ahuja et al., 1995) discusses various applications of network optimization from the fields of operation research, computer science, medicine, engineering, and applied mathematics.

Minimum cost circulation is a generalization of a maximum flow problem. We refer the reader to (Goldberg and Tarjan, 1990) for an outline of the problem, and a solution by cost scaling successive approximation. (Goldberg, 1997) and (Bünnagel et al., 1998) further provide an efficient implementation of same algorithm with heuristics improvements. This work is closely related to Alg. 1. This improves practical running time of the algorithm but does not improve worst case complexity of algorithm which is $O(n^{2}m\log(Cn))$ where $C$ is a constant that bounds the largest cost in the graph. This algorithm works practically better than other min cost flow algorithms as proved in (Goldberg, 1997).

(Burkard and Cela, 1999) describes different aspects of linear assignment problem and various applications of it. (Soong and de Montigny, 2001) discusses panel fusion with unconstrained and constrained statistical matching. The constrained matching is based on a similar transportation problem solved with widely known stepping algorithm. It also demonstrates small scale fusion of TV panelists with magazine/product usage survey.

Network flow algorithms are widely used in modern assignment problems across different domains. (Butt and Collins, 2013) and (Zhang et al., 2008) demonstrates effective use of min cost network flow in computer vision problem. (Lenz et al., 2015), (kit, 2019) extends this work to improve the algorithmic complexity of min cost flow in online vehicle tracking systems. (Tian et al., 2015) provides use of min-cost flow network in text detection systems. We want to utilize this cross domain research work on minimum cost flow network to improve computational complexity of offline panel fusion at scale.

Our algorithm is aimed at transforming the generic panel fusion problem into a network flow and solving it with massive panel datasets. As discussed in section 3.2, Iterative relaxation of clustering (partitioning) parameters provide optimization ideal for distributed systems. Feature selection, normalization, and clustering (partitioning) is very generic so can be tweaked based on problem definition. We evaluated our optimization with sampled datasets and found similar results as solved with a single huge bipartite graph.

Our methodology allows panel fusion at scale to be solved efficiently and in an optimal way with commodity hardware.

6. Summary and Future Work

In this paper, we aim to solve the problem of scalable panel fusion by panelists assignments from two independent datasets. This is an important problem to solve to meet the demands of highly specific but accurate audience measurement. Media is being consumed by a plethora of platforms and devices with various consumption patterns which makes use of a single monolithic panel dataset obsolete. Panel fusion methodology helps to combine such disparate independent panel data sources together to build a comprehensive and cohesive audience panelist dataset. Learning audience behavior from such a fused panel dataset is critical for the success of accurate content and advertisement measurement. We develop a methodology to successfully transform panel fusion and its constraints into a transportation problem and solving it with minimum cost circulation methods. Cost scaling successive approximation algorithm is used to solve minimum cost flow optimally. We provide an optimized iterative relaxation fusion algorithm to solve real world large scale assignment problems. We evaluated our optimization method with Comscore’s census and traditional panel datasets. The optimization method provides similar assignments compared to the naive method where all assignments are generated using a single huge graph. Optimized algorithm is computationally very efficient and can easily scale with general purpose distribute platforms. At the same time, our algorithm remains very generic where domain specific clustering (partitioning) methods can be easily applied while core iterative minimum cost circulation methodology remains the same. A generic implementation of this same algorithm running on Apache Spark is successfully used for large scale panel fusion problems in Comscore for various countries, platforms etc. Expanding this same algorithm beyond panel fusion and improving algorithmic complexity to solve generic combinatorial assignments will be our future work.