Wikipedia Text Reuse: Within and Without

TL;DR

This paper presents a large-scale analysis of text reuse within and outside Wikipedia, using advanced detection technology to uncover millions of reuse cases, and provides a new corpus and pipeline for future research.

Contribution

It introduces the first comprehensive corpus of Wikipedia text reuse cases and scales state-of-the-art detection methods to analyze reuse across the entire platform and web.

Findings

Discovered 100 million reuse cases inside Wikipedia

Identified 1.6 million reuse cases outside Wikipedia

Enabled new tasks like template induction and quality correction

Abstract

We study text reuse related to Wikipedia at scale by compiling the first corpus of text reuse cases within Wikipedia as well as without (i.e., reuse of Wikipedia text in a sample of the Common Crawl). To discover reuse beyond verbatim copy and paste, we employ state-of-the-art text reuse detection technology, scaling it for the first time to process the entire Wikipedia as part of a distributed retrieval pipeline. We further report on a pilot analysis of the 100 million reuse cases inside, and the 1.6 million reuse cases outside Wikipedia that we discovered. Text reuse inside Wikipedia gives rise to new tasks such as article template induction, fixing quality flaws due to inconsistencies arising from asynchronous editing of reused passages, or complementing Wikipedia's ontology. Text reuse outside Wikipedia yields a tangible metric for the emerging field of quantifying Wikipedia's influence on the web. To foster future research into these tasks, and for reproducibility's sake, the Wikipedia text reuse corpus and the retrieval pipeline are made freely available.

Tables

Table 1. Table 1: Overview of input datasets (left), source retrieval performance (middle), and retrieved text reuse (right). Counts in the tables left and right are in millions.

Dataset	Count
Wikipedia
Articles	4.2
Paragraphs	11.4
Common Crawl
Websites	1.4
Web pages	591.0
Paragraphs	187.0

Table 2. Table 2: Examples of the two types of text reuse within Wikipedia—structure reuse (top) and content reuse (bottom)—along ontological article relations (right).

Title: Niedźwiedzie, Pisz County	Title: Zimna Woda, Zgierz County
Niedźwiedzie is a village in the administrative district of Gmina Pisz, within Pisz County, Warmian-Masurian Voivodeship, in northern Poland. It lies approximately south-east of Pisz and east of the regional capital Olsztyn.	Zimna Woda is a village in the administrative district of Gmina Zgierz, within Zgierz County, Łódź Voivodeship, in central Poland. It lies approximately north-west of Zgierz and north-west of the regional capital Łódź.

Equations

$$\underset{\text{mbox}Searchpruning}{\underbrace{\exists c_i\in d_1,c_j\in d_2:h(c_i)\cap h(c_j)\ne \varnothing }}\to \rho (d_1,d_2)=\underset{\begin{array}{c}{c}_i\in d_1\\ c_j\in d_2\end{array}}{\max }(\phi (c_i,c_j)),$$

Full text

1 Introduction

Text reuse is second nature to Wikipedia: inside Wikipedia, the articles grouped in a given category are often harmonized until templates emerge, which are then adopted for newly created articles in the same category. Moreover, passages are copied verbatim from one article to another, such as, in cases where they form a hierarchical relationship. While the reuse of text inside Wikipedia has been a de facto policy for many years, neither the MediaWiki software nor tools developed by and for the Wikipedia community offer any reuse support beyond templating: unless a dedicated Wikipedia editor takes care of it, a copied passage will eventually diverge from its original, resulting in inconsistency.

Outside Wikipedia, we distinguish reuse of Wikipedia’s articles by third parties, and reuse of third-party content by Wikipedia. The former is widespread: for example, passages of articles are manually reused in quotations and summaries, or automatically extracted to search result pages. Many sites mirror Wikipedia partially or in full—sometimes with proper attribution, other times violating Wikipedia’s lenient copyrights. The latter form of reuse is discouraged.

With a few exceptions noted below, Wikipedia text reuse has not been analyzed at scale. We attribute the lack of studies to the lack of open and scalable technologies capable of detecting text reuse regardless of reformatting, paraphrasing, and light summarization, as well as to the significant computational overhead required to process the entire Wikipedia. Only recently, resulting from a six-year effort to systematically evaluate reuse detection algorithms in the PAN lab at the CLEF conference, new classes of algorithms emerged that specifically address the detection of such kinds of text reuse from large text corpora.

To foster research into Wikipedia text reuse, we compile the first Wikipedia text reuse corpus, obtained from comparing the entire Wikipedia to itself as well as to a 10%-sample of the Common Crawl. For this purpose, we scaled up the aforementioned algorithms for the first time to render the computations feasible on a mid-sized cluster. Finally, we carry out a first exploratory analysis, enabling us to report on findings that cover insights on how Wikipedia text reuse detection could be used as a tool for both helping the Wikipedia community to ensure the encyclopedia’s consistency and coherence, as well as for quantifying the influence of Wikipedia on the web at large.

2 Related Work

Wikipedia’s openness and success fuels tons of research about the encyclopedia111https://en.wikipedia.org/wiki/Academic_studies_about_Wikipedia as well as on how it can be exploited, e.g., for disease forecasting [8], movie success prediction [11], and much more. Wikipedia’s influence on the web has recently become a focus of interest: for instance, posts on Stack Overflow and Reddit that link to Wikipedia have been found to outperform others in terms of interactions [19]. Other works have studied Wikipedia’s role in driving research in the scientific community [18], and its importance to search engines in enriching search result pages [10]. The ever increasing quality of Wikipedia drives the reuse of its content by third parties, but in a “paradox of reuse” reduces the need to visit Wikipedia itself [17], depriving the encyclopedia of potential new editors.

In general, text reuse detection is applied in many domains [2], such as the digital humanities (to uncover the influence of ancient authors [7]), and in journalism and science (to study author perspectives [6] as well as to pursue copyright infringement and plagiarism [5]). Text reuse detection divides into the subtasks of source retrieval and text alignment [13, 16], where the former retrieves a set of candidate sources for reuse given a questioned document, and the latter the mutually reusing passages given a document pair. Approaches addressing each task have been systematically evaluated at PAN [13].

Regarding Wikipedia in particular, text reuse detection has the potential to aid the community in improving the encyclopedia as well to serve as a new method of quantifying its influence on the web. However, Wikipedia text reuse has been mostly disregarded to date, except for two pioneering studies: Weissman et al. [20] report on text reuse within Wikipedia, and its connection to article quality. Using similarity hashing, near-duplicate sentences are identified that are redundant or contradictory. Similarly, Ardi et al. [1] employ hashing to detect near-duplicates of Wikipedia articles in the Common Crawl. Both neglect the text alignment step, yielding reuse cases that are either too fine-grained or too coarse-grained for reuse analysis. Our text reuse detection pipeline incorporates similar hashing techniques for source retrieval but further filters and refines the results through text alignment to obtain the actual reused text passages. In this respect, our corpus captures better the original user intent of reusing a given passage of text while saving its users the computational overhead.

3 Corpus Construction

Given two document collections $D_{1}$ and $D_{2}$, we aim to identify all cases of text reuse as pairs of sufficiently similar text spans. Here, $D_{1}$ comprises all English Wikipedia articles and $D_{2}=D_{1}$ for within-Wikipedia detection, whereas otherwise $D_{2}$ comprises a 10% sample of the Common Crawl (see Table 1 (left)). Our processing pipeline first carries out source retrieval to identify promising candidate document pairs, which are then compared in detail during text alignment.

3.1 Source Retrieval

In source retrieval, given a questioned document $d_{1}\in D_{1}$ and $D_{2}$, the task is to rank the documents in $D_{2}$ in order of decreasing likelihood of sharing reused text with $d_{1}$. An absolute cutoff rank $k$ and/or a relative score threshold $\tau$ may be used to decide how many of the top-ranked documents become subject to the more expensive task of text alignment with $d_{1}$. These parameters are typically determined in terms of the budget of computational capacity available as well as the desired recall-level. An ideal ranking function would rank all documents in $D_{2}$ that reuse text from $d_{1}$ highest; however, the typical operationalization using text similarity measures does not reach this ideal. The higher the desired recall-level, the lower the precision and the higher the computational overhead.

Our computational budget were 2 months worth of processing time on a 130 node cluster (12 CPU cores and 196 GB RAM each) running Apache Spark, an the goal was to maximize recall. Since $D_{1}$ (Wikipedia) as a whole is questioned, we generalized source retrieval toward a pruned all-pairs similarity search, ranking all $(d_{1},d_{2})\in D_{1}\times D_{2}$ based on the following pruned ranking function $\rho$:

[TABLE]

where $c$ is a passage-length chunk of document $d$, $h$ is a locality-preserving hash function, and $\varphi$ a text similarity measure. The design goal was to apply great leniency during search pruning (a single hash collision suffices for rank scoring), and to take into account that reuse is a passage-level phenomenon.

To decide upon the hash function $h$ and the similarity measure $\varphi$, and to tune their parameters, we compiled a ground truth by applying the subsequent step of text alignment directly, extracting the mutual reuse of each of 1000 long Wikipedia articles from all other articles. This way, $h$, $\varphi$, and hence $\rho$ are optimized against our text alignment algorithm so as not to prevent its detections. We considered two hashing schemes for $h$: (1) random projections, an instantiation of the data-independent locality-sensitive hashing (LSH) family [4], and (2) variational deep semantic hashing (VDSH), a data-dependent learning-to-hash technique [3]. We further considered four text similarity measures for $\varphi$: (a) cosine similarity on a $\mathit{tf}\kern-1.49994pt\cdot\kern-1.49994pt\mathit{idf}$-weighted unigram representation, (b) Jaccard similarity on stop word $n$-grams [15], (c) cosine similarity on a simple additive paragraph vector model [12], and (d) a weighted average of (b) and (c).

Table 1 (middle) shows our evaluation results for the two components of the source retrieval pipeline. For search pruning, we selected VDSH with a 16-bit hash, which reduces the number of required evaluations of the $\rho$ measure by three orders of magnitude compared to an exhaustive comparison, while retaining the majority of text reuse cases. To construct the ranking function $\rho$ itself, we settle on cosine similarity in the $\mathit{tf}\kern-1.49994pt\cdot\kern-1.49994pt\mathit{idf}$ space as the similarity measure $\varphi$ for its superior recall compared to all other models.

3.2 Text Alignment

Given a candidate document pair, text alignment extracts spans of reused text—if any—through the steps seed generation (identification of short exact matches), seed extension (clustering of matches to form spans), and post filtering. The state of the art evaluated at PAN is determined on datasets orders of magnitude smaller than our setting, often using complex setups that turned out to be difficult to scale and to be reproduced, while lacking open source implementations. We hence resorted to ideas from the literature that offer a reasonable trade-off between performance, robustness, and speed, and tuned their parameters222We employed word 3-gram seeds, extended via DBScan clustering ($\varepsilon=150$ and $\mathrm{minPoints}=5$), and filtered cases less than 200 words long or 0.5 cosine similarity. based on the standard PAN-13 training data. Our text alignment achieves a macro-averaged plagdet score of 0.64 (0.84 on just the unobfuscated subset) on the corresponding PAN-13 test data. In terms of raw detection performance, this is in the lower middle range of PAN results [14].

In our pruned all-pairs search setting, each input record to the text alignment step specifies the candidate pairs involving one document $d_{i}$ from collection $D_{1}$, in the form of a list of all candidate documents from the other collection $D_{2}$ sorted by descending $\rho$-score. Text alignment is applied sequentially to this list until one of two stopping criteria is met: (1) the current candidate pair is below the $\rho$-threshold (0.025 in our implementation), or (2) another threshold on the number of consecutive miss-cases—i.e., candidate pairs in which the text alignment finds no reuse—is exceeded (we use 250). Both thresholds can be configured based on the time available for the task; our values were determined experimentally using the aforementioned ground-truth sample.

4 Corpus Analysis

Table 1 (right) shows basic statistics of the text reuse we uncovered; most interestingly, we find nearly 70 times more reuse cases within Wikipedia than without, but involving only one third as many articles. Based on this insight, we identify two fundamentally different kinds of text reuse within Wikipedia, the first of which makes up for the bulk of this discrepancy. When articles use the same text structure, but different facts—e.g., when geographical locations are described in terms of their surroundings—we refer to this as structure reuse (Table 2, top left) and consider it non-problematic, and perhaps unavoidable redundancy. Resulting from Wikipedia’s editorial process, structure reuse is much more likely to occur within Wikipedia than without. On the other hand, articles may contain factually nearly-identical passages, likely after copying from one to the other. We consider such content reuse likely to result in inconsistency and contradiction as the articles diverge over time (Table 2, bottom left). Ideally, such redundant sections should be replaced with a single, authoritative source. In this sense, text reuse analysis can help the Wikipedia community locate and improve articles with undesirable redundancy.

Our current observations indicate that the ontological relationship between articles’ topics correlates with the type of text reuse: Structure reuse occurs more frequently when articles represent concepts on the same level in the ontology tree (Table 2, top right), while two articles whose subjects are vertically aligned—as is the case with “is a” or “part of” relationships—are more likely to exhibit content reuse (Table 2, bottom right). The latter association can also be envisioned as a solution to the sub-article matching task [9]: the occurrence of content reuse between articles can serve as an indicator of the ontological relationship between the concepts that they represent. However, distinguishing content and structure reuse automatically is not trivial. Our initial attempt at classifying reuse cases used heuristics based on the ratio of reused to original text in the articles, as well as the Jaccard similarities between the sets of named entities and word 10-grams. Based on two samples of 100 random cases classified as each, this yields 100% precision for structure reuse, but only 57% for content reuse. While our heuristics identify 95.5 million (87%) of all within-Wikipedia cases as structure reuse, the true number likely exceeds 100 million assuming our error estimates are accurate.

In the Common Crawl sample we examined, 4,898 websites host at least one page that reuses text from a Wikipedia article for a total of 1.6 million cases.333The top three being wikia.com (563), rediff.com (55), and un.org (28 reusing pages). We presume that Wikipedia’s policy of avoiding reuse from third parties inside its articles is enforced by its editors, so that nearly all of the cases will be third parties reusing Wikipedia’s articles instead. Most (94%) of the pages violate the terms of Wikipedia’s license444https://en.wikipedia.org/wiki/Wikipedia:Reusing_Wikipedia_content by not referencing Wikipedia as a source (i.e., the term “Wikipedia” does not occur). With only a handful exceptions, such as un.org, all of the sites display advertisements, which extends to the pages containing the reuse. Furthermore, in nearly all of the cases, the reuse accounted for more than 90% of the main content, begging the question of their usefulness.

To quantify the degree to which sites reusing Wikipedia content may be profiting from this reuse, we conservatively estimate the potential advertisement revenue generated by the reused content. We make the simplifying assumption that all reusing websites host only one ad per page and that advertisements are billed according to cost per mille (CPM), achieving a revenue per mille (RPM) of about half (1.4 USD) the average estimated CPM on the web in 2018 (2.8 USD).555https://monetizepros.com/cpm-rate-guide/display/ Accounting for the fact that reusing pages are generally ranked lower than Wikipedia in search results, we use 10% of the monthly page view counts on reused articles (as per Wikipedia’s API) as estimates for the page views of reusing pages. With these approximations, we arrive at a lower bound estimate of 45,000 USD monthly ad revenue generated by these 4,898 sites. Extrapolated to the entire web (say, 600,000 reusing sites out of 180 million active sites as per netcraft.com), we arrive at 5.5 million USD estimated monthly ad revenue; which adds up to about 72% of Wikipedia’s last year’s worldwide fundraising returns.

5 Conclusion

In an effort to bring text reuse analysis to very large corpora, we propose a scalable pipeline comprising the source retrieval and text alignment subtasks. We address challenges of scale primarily in the former by way of candidate filtering, and evaluate a set of hashing and text similarity techniques for this purpose. Using our framework, we compile two text reuse datasets—within and without Wikipedia—which we make publicly available for further research into text reuse phenomena. We hope our data will stimulate future research targeting Wikipedia quality improvement—e.g., by template induction or automatic detection of inconsistency—and understanding Wikipedia’s influence on the web at large.