Identifying Duplicate and Contradictory Information in Wikipedia
Sarah Weissman, Samet Ayhan, Joshua Bradley, and Jimmy Lin
TL;DR
This paper presents a scalable method using minhash and MapReduce to detect duplicate and contradictory sentences in Wikipedia, revealing content copying and quality issues.
Contribution
It introduces a novel application of near-duplicate detection techniques to Wikipedia, categorizing sentence clusters and highlighting quality concerns.
Findings
High prevalence of copied content in Wikipedia
Presence of contradictory statements indicating quality issues
Effective clustering of similar sentences using minhash
Abstract
Our study identifies sentences in Wikipedia articles that are either identical or highly similar by applying techniques for near-duplicate detection of web pages. This is accomplished with a MapReduce implementation of minhash to identify clusters of sentences with high Jaccard similarity. We show that these clusters can be categorized into six different types, two of which are particularly interesting: identical sentences quantify the extent to which content in Wikipedia is copied and pasted, and near-duplicate sentences that state contradictory facts point to quality issues in Wikipedia.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Wikis in Education and Collaboration