Feature Analysis for Assessing the Quality of Wikipedia Articles through Supervised Classification

TL;DR

This paper investigates the automatic assessment of Wikipedia article quality using handcrafted features and supervised machine learning, aiming to improve content verification and reduce misinformation.

Contribution

It introduces a comprehensive set of features for classifying Wikipedia articles by quality, expanding on prior work with detailed feature analysis and evaluation.

Findings

Supervised classifiers achieved encouraging accuracy.

A wider set of features improved classification performance.

Results demonstrate the effectiveness of handcrafted features.

Abstract

Nowadays, thanks to Web 2.0 technologies, people have the possibility to generate and spread contents on different social media in a very easy way. In this context, the evaluation of the quality of the information that is available online is becoming more and more a crucial issue. In fact, a constant flow of contents is generated every day by often unknown sources, which are not certified by traditional authoritative entities. This requires the development of appropriate methodologies that can evaluate in a systematic way these contents, based on `objective' aspects connected with them. This would help individuals, who nowadays tend to increasingly form their opinions based on what they read online and on social media, to come into contact with information that is actually useful and verified. Wikipedia is nowadays one of the biggest online resources on which users rely as a source of information. The amount of collaboratively generated content that is sent to the online encyclopedia every day can let to the possible creation of low-quality articles (and, consequently, misinformation) if not properly monitored and revised. For this reason, in this paper, the problem of automatically assessing the quality of Wikipedia articles is considered. In particular, the focus is on the analysis of hand-crafted features that can be employed by supervised machine learning techniques to perform the classification of Wikipedia articles on qualitative bases. With respect to prior literature, a wider set of characteristics connected to Wikipedia articles are taken into account and illustrated in detail. Evaluations are performed by considering a labeled dataset provided in a prior work, and different supervised machine learning algorithms, which produced encouraging results with respect to the considered features.

Tables

Table 1. TABLE I: Experiment 1 - Accuracy (higher is better)

Classifier	Proposed Approach	Baseline [10]
DT	0.474	0.484
KNN	0.424	0.417
LR	0.497	0.499
NB	0.304	0.310
RF	0.592	0.601
SVC	0.506	0.539
NN	0.503	0.490
GB	0.618	0.599

Table 2. TABLE II: Experiment 1 - MSE (lower is better)

Classifier	Proposed Approach	Baseline [10]
DT	1.773	1.774
KNN	2.123	2.059
LR	1.359	1.411
NB	3.573	3.438
RF	1.167	1.073
SVC	1.358	1.426
NN	1.204	1.353
GB	0.919	1.029

Table 3. TABLE III: Experiment 2.1 - Accuracy (higher is better)

Classifier	TF	RF	NF
DD	0.378	0.316	0.297
KNN	0.415	0.287	0.303
LR	0.469	0.395	0.332
NB	0.3	0.255	0.198
RF	0.502	0.391	0.373
SVC	0.46	0.378	0.331
NN	0.481	0.392	0.373
GB	0.514	0.393	0.354

Table 4. TABLE IV: Experiment 2.1 - MSE (lower is better)

Classifier	TF	RF	NF
DD	2.085	3.013	3.607
KNN	2.101	4.396	3.956
LR	1.481	2.797	3.582
NB	3.467	6.413	9.055
RF	1.352	2.204	2.790
SVC	1.522	3.162	3.846
NN	1.284	2.665	2.854
GB	1.171	2.182	2.968

Full text

Wikipedia is nowadays one of the biggest online resources on which users rely as a source of information. The amount of collaboratively generated content that is sent to the online encyclopedia everyday can let to the possible creation of low-quality articles (and, consequently, misinformation) if not properly monitored and revised. For this reason, in this paper, the problem of automatically assessing the quality of Wikipedia articles is considered. In particular, the focus is on the analysis of hand-crafted features that can be employed by supervised machine learning techniques to perform the classification of Wikipedia articles on qualitative bases. With respect to prior literature, a wider set of characteristics connected to Wikipedia articles are taken into account and illustrated in detail. Evaluations are performed by considering a labeled dataset provided in a prior work, and different supervised machine learning algorithms, which produced encouraging results with respect to the considered features.

I Introduction

The development of the Web and digital technologies have allowed to considerably reduce the costs of production and the breakdown of information while increasing the ease of access. Web 2.0 technologies, in particular, have given everyone the chance to generate and spread content online, in most cases without the intermediation of any traditional authoritative entity in charge of content control [16]. This augments the probability for people to incur into misinformation (e.g., opinion spam, fake news, …) [33], or low-quality information [2]. In the online scenario, traditional methods to estimate information quality – such as the scrupulous analysis of contents by experts – have become impractical, due to the huge amount of new content that is generated and shared every day on the Web. Therefore, it is necessary to design scalable and inexpensive systems to automatically estimate the quality of the information diffused, based on ‘objective’ evidence. In this way, it could be possible to support users in relying on ‘certified’ information, in an era where people increasingly form their opinions based on what they read online and on social media [30].

One of the main sources of knowledge freely accessible and editable by users is today the online encyclopedia Wikipedia.111https://www.wikipedia.org/ The peculiarity of the platform, i.e., the fact that it allows anyone to create and modify articles, constitutes both a strength and a weakness: on the one hand, this encourages the collaborative construction of knowledge, but, on the other hand, this can lead to the possible generation of low-quality or biased articles. To overcome this problem, groups of volunteers periodically monitor the content of Wikipedia articles, but their limited number confronted with the articles growth rate do not allow an overall and constant control. Furthermore, the subjectivity connected to human assessors results in a different quality evaluation for different articles belonging to distinct topic areas. All these open issues have to be tackled for maintaining the authoritativeness of the platform. In this context, the proposed approach aims at automating the classification of Wikipedia articles on qualitative bases, by employing supervised learning. The approach focuses, in particular, on the analysis of multiple hand-crafted features that can be employed by well-known machine learning techniques, some of which previously applied in the literature. An in-depth analysis has been performed both on the syntax, the style and the editorial history of Wikipedia articles, and on the Wikipedia classification process, highlighting very relevant aspects not previously treated in the literature. For evaluation purposes, a labeled dataset generated and made publicly available in a previous work has been employed [1].

II Background: Wikipedia

Wikipedia is a collaborative encyclopedia, launched on January 15, 2001 by Jimmy Wales and Larry Sanger. The idea of a collaborative online encyclopedia is antecedent; in 1993, Rick Gates proposed this model under the name of Interpedia [28]. Wikipedia is to all effects a ‘spiritual’ successor of this proposal. The philosophical concept at the bases of Wikipedia was proposed by Richard Stallman in December 2000, in contrast to the digital encyclopedias then existing, and it is based on the idea that no centralized organization should have control over the editing process.222https://en.wikipedia.org/wiki/History_of_Wikipedia On Wikipedia, users, besides being able to freely read articles, can also modify them and create new ones. This poses the responsibility for the quality of contents totally in the hands of those who contribute to them. From a purely theoretical point of view, the collaborative aspect of Wikipedia should avoid, on the one hand, the prevalence of an individual’s point of view on those of others, and, on the other hand, bias and manipulation, since contents are subjected to control and revision by a large number of people from all over the world, resulting in a high quality of articles. Unfortunately, from a practical point of view, things are not so simple.

According to Alexa Internet, a company that analyzes Web traffic, Wikipedia is the fifth most visited Website in the world in 2018.333https://www.alexa.com/topsites In the first six months of 2018 more than 250 million English Wikipedia pages were visited on a daily basis.444https://tinyurl.com/yb5jfzdq As of August 2018, the English version of Wikipedia is constituted by 5.7 million articles and 650 new articles are created per day, while more than 3 million changes are made to existing articles monthly.555https://stats.wikimedia.org/EN/ReportCardTopWikis.htm#lang_en It is therefore impossible for the 1,200 administrators of the English version to monitor the entire publishing activity carried out by editors.666https://en.wikipedia.org/wiki/Special:Statistics The huge flow of new information that everyday characterizes the publishing activity on Wikipedia, has the defect of introducing into the platform a huge number of just sketched or low-quality articles, whose utility for the users appears to be doubtful and could also cause, in the worst case, the proliferation of misinformation among the less attentive readers. To cope with this problem and to indicate to those who contribute to the platform the qualitative status of an article, the Editorial Team of Wikipedia has defined some characteristics that an article should have in order to be considered of good quality,777https://en.wikipedia.org/wiki/Wikipedia:Content_assessment#Grades and distinct quality classes in which each article can be categorized based on its characteristics.

II-A The Wikipedia Quality Grading Scheme

Wikipedia is characterized, within its community, by groups of contributors, called WikiProjects, which are focused on improving the articles belonging to particular topic areas (e.g., Mathematics, History, etc.). Today, the English version of Wikipedia has more than two thousand of these groups.888https://en.wikipedia.org/wiki/Wikipedia:WikiProject Within each WikiProject, a so-called assessment team deals with the evaluation of the quality of the articles. The qualitative evaluation of an article is, in line with the philosophy of the platform, an activity that every contributor can perform. According to Wikipedia,999https://en.wikipedia.org/wiki/Wikipedia:Content_assessment “generally an active project will develop a consensus, though be aware that different projects may use their own variation of the criteria more tuned for the subject area”. The evaluations provided by contributors are used primarily for internal uses of the project, but they can also be accessed by readers to verify that the quality of an article is high before trusting its content.

This evaluation relies on the WikiProject article quality grading scheme,101010https://en.wikipedia.org/wiki/Template:Grading_scheme which divides the articles into seven distinct categories: $(i)$ Featured Articles, denoted as FA-Class (FA) articles; $(ii)$ A-Class (A) articles; $(iii)$ Good Articles, denoted as GA-Class (GA) articles; $(iv)$ B-Class (B) articles; $(v)$ C-Class (C) articles; $(vi)$ Start-Class (Start) articles; $(ii)$ Stub-Class (Stub) articles. The FA-Class includes the best articles on the platform, i.e., those considered complete and exhaustive from every point of view. In contrast, the Stub-Class includes all those articles that have a very basic description of the topic they deal with, or which are of very low quality. Intermediate classes are quality decreasing compared to the order in which they were previously listed.

Unfortunately, the number of new Wikipedia articles created every day and the number of changes made to existing ones makes it impossible the monitoring of their content by a small group of people, and the frequent verification and update of the quality classes they belong to. To overcome this issue, over the years, several works tackling the problem of automatically categorizing Wikipedia articles with respect to the above-mentioned quality classes have been proposed. The are illustrated in the following section.

III Background and Related Work

The relevance of the information quality problem, the widespread use of Wikipedia and its collaborative nature have stimulated different studies aimed at automatically identifying the quality of its articles. The majority of these studies have addressed this issue as a classification task, and have proposed different approaches to categorize Wikipedia articles into quality classes. The first approaches employing machine learning algorithms to perform classification, inferred evidence of the quality of articles only by considering text features, i.e., features connected to the length of the text [5], the language usage [25], or some lexical aspects [35]. Other works have proposed graph-based models to estimate a quantitative value representing the quality of an article [20, 21, 23]. These models consider and combine different metrics related to both: $(i)$ the graph representing the editorial process of the articles, highlighting the relationships (edges) between articles and editors (nodes); $(ii)$ the graph representing links (edges) among articles (nodes), i.e., the Wikipedia articles graph. In general, the models proposed within this group evaluate both authors authority and articles quality. Another category of approaches employs (supervised) machine learning techniques acting on multiple kinds of features, encompassing text features and other features related to the writing style, the readability level, the analysis of the article structure, and other network-related metrics, to perform classification [10, 11, 27, 32].

Recently, a few approaches based on the use of Deep Learning have been proposed [12, 13]. These approaches have proven to be effective in classifying Wikipedia articles over quality classes. Despite this, they do not involve hand-crafted feature analysis, while the aim of this paper is to study and investigate the impact that specific groups of features have in assessing information quality on Wikipedia. The next section, which represents the core of the paper, is devoted to the description and the analysis of the features that have been considered.

IV Feature Analysis

The choice of the features used to represent the elements on which to perform an automatic classification, is a fundamental operation in the majority of data-driven approaches [18]. With respect to the issue of classifying Wikipedia articles with respect to different quality classes, an initial work has been devoted to the study of previously employed features to represent Wikipedia articles. According to [10], three main groups of features can be identified: Text Features, Review Features, and Network Features. The former group includes features extracted directly from the text, the second derives them from the analysis of the writing process, and the latter from the analysis of the graph of the articles. In state-of-the-art approaches, the number of features employed was quite limited: the most complete study is [10], in which 69 features were considered. The present work extends the number of features that can be used for the classification of Wikipedia articles based on their quality, coming to consider 264 features. This consistent number derives in large part from an in-depth analysis carried out on the syntax used by the authors, in order to delineate, as precisely as possible, the use of language and the way in which sentences are constructed, and, therefore, the stylistic characteristics of the text. Furthermore, also the editorial history has been analyzed in depth, highlighting various aspects not yet considered in the literature, such as the contributions deriving from the changes made to the articles by occasional users. The set of features extracted from the graph of the articles was not expanded with respect to previous works.

In the following, the considered features are illustrated. Because of their high number, a synthetic description will be provided only when necessary, to explain the rationale behind their choice with respect to the problem considered. With respect to state-of-the-art features, new features introduced in this paper are indicated by an asterisk.

IV-A Text Features

Text Features are the characteristics of the articles that can be extracted directly from their text. These features allow to highlight different aspects of the articles, such as the writing style, the structure, and the lexicon used. For this reason, they can be further divided into four sub-categories: $(i)$ Length Features, connected to some length aspects of the articles; $(ii)$ Structure Features, capturing the way in which articles are structured (i.e., paragraphs, sub-paragraphs, etc.); $(iii)$ Style Features, highlighting the writing style and, therefore, the choices concerning the structuring of sentences and the use of the lexicon in drafting the articles; $(iv)$ Readability Features, indicating the degree of readability of the articles, i.e., the minimum scholastic level that is necessary to understand their contents.

In this paper, the study of new features focused on this area in particular, as the articles are mainly written texts and the textual characteristics turn out to be those that require less time and computational resources to be extracted. Moreover, these characteristics are, apart from rare cases, applicable in any context where there is a need to classify written texts on the basis of their quality, i.e., they are platform-independent.

IV-A1 Lenght Features

The length of an article can be an indicator of its quality. In fact, a good-quality text in a mature stage is reasonably neither too short (incomplete topic coverage), nor excessively long (verbose content). Further, in Wikipedia, Stub articles (draft quality) are short in the majority of the cases, reinforcing the correlation between length and quality [10]. In this work, the following features have been considered: $(1),(2),(3)$ Character [32]/Word [27]/Sentence [10] count: the number of characters (including spaces)/words/sentences in the text; in addition, the new feature (4) Syllable count* has been introduced, counting the number of syllables in the text.

IV-A2 Structure Features

This group of features focuses on the way an article is (well/badly) organized. According to the Wikipedia quality standards,111111https://en.wikipedia.org/wiki/Wikipedia:Version_1.0_Editorial_Team/

Release_Version_Criteria a good article must be reasonably clear, organized adequately, visually adequate, and point to appropriate references and/or external links. The considered structural features are listed in the following, where the majority of state-of-the-art ones come from [10], unless otherwise indicated: (1), (2) Section/Subsection count: the number of sections/subsections in the article. The intuition behind these features is that a good article is organized in sections (e.g., Introduction, Summary, List of references, and External links) and subsections; (3) Paragraph count*: the number of paragraphs constituting the article. The intuition behind this feature is that, in a high quality article, the text of sections and subsections should be further subdivided to facilitate the operation of reading and understanding the topics covered; (4), (5) Mean section/paragraph size; (6), (7) Size of the longest/shortest section, expressed in characters; (8) Longest-Shortest section ratio*. These features are useful to detect unusual section organization of articles with empty or very small sections, which could indicate incomplete content and drafts. Furthermore, (9) Standard deviation of the section size; (10) Mean of subsections per section; (11) Abstract size, expressed in characters. Mature articles are expected to have an introductory section summarizing its content; (12) Abstract size-Article Length ratio*: an article presenting an abstract whose length is very similar to its total length is probably incomplete. Features $(4)-(12)$ focuses on the correct balancing of an article. Other structure features are: $(13)-(15)$ Citation count/count per section/count per text length: the number of citations in the article/in sections/with respect to the total length of the article. A good-written article provides a sufficient and balanced number of citations; $(16)-(18)$ External link count [32]/links per section/links per text length: the same rationale behind features $(13)-(15)$; $(19),(20)$ Image count [32]/Images per section: the number of images in the text and the ratio between the number of images and sections. Pictures contribute to make content clearer and visually pleasant; $(21)$ Images per text length*: the ratio between the number of images and the length of the article, expressed in number of sentences.

IV-A3 Style Features

Aim of these features is to capture the writing style of contributors, i.e., “some distinguishable characteristics related to the word usage, such as short sentences” [10]. Many of the style features reported here below have been employed in [10]. When possible, for each feature the fist work having proposed it will be also indicated. The considered style features are: $(1),(2),(3)$ Mean [35]/Largest [27]/Shortest sentence size*: the average number of words per sentence/the number of words of the longest/shortest sentence; $(4),(5)$ Large/Short sentence rate [27]: the percentage of sentences whose length is ten words greater/five words lesser than the article average sentence length; $(6),(7)$ Question count [27]/ratio*: the number of questions in the article/the ratio between question count and the total number of sentences in the article; $(8),(9)$ Exclamation count/ratio**; $(10)-(17)$ Number of sentences that start with a pronoun / an article / a coordinating conjunction / subordinating preposition or conjunction [27]** / a determiner* / an adjective* / a noun*/ an adverb***; $(18)-(25)$ Number of sentences that start with a pronoun- / an article- / a coordinating conjunction- / a subordinating preposition or conjunction- / a determiner- / an adjective- / a noun- / an adverb-Sentence count ratio*: these features are built by considering the ratio between the value of features $(10)-(17)$ and the total number of sentences that make up the article; $(26)-(44)$ Number of modal auxiliary verbs / passive voices* [27]** / ‘to be’ verbs* / different words* / nouns* / different nouns* / verbs* / different verbs* / pronouns** [10] ** / different pronouns* / adjectives* / different adjectives* / adverbs* / different adverbs* / coordinating conjunctions* / different coordinating conjunctions* / subordinating prepositions and conjunctions* / different subordinating prepositions and conjunctions*** in the whole article; features $(45)-(62)$, which is a whole new group of features with respect to the literature, are the same as the group $(26)-(44)$ but computed per each sentence constituting the article; $(63)-(80)$ Ratio between the number of modal auxiliary verbs / passive voices / ‘to be’ verbs** [27] / different words [35] / nouns [35] / different nouns / verbs* [35] / different verbs / pronouns / different pronouns* / adjectives* / different adjectives* / adverbs* / different adverbs* / coordinating conjunctions* / different coordinating conjunctions* / subordinating prepositions and conjunctions* / different subordinating prepositions and conjunctions* and the total number of words in the article**; $(80)-(82)$ Ratio between the number of modal auxiliary verbs / passive voice count / ‘to be’ verb* and the total number of verbs in the article**; $(83)-(89)$ Ratio between the number of different nouns [35] / different verbs [35] / different pronouns / different adjectives / different adverbs* / different coordinating conjunctions* / different subordinating prepositions and conjunctions* and the total number of different words in the article**; $(90)-(91)$ Average number of syllables/characters per words; $(92)$ Top-$m$ most discriminant character trigrams [25]: they unveil the preferences of the authors for sentence transitions, as well as the utilization of stop-words, adverbs, and punctuation; $(93)$ Top-$n$ most discriminant POS trigrams: they unveil the preferences of authors in constructing sentences [25]. To compute $m$ and $n$ for features $(92)$ and $(93)$, the ${\chi}^{2}$ statistical method provided by the Python library scikit-learn has been employed.121212http://scikit-learn.org/stable/modules/generated/sklearn.

feature_selection.chi2.html

IV-A4 Readability Features

These features are numerical indicators of the US grade level,131313https://en.wikipedia.org/wiki/Educational_stage#United_States i.e., the comprehension level that a reader must possess to understand what is debated in a piece of text. They were first used, to tackle the considered problem, in [27]. They comprise several metrics combining word counts, sentences, and syllables. The intuition behind these features is that “good articles should be well written, understandable, and free of unnecessary complexity” [10]. The set of features considered includes: $(1)$ Automated Readability Index [31]; $(2)$ Coleman-Liau Index [9]; $(3)$ Flesch Reading Ease [17]; $(4)$ Flesch-Kincaid Grade Level [29]; $(5)$ Gunning Fog Index [19]; $(6)$ Läsbarhets Index [4]; $(7)$ SMOG-Grade [26]; $(8)$ Dale-Chall Readability Formula* [8]. This latter metric, not previously employed for the quality assessment of Wikipedia articles, has been designed to numerically evaluate the difficulty of understanding that a reader encounters when s/he reads a text in English.

IV-B Review Features

These features take their name from the fact that they are extracted from the review history of each article, i.e., how many times and in which way the article has been modified. They can measure the degree of maturity and stability of an article, since no extensive corrections could indicate good-quality articles having reached a maturity level, while a lack of stability could indicate different kinds of controversies (e.g., with respect to neutrality, correctness, etc.). In the following list of review features, registered users are those having an explicit user profile and a username, anonymous users are those identified only by their IP address, and occasional users are those who edited the article less than four times (they may belong to one of the two categories mentioned above).

The considered features are: $(1)$ Age [27]: the age (in days) of the article. Very recent articles are not normally considered of very high quality since they usually go through a refinement process; $(2)$ Age per review [10]: the ratio between age and number of reviews. It is used to verify the average period of time an article remains without revision; $(3)$ Review per day [10]: the percentage of reviews per day, to verify how frequently the article has been reviewed; $(4)$ Reviews per user [14]: the ratio between the number of reviews and the number of users. This feature is useful to infer how much reviewed is an article when contrasted against the number of reviewers; $(5)$ Reviews per user standard deviation [14]: this feature is useful to infer how balanced is the reviewing process among the reviewers; $(6)$ Discussion count [14]: the number of discussions posted by the users about the article. This is useful to infer conflict resolution and teamwork dynamics; $(7)$ Review count [24]: the total number of reviews. $(8)$ User count [24]: the total number of unique users that have contributed to the article. More contributors an article has, more objective its content is supposed to be; $(9)-(11)$ Registered / anonymous / occasional* user count**; $(12)-(14)$ Registered / anonymous / occasional user rate*: Percentage of registered / anonymous / occasional contributors; $(15)$ Registered/Anonymous user ratio*: the ratio between registered and anonymous contributors; $(16)-(18)$ Registered [32] / anonymous [32] / occasional review count* [32]: the umber of reviews made by registered / anonymous / occasional users; $(19-22)$ Registered / anonymous / occasional [14] review rate***: the percentage of reviews made by registered / anonymous / occasional users; $(22)$ Registered-Anonymous review ratio*: the ratio between reviews made by registered users and anonymous users. The previous four features aims to highlight a possible qualitative difference based on the ratios between the total number of reviews an article has and the reviews made by registered, anonymous and occasional users. $(23)$ Revert count [32]: the number of times an article has been taken to a previous state (review annulment); $(24)$ Reverts count-Review count ratio*: the ratio between reverts count and review count; $(25)$ Diversity [32]: the ratio between the total number of contributors and the number of reviews; $(26)$ Modified lines rate [10]: the number of lines modified when comparing the current version of an article with three-months older version. This is a good indicator of how stable an article is; $(27)$ Last three-months review count*: the number of reviews made in the last three months. This feature could indicate that the content of an article is controversial, the article is about evolving events or it is in the beginning of its editorial process; $(28)$ Last three-months review rate [14]: the percentage of reviews made in the last three months; $(29)$ Most active users review count*: the number of reviews made by the most active 5% of users; $(30)$ Most active users review rate [14]: the percentage of reviews made by the most active 5% of users; $(31)$ ProbReview [20]: this measure tries to assess the quality of a Wikipedia article based on the quality of its reviewers. Recursively, the quality of the reviewers is based on the quality of the articles they reviewed.

IV-C Network Features

Network features are extracted from the articles graph, which is built by considering citations among articles. These citations can provide evidences of the popularity of the articles. In addition, a high-quality article is expected to be used as a reference point for articles dealing with interconnected topics. Extracting this kind of feature is particularly onerous, due the magnitude of the graph. For this reason, state-of-the-art features have been considered: $(1)$ PageRank [6]: the PageRank of an article, previously employed in [27]; $(2)$ In-degree [10]: the number of times an article is cited by other articles; $(3)$ Out-degree [14]: the number of citations of other articles; $(4)-(7)$ Assortativity in-in / in-out / out-in / out-out: “the ratio between the degree of the node and the average degree of its neighbors. The degree of a node is defined as the number of edges that point to it (in-degree) or that are pointed by it (out-degree)” [10]; $(8)$ Local clustering coefficient [34]: it aims at evaluating if an article belongs to a group of correlated articles; $(9)$ Reciprocity: the ratio between the number of articles that cite a specific article and the number of articles cited by that article; $(10)$ Link count [14]: the number of links to other articles. It differs from out-degree since it counts also links to articles that have not been written yet (red links);141414https://en.wikipedia.org/wiki/Wikipedia:Red_link $(11)$ Translation count [14]: the number of versions of an article in other languages. Features related to assortativity and clustering coefficient were proposed in [3, 7, 15] for spam detection in Web pages. They were previously used in [10].

V Experimental Evaluation

In this section, the effectiveness of the considered features is evaluated by testing different supervised machine learning classifiers, by employing the labeled datasets introduced in [1] and detailed in the following.

V-A The employed dataset

In the literature, prior supervised approaches have classified the articles of Wikipedia with respect to a subset of the quality classes considered in this article (illustrated in Section II-A). In fact, none of the previous studies have referred to the current scale, for different reasons: $(i)$ when the study was made the proposed quality scale was different [10]; $(ii)$ the authors decided to simplify the classification task on purpose. In [27] the Stub-Class (drafts) has not been considered because it was believed to be too trivial to discern articles belonging to that class. In [35] the authors consider only Featured Articles and Start-Class articles. In [5, 25, 35] the articles have been classified only as Featured Articles and Random Articles (or non-Featured Articles); $(iii)$ the approaches did not perform a ‘real’ classification into quality classes, but they provided a ranking of articles with respect to their quality. The ranking produced by [20] is supposed to reflect the hierarchy of the quality classes: “the perfect ranking should place all FA-Class articles before all A-Class articles, followed by all GA-Class articles and so on”, while in [23] the ranking is intended to identify relevant VS non-relevant articles, i.e., Featured VS non-Featured Articles.

Therefore, it has been necessary to proceed with the construction of a new dataset, as illustrated in [1], by selecting labeled articles from the seven considered quality classes directly from Wikipedia. In doing so, it has been necessary to consider that the guidelines for the classification provided by the Wikipedia Editorial Team are generic, and need to be specialized/refined depending on the topic area of interest. For example, a high-quality article discussing Photography is expected to have more images than one of similar quality dealing with Computer Science, while a History article is supposed to contain more dates with respect to Technology ones, which probably will contain more technical details. The linguistic register is another aspect that it is influenced by thematic areas: a high-quality Economy article will be characterized by a more complex lexicon compared to the one employed in a Literature for children article. For this reason, different labeled datasets must be generated for different topic areas, as done in [20, 23, 25]. In this paper, in order to be able to experimentally evaluate the effectiveness of the features illustrated in Section IV through different supervised classification strategies detailed in Section V-B, the labeled dataset generated in [1] for the topic area Military History has been employed. The dataset consists of 400 articles randomly selected for each quality class, for a total of 2,800 articles. The dataset is therefore balanced with respect to classes. The dataset is publicly available, and further details are included in the documentation associated with the dataset.151515http://www.ir.disco.unimib.it/WikipediaDataset/

V-B Experiments and results

Two experiments are illustrated in this section. Each experiment tests eight different classifiers based on distinct supervised machine learning techniques: Decision Tree (DT), K-Nearest Neighbors (KNN), Logistic Regression (LR), Naive Bayes (NB), Random Forest (RF), Support Vector Classifier (SVC), Neural Networks (NN) and Gradient Boosting (GB). In each experiment, the performance of classification has been evaluated in terms of Accuracy and Mean-Squared Error (MSE) [22].

V-B1 Experiment 1

In the first experiment, a comparative evaluation has been performed between the proposed approach and the state-of-the-art approach described in [10], which employed supervised classifiers acting on the higher number of hand-crafted features among prior works in the literature. This experiment allows to evaluate the effectiveness of the features analyzed in this paper and those proposed by the baseline in classifying Wikipedia articles with respect to the seven quality classes (i.e., FA-Class, A-Class, GA-Class, B-Class, C-Class, Start-Class, and Stub-Class).

As reported in Table I and Table II, the set of features analyzed in this paper in conjunction with Gradient Boosting allow to obtain the best results in terms of both Accuracy ($61.8\%$) and MSE ($0.919$), with an improvement in terms of both measures with respect to [10].

V-B2 Experiment 2

The second experiment consists in the classification of Wikipedia articles w.r.t. the seven quality classes by considering, in turn, only the features belonging to each of the three groups in which they can be categorized, i.e., Text Features (TF), Review Features (RF) and Network Features (NF). This experiment aims to identify which group of features is the most discriminating one.

In Table III and Table IV the comparison between the accuracy and MSE values obtained with respect to each group of features are respectively reported.

As it emerges from both tables, Network Features apper to be the less effective, while Text Features provide the best level of Accuracy and MSE by employing Gradient Boosting. In particular, TF+GB provides an Accuracy value of $51.4\%$, and an MSE value of $1.171$.

In [1], other experiments have been provided with respect to other baselines and with respect to the the negative impact that possible noise in the generation of the labeled dataset can have on the classification task.

VI Conclusions

In this paper, the problem of assessing the quality of Wikipedia articles has been considered. The collaborative nature at the basis of Wikipedia, where everyone can freely contribute to the creation or modification of articles, represents both an advantage but also a possible drawback of the platform. In fact, if from the one side this could guarantee heterogeneity of sources, skills, and control by a large number of contributors, from the other side it could result in the proliferation of unverified and low-quality contents.

For this reason, in the last years, automatic approaches for the classification of Wikipedia contents with respect to given quality classes have been proposed. Most solutions are based on supervised learning techniques, employing multiple kinds of features connected to different aspects of the articles and their authors. With respect to state-of-the-art approaches, in this paper a solution considering a higher number of features has been proposed. The choice of the features is based on an in-depth analysis that encompass the syntax, the style and the editorial history of Wikipedia articles, as well as on a deep investigation of the way in which Wikipedia articles are labeled by WikiProject teams with respect to quality. The promising results obtained confirm the effectiveness of the proposed feature analysis and the interest in continuing the study of the problem.