Using Micro-collections in Social Media to Generate Seeds for Web Archive Collections

TL;DR

This paper explores the use of social media micro-collections as a novel source for generating seed URIs to improve web archive collections, analyzing their characteristics and comparing them to traditional sources.

Contribution

It introduces the concept of social media micro-collections as a new seed source and provides a detailed analysis of their characteristics and effectiveness compared to conventional methods.

Findings

Generated 23,112 seed collections from social media posts.

Collected 120,444 URIs from various sources including micro-collections.

Characterized seed collections across multiple dimensions such as URI distribution and diversity.

Abstract

In a Web plagued by disappearing resources, Web archive collections provide a valuable means of preserving Web resources important to the study of past events ranging from elections to disease outbreaks. These archived collections start with seed URIs (Uniform Resource Identifiers) hand-selected by curators. Curators produce high quality seeds by removing non-relevant URIs and adding URIs from credible and authoritative sources, but it is time consuming to collect these seeds. Two main strategies adopted by curators for discovering seeds include scraping Web (e.g., Google) Search Engine Result Pages (SERPs) and social media (e.g., Twitter) SERPs. In this work, we studied three social media platforms in order to provide insight on the characteristics of seeds generated from different sources. First, we developed a simple vocabulary for describing social media posts across different platforms. Second, we introduced a novel source for generating seeds from URIs in the threaded conversations of social media posts created by single or multiple users. Users on social media sites routinely create and share posts about news events consisting of hand-selected URIs of news stories, tweets, videos, etc. In this work, we call these posts micro-collections, and we consider them as an important source for seeds because the effort taken to create micro-collections is an indication of editorial activity, and a demonstration of domain expertise. Third, we generated 23,112 seed collections with text and hashtag queries from 449,347 social media posts from Reddit, Twitter, and Scoop.it. We collected in total 120,444 URIs from the conventional scraped SERP posts and micro-collections. We characterized the resultant seed collections across multiple dimensions including the distribution of URIs, precision, ages, diversity of webpages, etc...

Tables

Table 1. Table 1. Post Classes for Social Media Posts. All non- P 1 A 1 collections are combined to create Micro-Collections (MC).

Acronym	Post Count	Author Count	Definition/Example
P1A1	Single (1)	Single (1)	A Single Post from a single Author, e.g., an isolated tweet or post on Reddit (Fig. 2) or Facebook. These posts are visible to seeds generators that scrape SERPs.
P1An	Single (1)	Multiple (n)	Single Post from multiple Authors, e.g., the references contributed by multiple Wikipedia editors.
PnA1	Multiple (n)	Single (1)	Multiple Posts from a single Author, e.g., a thread of tweets (Fig. 1) from a Twitter user.
PnAn	Multiple (n)	Multiple (n)	Multiple Posts from multiple Authors, e.g., a tweet conversation consisting of multiple tweets or posts from different Twitter or Reddit (or Facebook) users.

Table 2. Table 2. Temporal characteristics of dataset topics

			Occurrence definition
Topic [Wikipedia Page]	Expectation (Expected/Unexpected)	Recurrence (Recurring/Non-Recurring)	Start definition (Defined/Undefined)	End definition (Defined/Undefined)
Ebola Virus Outbreak (Wikipedia, 2014)	Unexpected	Recurring (Irregular)	December 2013 (Centers for Disease Control and Prevention (CDC), 2019)	June 2016 (Centers for Disease Control and Prevention (CDC), 2019)
Flint Water Crisis (Wikipedia, 2016)	Unexpected	Non-Recurring	March 2014 (Robbins, 2016)	Undefined
MSD Shooting (Wikipedia, 2017)	Unexpected	Non-Recurring	February 14, 2018	February 14, 2018
2018 World Cup (Wikipedia, 2018a)	Expected	Recurring	June 14, 2018	July 15, 2018
2018 Midterm Elections (Wikipedia, 2018b)	Expected	Recurring	November 6, 2018	November 6, 2018

Table 3. Table 3. Post class counts (Class), Social media posts (Posts), and URI counts (URIs) for dataset generated by extracting URIs from post classes ( P 1 A 1 , P n A 1 , and P n A n ) of Reddit, Twitter, Twitter Moments, and Scoop.it. The Micro-collection (MC) post class is formed by combining posts in P n A 1 and P n A n post classes.

				Micro-collections (MC)
	P1A1 Counts			PnA1 Counts			PnAn Counts
	Class	Posts	URIs	Class	Posts	URIs	Class	Posts	URIs
Reddit Relevance	766	766	1,776	56	115	206	542	36,124	3,387
Reddit Top	931	931	10,857	37	177	319	1,021	100,006	18,992
Reddit New	854	854	8,056	26	68	1,062	340	9,298	6,412
Reddit Comments	834	834	8,381	53	423	691	1,077	117,378	18,781
Twitter Top	2,936	2,936	3,548	540	4,983	3,026	4,009	79,347	12,457
Twitter Latest	2,341	2,341	2,792	639	6,366	3,628	4,471	82,499	13,576
Twitter Moments	NA	NA	NA	NA	NA	NA	73	1,285	621
Scoop.it	1,533	1,533	1,533	33	1,083	343	NA	NA	NA
Subtotal	10,195	10,195	36,943	1,384	13,215	9,275	11,533	425,937	74,226
Total	Class: 23,112			Posts: 449,347			URIs: 120,444

Table 4. Table 4. Probability (e.g., P ​ ( p P 1 ​ A 1 R ​ e ​ d ​ d ​ i ​ t = 1 ) 𝑃 superscript subscript 𝑝 subscript 𝑃 1 subscript 𝐴 1 𝑅 𝑒 𝑑 𝑑 𝑖 𝑡 1 P(p_{P_{1}A_{1}}^{Reddit}=1) = 0.63 ) of the event that a social media post from a given post class (e.g., Reddit P 1 A 1 ) has k 𝑘 k HTML URIs (e.g., k = 1 𝑘 1 k=1 ). See also Appendix 1 for additional figures of the distribution of the number of links in a post, per post class, per social media.

k	P1A1	MC	PnA1	PnAn	All	P1A1	MC	PnA1	PnAn	All	P1A1	MC	PnA1	All
1	.63	.23	.43	.22	.37	.98	.69	.60	.70	.75	1.00	.21	.21	.97
2	.11	.12	.13	.12	.12	.02	.17	.17	.17	.14	.00	.00	.00	.00
3-4	.06	.15	.09	.15	.12	.00	.08	.11	.08	.06	.00	.12	.12	.01
5+	.20	.50	.35	.51	.39	.00	.07	.12	.06	.05	.00	.67	.67	.03

Table 5. Table 5. Conditional probability (e.g., P ​ ( r ​ e ​ l ​ e ​ v ​ a ​ n ​ t | p P 1 ​ A 1 R ​ e ​ d ​ d ​ i ​ t = 1 ) 𝑃 conditional 𝑟 𝑒 𝑙 𝑒 𝑣 𝑎 𝑛 𝑡 superscript subscript 𝑝 subscript 𝑃 1 subscript 𝐴 1 𝑅 𝑒 𝑑 𝑑 𝑖 𝑡 1 P(relevant|p_{P_{1}A_{1}}^{Reddit}=1) = 0.64 ) of the event that the URIs in a social media post from a given post class (e.g., Reddit P 1 A 1 ) are relevant, given that the post has k 𝑘 k (e.g., k = 1 𝑘 1 k=1 ) HTML URIs. Column markers: minimum and maximum . See also Appendix 10 for additional figures of the average precision per count of links in a post, per post class, per social media. For the P 1 A 1 , k = 5 + 𝑘 limit-from 5 k=5+ Twitter cell, the probability was calculated for just one post with eight HTML URIs.

k	P1A1	MC	PnA1	PnAn	All	P1A1	MC	PnA1	PnAn	All	P1A1	MC	PnA1	All
1	.64	.54	.54	.54	.60	.63	.60	.49	.61	.61	.76	.00	.00	.76
2	.80	.59	.57	.59	.65	.50	.61	.64	.60	.60	NA	.00	.00	.00
3-4	.62	.45	.50	.44	.48	.33	.46	.51	.45	.46	NA	.50	.50	.50
5+	.51	.50	.53	.50	.50	1.00	.42	.46	.41	.42	NA	.59	.59	.59

Equations

$$P(p_c^s=k)=\sum _{t=1}^{|T|}\frac{p_{c_t}^s=k}{|c_t|}$$

$$P(p_{All}^s=k)=\sum _{t=1}^{|T|}\sum _{c\in C}\frac{p_{c_t}^s=k}{|c_t|}$$

Full text

Seeds; Collection building; Web Archiving; Social Media; Crawling.

1. Introduction and Background

In an attempt to save the digital history of unfolding world events before they are lost due to link rot (Klein et al., 2018b; Zittrain et al., 2014; Bar-Yossef et al., 2004), we often see the creation of Web archive collections following the occurrence of a major news event. For example, an Archive-It Ebola virus collection (National Library of Medicine, 2014) was created months after the 2014 Ebola outbreak. It consists of groups of webpages of government organizations and public health care workers associated with the Ebola outbreak event. However, some important events occur without the creation of Web archive collections. For example, on February 14, 2018, there was a shooting that claimed the lives of 17 people at the Marjory Stoneman Douglas (MSD) High School in Florida. In the aftermath of the tragic event, the teenage students boldly stepped into the highly politically divisive gun control debate demanding stricter gun control measures (Sean Rossman, 2018; Stephanie Ebbs, 2018). Less than two weeks after the shooting, major retailers Walmart and Dick’s Sporting Goods increased the minimum age required to purchase firearms and ammunition from 18 to 21 (Colin Dwyer, Camila Domonoske, and Emily Sullivan, 2018). Dick’s additionally discontinued the sale of assault-style rifles, high capacity magazines, and bump stocks. On March 9, 2018, Governor Rick Scott of Florida signed the Marjory Stoneman Douglas High School Public Safety Act bill into law. Among other gun control measures, it raised the minimum age for buying rifles to 21, banned bump stocks, and instituted background checks. The ripple effects of the activism of the MSD students is still being felt, and most would agree that this incident deserves highlight as part of the broader gun control discourse in the US, thus worthy of a Web archive collection. However, one year after the shooting there is still no corresponding Archive-It collection. Any subsequent collection would likely not be able to collect all the contemporary resources shared in social media or even reported in the news (SalahEldeen and Nelson, 2012; Nwala et al., 2018b).

The MSD shooting example illustrates gaps in Web archive collections for important events. One major reason for the lack of Web archive collections for important events is tied to how Web archive collections are created. Web archive collections begin with high quality seeds URIs (Uniform Resource Identifiers) selected by curators, a time consuming process often done manually. Amidst an abundance of important local and global events, various organizations such as the Internet Archive cope with the shortage of curators by routinely requesting (Fig. 3) for users to contribute links (seeds) to Archive-It collections, e.g., the 2012 Hurricane Sandy (Internet Archive, 2012), and the 2013 Boston Marathon Bombing (Internet Archive, 2013) collections. But this crowd-sourced approach to collection building, while useful, is not enough. In some other cases, archived collections are initiated months or years after the precipitating event. This could have serious consequences since Web archive collections that start late could omit webpages that address the early stages of events (Nwala et al., 2018b; Klein et al., 2018a). Consequently, it is important to start collecting seeds for Web archive collections early. This calls for a method for generating seeds automatically and on demand. Two prominent sources for automatically generating seeds have been adopted for extracting (scraping) links over the years: Web (e.g., Google, Fig. 4) and social media (e.g., Twitter, Fig. 5) Search Engine Result Pages (SERPs).

In this work, we explore a new source (we call micro-collections) for generating seeds beyond URIs returned by SERPs. It is important to note that our proposed method is not just concerned about finding what a search engine such as Google may find. Even though search engines often produce quality seeds, our micro-collection method of generating seeds is more concerned about finding quality, “hard-to-find,” and heterogeneous seeds that may not be popular enough to be easily retrievable through a simple Web search. We define a micro-collection as a post or group of social media posts that exhibit some properties associated with collection building. Web archive curators spend time selecting and filtering seed URI candidates. Similarly, social media users often perform similar tasks when faced with the decision of choosing what URIs to include in a “non-standard” social media post. For example, the Twitter account Doing Things Differently (@dtdchange) (Doing Things Differently, 2012) created a chain of tweets (Fig. 1 (Doing Things Differently, 2015)) by replying to each subsequent tweet in order to chronicle the Flint water crisis story. This reply thread spans almost 3 years and consists of 75 tweets (as of April 8, 2019) each containing a URI. These tweets exhibit curatorial discretion (selection and filtering), and thus we consider the thread a micro-collection for the Flint water crisis story. Another example of micro-collections are Reddit posts (Fig. 2 (Ilsensine, 2014a, b)) created by the user Ilsensine (Ilsensine, 2013) for the 2014 Ebola virus outbreak story. In total, the posts contain over 102 external references and were published less than two weeks after the World Health Organization (WHO) declared the 2014 Ebola outbreak a Public Health Emergency of International Concern (WHO, 2014). We distinguish micro-collections from standard social media posts by showing that micro-collections can be identified by considering the properties of the posts. The rationale for considering micro-collections as a good source for seeds is that the effort taken to create micro-collections is an indication of editorial effort and a demonstration of domain expertise.

Conventional techniques use SERPs from search engines and social media to extract seeds. Since seeds highly influence the nature of collections generated after the seeds are crawled, we consider it pertinent to understand the nature of the seeds returned from the services often used to generate seeds. Accordingly, we conducted a study to investigate the nature of the seeds generated from different sources on popular social media sites (Reddit and Twitter) and a less popular social media site (Scoop.it). First, we created a classification called post class from four pairs (P1A1, P1An, PnA1, PnAn - Table 1) of acronyms for identifying social media posts regardless of platform. A post class is formed by combining two acronyms, P and A, with subscripts (1 - single or $n$ - multiple), both combined to represent the count of Posts and Authors, respectively. Second, we generated 23,112 collections of seeds extracted from the various post classes by issuing five queries against the following social media sources: Reddit, Twitter (and Twitter Moments111A service launched by Twitter (in October 6, 2015) that enables users to collect and share tweets of noteworthy events as they unfold. A collection of tweets is called a moment.), and Scoop.it. In total we collected 120,444 URIs from 449,347 social media posts. Third, for a combination of social media and post classes, we studied the resultant collections across the following dimensions: the distribution of links, the probability distribution of URI counts for various post classes, the precision of the seeds, ages of webpages, the diversity of seed hostnames, and overlap with the Google SERPs.

Our study resulted in the following contributions that collectively provide some insight on the nature of seeds generated from various social media post classes. First, the provision of a simple cross-platform vocabulary (post class) for describing social media posts facilitates comparing different social media posts across different platforms, ranging from tweets (P1A1) on Twitter to Reddit (P1A1) posts.

Second, we introduced micro-collections (MCs) as social media posts that exhibit properties associated with collection building, and proposed generating seeds from them. MCs are formed by combining seeds from P1An, PnA1, and PnAn. Seeds generated from scraping SERPs belong to the P1A1 post class. We showed that seeds generated from social media sources are not easily discoverable from Google. Third, we provided a means of characterizing and comparing seeds generated from different post classes. Fourth, we showed that MCs produced more seeds than P1A1, but P1A1 had a higher median probability (0.63) of producing relevant URIs than MCs (0.5) for all social media and SERP combination excluding seeds generated with hashtags. Finally, we showed that the ages of webpages extracted depends on a combination of features such as the topic and SERP vertical (e.g., Top vs. New). Similarly, we showed that the diversity of seed hostnames varied with post classes. These findings may provide useful information to curators using social media to generate seeds. For example, if a seed generation process prioritizes quantity of URIs (HTML and non-HTML), the curator may consult MCs first. However, if precision is the priority, then P1A1. Our research dataset comprising of 120,444 links extracted from 449,347 social media posts, as well as the source code for the application utilized to generate the seeds, are publicly available (Alexander Nwala, 2019).

2. Related work

The collection building process starts with seeds. The seeds are fed into a focused crawler’s crawl frontier to start the process of discovering more Web resources related to the collection topic. Chakrabarti et al. (Chakrabarti et al., 1999) introduced the first focused crawler in the 1999s as a means to build collections for specific topics, as opposed to a general-purpose crawler which does not take the topics of the documents under consideration during the crawling process. Since the first focused crawlers, there have been many variants of focused crawlers. Bergmark (Bergmark, 2002) used a focused crawler to crawl and classify webpages into various topics in science, mathematics, engineering and technology, discarding off-topic pages. Farag et al. (Farag et al., 2017) introduced the Event Focused Crawler, a focused crawler for events that uses an event model to represent documents and a similarity measure to quantify the degree of relevance between a candidate URI and a collection. An event is represented as a triple - Topic, Location, and Date. Similar to Farag et al., Risse et al. (Risse et al., 2014) introduced a new crawler architecture based on the ARCOMEM project. Instead of the conventional crawling of all webpages, ARCOMEM performs a semantic crawl of only webpages related to events and entities such as persons, locations, and organizations. Most focused crawling is performed on the live Web. Unfortunately, the live web is plagued by link rot and content drift, consequently, Klein et al. (Klein et al., 2018a) demonstrated that focused crawling on the archived Web results in more relevant collections than focused crawling on the live Web, for events that occurred in the distant past. Additionally, Klein et al. proposed extracting seeds from external references contained in the Wikipedia page of an event. We consider Wikipedia references examples of P1An micro-collections. Our focus in this work is the seed generation process, therefore we did not utilize a focused crawler. Instead we explored the various sources for extracting seeds from social media posts.

Not all collection building uses focused crawling. Gossen et al. (Gossen et al., 2016) proposed a methodology for extracting sub-collections from Web archive collections focused on specific topics and events (called the topic and event focused sub-collection). The topic and event focused sub-collection is defined as a collection of documents in a Web archive collected using a sub-collection specification. Our research differs from Gossen et al. in two major ways. First, Gossen proposes generating collections from within the Web archives, but we propose generating seeds from the live social Web. Second, Gossen proposed running an algorithm over a sub-collection specification on a Web archive to generate a sub-collection. This means the decision of whether a URI belongs in a sub-collection is encoded in the specification of an algorithm. However, in this work, we leverage the judgment of humans on social media.

In a similar work, Gossen et al. (Gossen et al., 2017) adapted some portions of the topic and event focused sub-collection in a method to extract event-centric documents from Web archives based on a specialized focused extraction algorithm. They defined two broad kinds of events based on time: planned and unexpected. The goal of the event-centric extraction process is, given an event input and a Web archive, generate an interlinked collection of documents relevant to the input event that meet the collection specification. The differences of our research with Gossen’s previous work (Gossen et al., 2016) transfer to this work. However, we adapted Gossen’s categorization of events as either planned or unexpected, and we renamed planned to expected (Table 2). Similar to Gossen et al., Nanni et al. (Nanni et al., 2018) presented an approach for extracting event-centric sub-collection from Web Archives. Their method extracts documents not only related to the input event, but also documents describing related events (e.g., premises and consequences). Nanni et al.’s method utilized Wikipedia pages as inputs to generate event-centric collections. In this work, however, we used Wikipedia references to generate our gold standard dataset.

Selecting good seeds is challenging and has not been extensively studied. Collection building researches often acknowledge the importance of selecting good seeds, and admit its link to the performance of their systems, but often they pay more attention on the mechanisms of building the collection, and not seed selection. The challenge of selecting good seeds is embodied in the idea that it is difficult to define “good.” This challenge is captured by Bergmark’s statement (Bergmark, 2002): “It is unclear what makes a good seed URL, but intuitively it should be rich in links, yet not too broad in scope.” Zheng et al. (Zheng et al., 2009) argued that the seed selection problem for Web crawlers is not a trivial, and proposed different seed selection strategies based on PageRank, number of outlinks, and website importance. They also showed that different seeds may result in collections that are considered “good” or “bad.” While there have been efforts made to automatically generate seeds, many of these methods (e.g., Prasath and Öztürk (Prasath and Öztürk, 2011)) target generating seeds for Web crawlers that build indexes for search engines, and not seeds for focused crawlers or Web archive collections.

Du et al. (Du et al., 2014) proposed a customized method of generating seeds for focused crawlers based on user past Web usage information that captures the interests of the user. Since this method depends on historical use information, its performance is tied to the availability of such historical data, which might be lacking due to the absence of domain knowledge or privacy concerns. As part of the Crisis, Tragedy, and Recovery Network project, Yang et al. (Yang et al., 2012) proposed using URIs found in tweet collections (generated with hashtags and keywords) as seeds to bootstrap Web archiving tasks quickly for sudden emergencies and disasters. Similarly, we consider extracting seeds from tweets, but expand the areas for extracting seeds beyond scraping Twitter SERPs. Additionally, we identify post classes of tweets as part of an effort to characterize the nature of seeds generated from different post classes (Table 1). Priyatam et al. (Priyatam et al., 2014) proposed extracting diverse seeds from tweets in a Twitter URI graph for the Web crawlers of digital libraries such as CiteSeerX. Even though their work does not target the generation of seeds for collections of stories and events, which is a focus of our work, the notion of diversity of seeds is adopted in our work (Section 4.4.4).

In previous work (Nwala et al., 2018a), we showed that collections generated from social media sources such as Reddit, Storify, Twitter, and Wikipedia are similar to Archive-It collections across multiple dimensions such as the distribution of sources and topics, content and URI diversity, etc. These findings suggest that curators may consider extracting URIs from these sources in order to begin or augment collections about various news topics. Here, we adopt a subset of the dimensions for comparing collections. Similarly, in another previous work (Nwala et al., 2018b) as part of an effort to understand the behavior of SERPs, a popular source for generating seeds, we investigated “refinding” news stories on the Google SERP by tracking the URIs returned from Google, everyday for over seven months. We discovered that the probability of finding the same URI of a news story diminished drastically after a week (0.01 – 0.11). These findings suggest it becomes more difficult to find the same news story with the same query on the Google SERP. Therefore, collection building efforts that scrape SERPs are highly sensitive to the query issue dates.

3. Research questions

Before generating seeds from micro-collections, we must first identify them. This leads to our first research question:

• •

RQ1: How do we identify, extract, and characterize micro-collections in social media?

Identifying micro-collections makes it easier to accurately describe and extract them. Subsequently, it would be important to quantify the amount of micro-collections relative to conventional social media posts. As part of proposing the extraction of seeds from micro-collections, it is pertinent to verify if they are prevalent on the web.

There are currently two popular sources for automatically or semi-automatically generating seeds. The first involves extracting seeds from SERPs (e.g., Google). The second involves extracting seed URIs from tweets surfaced by hashtags or text queries on Twitter. We propose a third source for extracting seeds - extracting seeds from micro-collections. Therefore, it is important that we compare the new source to the previous popular sources. Such comparison could enable us understand if these sources are similar, and such information would be highly informative to future collection building processes. This leads to our second research question:

• •

RQ2: Do seeds from micro-collections differ from seeds from SERPs?

4. Methodology

Here we explain considerations made in the selection of our dataset topics, the dataset generation process, the measures extracted from the dataset and how they informed our research questions.

4.1. Topic selection

A central objective of our research was to outline the characteristics of, and differences between, collections generated by scraping SERPs (P1A1 post class - Table 1) and micro-collections (MC post class). Therefore, the choice of queries was not arbitrary. Instead, we developed a temporal classification system (partly informed by Gossen et al. (Gossen et al., 2017)) of real world stories and events based on three temporal (Table 2) attributes: Expectation, Recurrence, and Occurrence definition - Start and End date definitions. A story can be described by a combination of different states of the temporal attributes.

For the expectation attribute, an event may be expected or unexpected. For example, the Ebola outbreak event was unexpected. Thus we classify this event as an unexpected event. For the recurrence attribute, an event may occur repeatedly at regular or non-regular intervals. For example, the FIFA World Cup tournaments recur at four-year intervals, thus we consider this event a recurring event. Ebola outbreaks in general may also be considered a recurring event, even though they occur at irregular intervals. For the occurrence definition attribute, an event may have a defined or undefined start and end date. For example, the MSD Shooting event started and ended the same day (February 14, 2018), but the Flint water crisis event started in April 2014, and is still ongoing (no end definition).

Following the specification of the temporal classification system, we selected five topics (Table 2) specified by the following queries and hashtags (for Twitter):

1. (1)

“ebola virus outbreak” (#ebolavirus) 2. (2)

“flint water crisis” (#FlintWater) 3. (3)

“stoneman douglas high school shooting” (#MSDStrong) 4. (4)

“2018 world cup” (#WorldCup) 5. (5)

“2018 midterm elections” (#election2018)

In addition to text queries, for Twitter, we selected hashtag queries for each topic to discern if seeds generated with text-based queries differ from those extracted with hashtag queries.

4.2. Dataset generation and segmentation of social media posts into post classes

For Reddit, we issued all five queries to four Reddit SERPs (Relevance, Top, New, and Comments), and extracted posts from the SERPs. For each query we extracted a maximum of 500 posts and recursively extracted a maximum of 500 comment replies from each post extracted from the SERP.

For Twitter, similar to Reddit, we issued all five text and hashtag queries to the two Twitter SERPs (Top and Latest), and extracted tweets from the SERPs with the use of the Local Memory Project (Nwala et al., 2017) local news generator (Alexander Nwala, 2016). For each query, we extracted a maximum of 500 tweets and recursively extracted a maximum of 500 tweet replies for each tweet extracted from the SERP.

For Reddit and Twitter, the posts directly visible from the SERP were assigned to the P1A1 post class. We use the term “post” in order to be general. Different social media platforms have different names for posts, for example, on Twitter, a post is called a tweet. Posts with replies were assigned either to the PnA1 or PnAn class depending on the number of authors. Posts from the SERP with a reply or a contiguous set of replies exclusively authored by a single user were assigned to the PnA1 post class. Finally, posts with a reply or a series of replies authored by multiple users were assigned to the PnAn post class. The P1An micro-collection post class is rare and not available in Twitter, Reddit, or Scoop.it. However, our gold standard data was extracted from Wikipedia references which belong to P1An.

For Twitter Moments, we issued all five queries to Google with (“site:twitter.com/i/moments”) in order to restrict the search results to links from Twitter Moments. Next, we extracted Twitter Moments URIs from the first two pages of the Google default SERP. Next, we dereferenced URIs and extracted the tweets. Tweets from Twitter Moments are authored by multiple users, and thus assigned the PnAn label.

In addition to the extraction of posts from well-known social media (Reddit and Twitter), we considered a lesser known social media Scoop.it (https://www.scoop.it/). Scoop.it is a content curation social media service that enables users to bookmark a single URI (scoop) or multiple URIs (topics). For Scoop.it, we issued all five queries to the Scoop.it SERPs (Scoops and Topics), and extracted posts (scoops) from the SERPs. The scoops visible from the Scoops SERP were assigned to the P1A1 post class. For a single dataset topic, the scoops found in the Topic SERP were assigned to the PnAn post class since they are authored by multiple users.

From all social media posts, we extracted the URIs to create collections corresponding to the post class from which the URIs were extracted. Social media posts often link to intra-site posts (e.g., tweet URI in a tweet). We dereferenced and extracted seeds from such intra-site URIs, and substituted them with the extracted seeds.

4.3. Gold standard dataset generation

The following steps were taken in order to generate the gold standard dataset to facilitate measuring precision of URI collections extracted from the various post classes.

First, we selected a corresponding Wikipedia page for the five topics (Table 2). Second, we extracted the URIs from the references section of each Wikipedia page. Third, we dereferenced the URIs from each reference corresponding to a topic (e.g., Flint water crisis) and removed the HTML boilerplate leaving only the plaintext documents (stopwords removed). The set of plaintext documents were concatenated into one document. Fourth, for each topic, we created a collection vector consisting of the normalized Term Frequency (TF) weights of the concatenated document.

4.4. Primitive measures extraction

We counted the number of URIs (HTML, non-HTML, and both) per topic, per social media source, and per post class (Table 3). Additionally, we extracted the distribution of posts with URIs by counting the number of posts with a specified number of links for a given social media source (e.g., Reddit) to facilitate probability distribution calculation (Table 4). The distribution answers questions such as: “for Reddit posts with links, how many posts had 1 link or 2 links?” Subsequently, the following measures were extracted from the dataset to address our research questions.

4.4.1. Probability distribution of posts with links

.

For all topics $T$ (e.g., World cup), given the set of post classes C\in\{\textbf{P{}{1}$A${}{1}},MC,\textbf{P{}{1}$A${}{n}},\textbf{P{}{n}$A${}{1}},\textbf{P{}{n}$A${}{n}}\}, given a social media seed source $s$ (e.g., Reddit), the probability $P(p_{c}^{s}=k)$ of the event that a post $p_{c}^{s}$ of post class $c\in C$ with a URI, has $k$ URIs (e.g., 1 URI) is calculated using Eqn. 1. $P(p_{P_{1}A_{1}}^{Reddit}=1)$ reads: “What is the probability of the event that a Reddit P1A1 post with a URI has one (i.e., $k=1$) URI?”

The general probability $P(p_{All}^{s}=k)$ of the event that a post $p_{All}^{s}$ with a URI from social media $s$ of any post class, has $k$ URIs is calculated using Eqn. 2. In Eqn. 1 & 2, if $c=P_{1}A_{1}$, and $t=1$, $|c_{1}|$ represents the count of P1A1 posts for the first ($t=1$) topic.

[TABLE]

[TABLE]

4.4.2. Precision of the URIs in post class collections

.

Given a candidate collection of seed URIs $C$ to be evaluated, the URIs may be extracted from a single post (P1A1) or multiple posts (e.g., PnA1) from a social media site (e.g., Reddit). We calculated the precision of $C$ as follows. First, the URIs in $C$ were processed in the same manner as the gold standard (Section 4.3), i.e., dereferenced and boilerplate removed, and $|C|$ plaintext documents concatenated. Second, a document collection matrix $M$ was created from $C$ and its corresponding gold standard (e.g., Flint water crisis gold standard). The first row of matrix consisted of the gold standard vector, and the second row of the matrix consisted of the vector of $C$ (document to be evaluated). The columns represent the normalize TF weights. Third, cosine similarity was calculated between the pair of rows. If the similarity exceeded relevance threshold of an empirically learned threshold of 0.25, $C$ was declared relevant, otherwise, it was declared non-relevant.

For a given topic (e.g., Flint water crisis) and SERP vertical (e.g., Twitter-Top), a URI or multiple URIs may be extracted from a post authored by a single (P1A1) or multiple (PnA1, PnAn) users. Each group of URIs extracted from a post has an associated precision value (Relevant URIs / Total URIs). The average precision metric for a post class (e.g., P1A1) is an average over all the precision value of all posts in the post class. It provides answers to questions such as: “what is the average precision of the URIs in the P1A1 post class?” For non-HTML URIs we evaluated precision by extracting text from the post that embedded the URI.

4.4.3. Age distribution of relevant webpages per post class

The distribution of ages is an aggregation of the ages of the relevant webpages in a given post class of a given social media. The age of a webpage was calculated by finding the difference between the publication date of a webpage and the date the post containing the webpage URI was retrieved. The publication dates of webpages were extracted with CarbonDate (SalahEldeen and Nelson, 2013) which estimates the creation date of webpages based on information polled from multiple sources such as the document timestamps, web archives, backlinks, etc. Publication dates of webpages may potentially provide useful information about the kinds of events discussed. For example, the Democratic Republic of Congo in Central Africa has been grappling with another Ebola outbreak (2017-Present). Therefore, webpages published before 2017 are not expected to discuss the 2017 outbreak.

4.4.4. Distribution of hostname diversity per post class

.

Given a collection of URIs $C$ for a given post class of a given social media, the hostname diversity (Nwala et al., 2018a) of $C$ is a single value ($d\in[0,1]$) that reports whether $C$ consists of URIs from a single host ($d=0.0$, e.g., www.cnn.com) or distinct hosts ($d=1.0$, e.g., www.cnn.com and www.foxnews.com). It answers questions such as: “how diverse are the hosts in the Reddit P1A1 post class?”

4.4.5. Overlap between Google collections and post class

We measured the overlap between URIs extracted from Google and URIs extracted from a combination of social media and post class. This was done in order to determine how easy it was to find the URIs scraped from social media micro-collections. Extracting seeds from micro-collections requires more effort than scraping Web search engine SERPs. For example, generating a collection of URIs of the PnA1 or PnAn post class requires independently dereferencing each social media post and extracting the replies from the post. Therefore, if the URIs discovered from micro-collections are easily discoverable via a search engine such as Google, it does not justify the extra effort of extracting seeds from micro-collections.

5. Results and Discussion

Recall the post class (Table 1) acronyms and their respective meanings and examples: P1A1 (e.g., a tweet) - single Post from a single Author, P1An (e.g., Wikipedia reference) - single Post from multiple Authors, PnA1 (e.g., twitter thread) - multiple Posts by a single Author, and PnAn (e.g., twitter conversation) - multiple Posts from multiple Authors.

To address the first research question, we identified micro-collections (MC = PnA1 $\cup$ PnAn) as the collection of social media posts that show some properties of collection building222Some P1A1 posts which are visible to SERP scrapers could be added to MC if they contain links above the median number of links, calculated from the same pool of social media posts. However, we did not make such a distinction in our study.. Next, we extracted the PnA1 and PnAn post classes by identifying social media posts with replies (comments) and extracted the parent post as well as the child posts.

Following the identification and extraction of micro-collections, to address the second research question, we characterized MCs and compared seeds extracted from them to seeds extracted from SERPs (P1A1). Here we present the results for each of the respective measures introduced in Section 4.4, and Appendices 1 - 10 includes additional figures for these metrics.

5.1. URI and post counts per post class

Micro-collections (MCs) are prevalent on the Web and outnumber (12,917 vs. 10,195) conventional SERP posts (P1A1). Also, in general, MCs produced more URIs (Appendices 2 - 4) than conventional SERP posts (P1A1). Additionally, MCs produced more non-HTML URIs than P1A1 across all topics. In fact, the total number of P1A1 non-HTML URIs were between 19% to 44% the size of MCs. These findings are potentially consequential for curators interested in enriching their collections with non-HTML resources.

From Table 3, for all topics in the Reddit SERPs except (Reddit-New), PnAn mostly produced the largest count of URIs (41,160), next to P1A1 (51% PnAn), next to PnA1 (3% PnAn): PnAn ¿ P1A1 ¿ PnA1. The relatively low number of Reddit PnA1 posts and URIs shows that it is a rare phenomenon for a Reddit user to reply to his/her initial post especially since Reddit does not impose any size restriction on the length of posts. For the Reddit-New SERP, P1A1 had more URIs (8,056) than PnAn (80% P1A1): P1A1 ¿ PnAn ¿ PnA1. This is likely due to the fact that in the New SERP, PnAn do not get sufficient opportunity to increase because they must compete with newer posts, since the SERP is in “newest first” order. Consequently, before PnAn sufficiently grow, they are pushed down (rank demotion) by newer P1A1 posts, and do not get sufficient exposure, leading to fewer replies which leads to a reduced PnAn size.

The results show a high degree of inter/extra-user engagement on Twitter, and thus for Post and URI Counts (Table 3), PnAn ¿ PnA1 ¿ P1A1. In contrast, Scoop.it showed lesser user engagement, and thus: P1A1 ¿ PnA1.

5.2. Probability distribution of posts with links

From Table 4, unsurprisingly, the probability of the event that a social media post with a URI of a given post class (P1A1 - PnAn) had more than one HTML URI ($k>1$) seemed to correlate with whether the social media platform restricts the size of posts. For example, due to the character limit imposed on tweets, the probability of the event that a tweet with a URI has only 1 HTML URI is 0.98 ($P(p_{P_{1}A_{1}}^{Twitter}=1)=0.98$). On the other hand, single tweets with 3+ HTML URIs are rare. We observed three tweets with 3 or 4 HTML URIs (out of 3,501 tweets).

5.3. Precision of post class URIs

Table 5 shows the conditional probability of the event that the URIs contained in a post of a given post class are relevant, given that the post has a specified count of URIs ($k$). Across almost all $k$ per post class, we see that the seeds generated from P1A1 posts had a higher probability (maximum: 1, median: 0.63, minimum: 0.33) of being relevant than MC (0.61, 0.5, 0.0). For example, for Reddit when $k=2$, P1A1 - 0.80, while MC - 0.59. This shows that P1A1 posts benefit from SERP filters; P1A1 posts are posts directly returned by SERPs and their text often matches a subset of the query. This indicates that a match between a query and a post text lends some relevance to the URI extracted from the post. However, given the fact that MCs do not all benefit from SERP filters since the vast majority of MCs are not extracted directly from the SERP, but from the reply or comment threads, the 0.5 median precision value indicates that comments and replies possess quality URIs. See also Appendices 5 - 7 for additional figures of the average precision of URIs per post class, per social media.

In general, P1A1 post URIs (all URIs, HTML, and non-HTML) had the highest average precision compared to PnA1 and PnAn for Reddit, Scoop.it, and Twitter posts extracted with text queries. For tweets extracted with hashtags, PnAn posts had the highest average precision compared to PnA1 and P1A1.

For Reddit, P1A1 ¿ PnA1 ¿ PnAn: across all topics, P1A1 posts had the highest average precision (all URIs) 80% of the time than PnA1 and PnAn. The Maximum, Median, and Minimum (MMM) average precision values were 0.88, 0.59, and 0.15, respectively. Next, PnA1 posts had a higher average precision than PnAn 70% of the time, MMM - (0.88, 0.50, 0.00), for PnAn - (0.70, 0.42, 0.07).

For tweets exposed with text queries, P1A1 ¿ PnAn ¿ PnA1: P1A1 (0.91, 0.66, 0.45) had the highest average precision 90% of the time than PnAn and PnA1. PnAn (0.74, 0.46, 0.28) had a higher average precision 70% of the time than PnA1 (0.58, 0.39, 0.35).

For tweets exposed with hashtags, PnAn ¿ PnA1 ¿ P1A1: PnAn (0.65, 0.29, 0.27) posts had the highest average precision 60% of the time than PnA1 and P1A1. PnA1 (0.45, 0.39, 0.21) posts had a higher average precision than P1A1 (0.50, 0.26, 0.11) 70% of the time. For example, from Fig. 6a, the average precision for P1A1 URIs in the Twitter-Top vertical for the Ebola virus outbreak topic was 0.86 (PnAn - 0.74) for posts extracted with the text query “ebola virus outbreak.” However, PnAn outperformed (0.65) P1A1 (0.49) when the query used to extract posts was the hashtag “#ebolavirus.”

For Scoop.it, P1A1 ¿ PnA1: P1A1 (0.87, 0.78, 0.55) posts had a higher average precision than PnA1 (0.80, 0.55, 0.27) 100% of the time. Similar to Twitter P1A1, Scoop.it P1A1 are derived directly from the SERP, and thus benefit from SERP filtering. PnA1 do not benefit from SERP filtering since they are not extracted directly from the SERP.

5.4. Age distribution of relevant webpages

We compared the ages of P1A1 and MC post class URIs, by focusing on the older topics (Ebola virus outbreak and Flint water crisis) for social media that supports P1A1, PnA1, and PnAn - Reddit and Twitter. MC posts consistently produce older webpages in the Twitter-Latest vertical. A possible explanation for this is: P1A1 tweets (extracted directly from the Twitter-Latest SERP) are highly likely to be new tweets if the topic is ongoing. Even though new tweets can include URIs of old stories, for ongoing news stories such as those we considered, new tweets are likely to include the URIs of the latest developments. We observed that the Twitter-Latest P1A1 tweets were created within days from the query issue dates, and thus more likely to produce new URIs for both topics. In contrast, MCs are extracted from conversations that can mix new and old tweets; a new tweet can reply to an old tweet that contains old URIs. Therefore, Twitter-Latest MCs produced a mix of tweets created within days and years from the query issue dates. See also Appendices 8 and 9 for additional figures of the age distribution of URIs per post class, per social media.

For the Reddit-Top/Relevance/Comments SERPs for Ebola virus outbreak, MCs and P1A1 produced older webpages with similar distributions. For example, for Ebola virus outbreak both post classes had a median webpage age of  4.3 years.

As expected, the Reddit-New, for both topics, MCs and P1A1 produced the newest webpages compared to other Reddit SERPs with median age $<$ 1 year.

P1A1 and MC posts from Twitter-Top produced webpages with similar age distributions. For example, for Flint water crisis both post classes had a median webpage age $<$ 5 months. In contrast, in the Twitter-Latest vertical, for both topics MCs produced older webpages than P1A1. For example, MCs for Ebola virus outbreak produced older webpages (median: 4.2 years) than those from P1A1 (19 days) (Fig. 6b).

5.5. Distribution of hostname diversity

For Reddit, the PnA1 posts produced the highest hostname diversity. For Twitter, P1A1 posts produced the highest hostname diversity.

For Reddit, PnA1 ¿ P1A1 ¿ PnAn: across all topics, PnA1 posts had the highest hostname diversity (HTML URIs) 95% of the time than P1A1 and PnA1. The Maximum, Median, and Minimum (MMM) hostname diversity values were 1.0, 0.55, and 0.0, respectively. Next, P1A1 posts had more diverse hostnames than PnAn 61% of the time, MMM - (0.6, 0.33, 0.11), for PnAn - (0.55, 0.28, 0.1).

For Twitter, P1A1 ¿ PnAn ¿ PnA1: P1A1 (0.70, 0.60, 0.43) produced more diverse hostnames 74% of the time than PnAn and PnA1. Similarly, PnAn (0.61, 0.45, 0.39) produced more diverse hostnames 79% of the time than PnA1 (0.74, 0.37, 0.31). Scoop.it did not produce enough URIs for two topics, as a result had fewer PnA1 to derive a fair comparison with P1A1.

Reddit and Twitter had P1A1 ¿ PnAn in common. This is not unexpected; hostname diversity rewards unique hosts, and given that the P1A1 collection is smaller than PnAn, it is more likely for P1A1 to fill in the hostname slots with additional different hosts than PnAn. However, for Twitter PnA1 had the lowest diversity unlike Reddit for the following reasons. First, PnA1 is the set of all threads authored by the same user. These threads on Twitter, especially those from News (e.g., @nytimes, @vice) and non-News organizations (e.g., @splcenter, @TurnoutPAC) tend to link to webpages within their websites, leading to a lower hostname diversity. This phenomenon was most prominent in the 2018 world cup and midterm elections topics.

5.6. Overlap: Google collections vs. post classes

All post classes showed small amount of overlap with the collections of URIs returned from the first 10 pages of Google for the respective dates the post class URIs were extracted. This highlights the fluidity of the Google SERP. Thus, URIs extracted from MC and P1A1 collections are not easily discoverable.

Reddit P1A1 and MC posts had overlap $<$ 0.1 85% of the time. Their MMM overlap were: 0.13, 0.04, and 0.1, respectively. Twitter P1A1 posts had overlap (0.09, 0.02, 0.0) $<$ 0.1 100% the time. Similarly, Twitter MC posts had overlap (0.13, 0.04, 0.0) $<$ 0.1 80% of the time.

5.7. Recommendations for generating seeds

Considering the results presented, it is clear that collections generated from social media SERPs (P1A1) are different from collections generated from micro-collections (MCs), and both post classes yield seeds not easily discoverable by scraping Google. Consider the following highlights and how they could affect decisions made in generating seeds from social media.

MCs are more prevalent and produce more seeds than P1A1. This means seed generation that prioritizes quantity would benefit from extracting seeds from MCs. P1A1 produced higher quality URIs for all social media SERP combinations except with seeds generates with hashtags. The poorer precision performance of hashtag queries compared to text queries shows that hashtags can be used as a vehicle for spreading non-relevant content, especially when the hashtag is popular. However, when users reply to a tweet that contains a link and a hashtag (the composition of PnAn set), it is likely they are responding to a relevant tweet. Replies may serve as a quality check. Therefore, MCs produced more relevant URIs when hashtags were used to surface tweets. Consequently, seed generation that prioritizes quality would benefit from extracting seeds from P1A1, but for Twitter, if hashtags are used, MCs should be considered first.

MCs consistently produced older webpages than P1A1 posts for the Twitter-Latest vertical because MCs included older tweets. Consequently, if seed generation from the Twitter-Latest vertical intends to extract older stories, MCs should be prioritized. Finally, we showed that P1A1 produced more diverse hostnames than MCs for Twitter unlike Reddit. Therefore, seed generation that intends to include different hosts should consider P1A1, instead of Twitter PnA1, since it showed a low level of hostname diversity due to reuse of the same domains, which is a common practice especially among news organizations.

6. Future work and Conclusions

We populated the MC set with posts from PnA1 and PnAn exclusively. However, we believe posts from P1A1 with a higher than normal median number of URIs may be added to the MC set. The median number of URIs in social media posts for all social media/SERP combination was 1, 86% of the time. Therefore, we may add to MC, P1A1 posts with URIs above the median. Also, our precision evaluation was biased to favor pages rich in text. Consequently, we observed false negative URIs: relevant URIs marked as non-relevant, a problem we plan to address. Finally, we plan to expand the set of metrics for comparing seeds to include metrics that account for the authority of hosts. For example, such metric would assign a higher authority weight to a CDC (Centers for Disease Control and Prevention) webpage than an obscure webpage for the same Ebola virus topic.

The seed URIs that form the building blocks of Web archive collections are often hand-selected by curators. Manual selection produces high quality URIs but it does not scale and requires domain knowledge. Due to a shortage of curators amidst an abundance of unfolding events, Google and Twitter have been widely adopted for automatically scraping seeds. In this work, we introduced a cross-platform vocabulary (post class: P1A1, P1An, PnA1, and PnAn) for describing social media posts. It is common practice to scrape social media such as Twitter for (P1A1) seeds, we introduced an overlooked source of social media posts - micro-collections (MCs), and showed that seeds generated from MCs are more prevalent and different from their P1A1 counterpart across multiple dimensions. Finally, we provided recommendations for curators generating seeds from these sources. For example, a seed generation that prioritizes quantity may target MCs first, while a precision priority favors P1A1. Our study outlines how to compare seeds generated from different venues on social media and our findings may inform the decisions made during seed generation. Our research dataset comprising of 120,444 links extracted from 449,347 social media posts, as well as the source code for the application utilized to generate the collections are publicly available (Alexander Nwala, 2019).

Acknowledgments

This work was supported in part by IMLS LG-71-15-0077-15.