The TagRec Framework as a Toolkit for the Development of Tag-Based Recommender Systems

TL;DR

The paper introduces TagRec, an open-source toolkit designed to facilitate the development and evaluation of tag-based recommender systems, demonstrated through four practical use cases and supporting large-scale research projects.

Contribution

It presents the current state of TagRec, highlighting its versatility and application in various tag-based recommendation tasks and evaluations.

Findings

Supported development in two large-scale European projects

Implemented four diverse use cases for tag recommendations

Contributed to 17 research papers on tag-based systems

Abstract

Recommender systems have become important tools to support users in identifying relevant content in an overloaded information space. To ease the development of recommender systems, a number of recommender frameworks have been proposed that serve a wide range of application domains. Our TagRec framework is one of the few examples of an open-source framework tailored towards developing and evaluating tag-based recommender systems. In this paper, we present the current, updated state of TagRec, and we summarize and reflect on four use cases that have been implemented with TagRec: (i) tag recommendations, (ii) resource recommendations, (iii) recommendation evaluation, and (iv) hashtag recommendations. To date, TagRec served the development and/or evaluation process of tag-based recommender systems in two large scale European research projects, which have been described in 17 research papers. Thus, we believe that this work is of interest for both researchers and practitioners of tag-based recommender systems.

Tables

Table 1. Table 1. Datasets, recommendation algorithms and evaluation metrics supported by TagRec . A complete list of the features is provided on the framework’s Github page.

Dataset	Description
Flickr	Image sharing (Kowald and Lex, 2015)
CiteULike	Scientific references (Kowald and Lex, 2015)
BibSonomy	Publication sharing (Kowald and Lex, 2015)
Delicious	Social bookmarking (Kowald and Lex, 2015)
LastFM	Music sharing (Kowald and Lex, 2015)
MovieLens	Movie rating (Kowald and Lex, 2015)
Twitter	Microblogging (Kowald et al., 2017)
TravelWell	Learning resource exchange (Kopeinik et al., 2016)
Aposdle	Work-integrated learning (Kopeinik et al., 2016)
MACE	Informal learning (Kopeinik et al., 2016)
KDD15	KDD 2015 cup (Kopeinik et al., 2016)
Algorithm	Description
MostPopular	Frequency-based (Kowald and Lex, 2016)
CF	Collaborative Filtering (Marinho and Schmidt-Thieme, 2008)
FolkRank / APR	Graph-based (Jäschke et al., 2007)
FM / PITF	Factorization Machines (Rendle and Schmidt-Thieme, 2010)
LDA	Topic modeling (Krestel et al., 2009)
MostRecent / GIRP	Time-based (Kowald and Lex, 2016)
3Layers	Human categorization theory (Seitlinger et al., 2013; Kowald et al., 2015; Kowald, 2015)
BLL / BLLAC	Human memory theory (Kowald et al., 2014; Kowald et al., 2015; Trattner et al., 2016; Kowald et al., 2017)
CIRTT	Tag- and time-based (Lacic et al., 2014)
SUSTAIN	Human category learning (Seitlinger et al., 2015; Kopeinik et al., 2017)
SimRank	Content-based (Zangerle et al., 2011)
BLLI,S,C	Temporal hashtag patterns (Kowald et al., 2017)
Metric	Description
Recall	Accuracy (Kowald and Lex, 2015)
Precision	Accuracy (Kowald and Lex, 2015)
F1-score	Accuracy (Kowald and Lex, 2015)
MRR	Ranking (Kowald and Lex, 2015)
MAP	Accuracy & ranking (Kowald and Lex, 2015)
nDCG	Accuracy & ranking (Kowald and Lex, 2015)
AILD	Diversity (Kowald and Lex, 2015)
AIP	Novelty (Kowald and Lex, 2015)
Runtime	Computational costs (Kowald and Lex, 2015)
Memory	Computational costs (Kowald and Lex, 2015)

Table 2. Table 2. Use cases realized with TagRec . To date, TagRec supported the recommender development and/or evaluation processes described in 17 research papers.

Tag recommendations	Research papers
Model of human categorization	(Seitlinger et al., 2013; Kowald et al., 2015; Kowald, 2015)
Activation processes in human memory	(Kowald and Lex, 2016; Kowald et al., 2014; Kowald et al., 2015; Trattner et al., 2016)
Informal learning settings	(Dennerlein et al., 2015, 2015; Dennerlein et al., 2014)
Resource recommendations	Research papers
Attention-interpretation dynamics	(Seitlinger et al., 2015; Kopeinik et al., 2017)
Tag and time information	(Lacic et al., 2014; Larrain et al., 2015)
Recommendation evaluation	Research papers
Real-world folksonomies	(Kowald and Lex, 2015)
Technology enhanced learning settings	(Kopeinik et al., 2016)
Hashtag recommendations	Research papers
Temporal effects on hashtag reuse	(Kowald et al., 2017)

Full text

Recommender Systems; Recommender Framework; Recommendation Evaluation; Tag Recommendation; Hashtag Recommendation

1. Introduction

Recommender systems aim to predict the probability that a specific user will like a specific resource. Therefore, recommender systems utilize the past user behavior (e.g., resources previously consumed by this user) in order to generate a personalized list of potentially relevant resources (Ricci et al., 2011). Popular application domains of recommender systems include online marketplaces (e.g., Amazon and Zalando), movie and music streaming services (e.g., Netflix and Spotify), job portals (e.g., LinkedIn and Xing), and social tagging systems (e.g., BibSonomy and CiteULike).

Social tagging systems bear particularly great potential for recommender systems as, by nature, they produce a vast amount of user-generated resource-annotations (i.e., tags). Thus, possible use cases of these tag-based recommender systems include the suggestion of resources to extend a user’s set of bookmarks (De Meo et al., 2010; Zanardi and Capra, 2008) and the suggestion of tags to assist in the annotation of these bookmarks. The latter one is known as the field of tag recommendations (Jäschke et al., 2007).

Over the past years, various recommendation frameworks and libraries have been developed in order to support the development and evaluation of recommender systems (see Section 4). While these frameworks cover a wide range of application domains, to the best of our knowledge, an open-source recommendation framework to design and evaluate tag-based recommender systems was still lacking. Therefore, in 2014, we have started developing TagRec, a standardized tag recommender benchmarking framework (Kowald et al., 2014). In the initial development phase of the framework, we mainly focused on evaluating tag recommendation algorithms. In 2015, the framework was extended by including resource recommendation algorithms that are based on social tagging data (Trattner et al., 2015).

The aim of this paper, however, is to present the current, updated state of TagRec. This includes the extension of the framework for (i) the analysis of tag reuse practices (Kowald and Lex, 2016), (ii) the evaluation of tag recommendations in real-world folksonomy and Technology Enhanced Learning settings (Kowald and Lex, 2015; Kopeinik et al., 2016), and (iii) hashtag recommendations in Twitter (Kowald et al., 2017).

Apart from that, we provide an updated framework description (see Section 2) as well as a summary of use cases in the field of recommender research that have been completed using TagRec (see Section 3). Research areas encompass tag recommendations, resource recommendations, recommendation evaluation and hashtag recommendations. To date, TagRec has served the recommender development and/or evaluation processes in two large-scale European research projects, which have been published in 17 research papers. We conclude the paper with a discussion on future work and potential improvements of the framework (see Section 5).

We believe that our work contributes to the rich portfolio of technical frameworks in the area of recommender systems. Furthermore, this paper presents an overview of use cases which can be realized with TagRec, and should be of interest for both researchers and developers of tag-based recommender systems.

2. TagRec

TagRec is a Java-based recommendation framework for tag-based information retrieval settings. It is open-source software and freely available via our Github repository111https://github.com/learning-layers/TagRec. The Github page also contains a detailed technical description on the usage of the framework.

Figure 1 illustrates TagRec’s system architecture. The framework consists of (i) a data processing component, which processes data sources, (ii) a data model and analytics component, which enables access to the processed data, (iii) recommendation algorithms, which calculate recommendations, (iv) an evaluation engine, which evaluates the algorithms, and (v) recommendation results, which can be passed to a client application. The mentioned components are described in more detail in the remainder of this section. Apart from that, we describe practical aspects of the framework that should be helpful when implementing and/or evaluating a recommendation algorithm. Finally, Table 1 provides an overview of the supported datasets, recommendation algorithms and evaluation metrics.

Data Processing. The data processing component is responsible for parsing and processing external data sources. Currently supported datasets are listed in Table 1. These datasets serve a wide range of application domains such as social bookmarking systems, learning environments, microblogging tools and music/movie sharing portals. The set of datasets can easily be extended by implementing custom data pre-processing strategies.

Furthermore, this component supports various data enrichment and transformation methods such as $p$-core pruning (Doerfel and Jäschke, 2013), topic modeling (Krestel et al., 2009), training/test set splitting (Kowald and Lex, 2015) and data conversion into related formats (e.g., for MyMediaLite (Gantner et al., 2011)).

Data Model and Analytics. The data model is created based on described data processing steps and provides an object-oriented representation of the data in order to ease the implementation process of a novel recommendation algorithm. Thus, it enables easy access to the entities in the datasets via powerful query functionality (e.g., get the set of tags a user has used in the past). Furthermore, the data model of TagRec is connected to Apache Solr222http://lucene.apache.org/solr/ and thus, enables fast access to content-based data of entities.

Another role of this component is the provision of basic data analytics functionality to get a better understanding of the dataset characteristics. For example, dataset statistics, such as the total number of distinct tags or the average number of bookmarks per user, can be retrieved.

Recommendation Algorithms. TagRec contains a wide range of recommendation algorithms (see Table 1). As later described in Section 3, algorithms for tag recommendations, resource recommendations and hashtag recommendations are provided. These algorithms can be used as baseline approaches for a newly implemented algorithm. The complete list of all variants of these algorithms is provided on the TagRec’s Github page.

A key contributions of TagRec is that next to well-established approaches, such as Collaborative Filtering, MostPopular and Factorization machines, it also encompasses approaches based upon cognitive models of information retrieval, human memory theory and category learning. In (Kowald and Lex, 2015), it has been shown that these cognitive-inspired approaches achieve high prediction accuracy estimates in comparison to classic recommendation algorithms. Besides, the algorithms have demonstrated their suitability for sparse datasets such and narrow folksonomies.

Evaluation Engine. The evaluation engine quantifies the quality of implemented recommendation strategies by applying a rich set of evaluation metrics as listed in Table 1. One drawback of most recommendation evaluation frameworks is their focus on accuracy and ranking estimates, which restricts the evaluation to the performance of recommender systems (Bordino et al., 2013).

To fill this gap, TagRec supports a variety of evaluation metrics to also offer indicators for diversity, novelty, runtime performance and memory consumption of algorithms. For evaluating an algorithm, TagRec has to be provided with three parameters, where the first one specifies the algorithm, the second one specifies the dataset directory and the third one specifies the file name of the dataset sample. For example, java jar tagrec.jar cf bib bib_sample runs Collaborative Filtering on a sample of the BibSonomy dataset. The calculated metrics are then either written to a “metrics” file or printed to the console.

Recommendation Results. As indicated in Figure 1, the algorithms’ recommendation results can be either forwarded to the evaluation engine to retrieve evaluation metrics or to a client application for further processing (e.g., visualization). The KnowBrain tool (Dennerlein et al., 2015) is an example of such a client application. It is an open source social bookmarking tool, which has been extended to cater the requirements of tag recommender evaluations in online settings.

A screenshot of KnowBrain’s graphical user interface is shown in Figure 2. It enables the bookmarking of Web links and their annotations by (i) selecting from a pre-defined set of categories, and by (ii) assigning a variable number of tags. The user’s tagging process is supported by a list of recommended tags that are selected based on algorithms of the TagRec framework. The elicitation of categories allows for semantic context-based recommendation algorithms such as 3Layers (Seitlinger et al., 2013). Furthermore, the comparison of actually used tags with recommended tags gives insights into the online performance (i.e., user acceptance) of recommendation strategies.

3. Use Cases

In this section, we describe use cases that have been implemented using the TagRec framework. To date, TagRec supported the recommender development and/or evaluation processes in two large-scale European research projects. Results have been published in 17 research papers (see Table 2).

3.1. Tag Recommendations

Tag recommendation systems assist users in finding descriptive tags to annotate resources. In other words, given a specific user and a specific resource, a tag recommendation algorithm predicts a set of tags a user is likely to apply in annotating the resource (Jäschke et al., 2007).

Within this context, TagRec was used for the creation of (i) cognitive-inspired algorithms, and (ii) the evaluation of approaches suitable for formal and informal learning settings.

Tag Recommendations Using a Model of Human Categorization. In (Seitlinger et al., 2013; Kowald et al., 2015; Kowald, 2015), the authors introduced a tag recommendation algorithm based on the human categorization models ALCOVE (Kruschke, 1992) and MINERVA2 (Hintzman, 1984). This algorithm is called 3Layers and simulates categorization processes in human memory. Therefore, the categories assigned to a given resource, which a user is going to annotate, are matched against already annotated resources of this user. Based on this matchmaking process, a set of tags associated with semantically related resources is recommended.

Since TagRec enables to link a list of categories to a resource, it supported the development of 3Layers by providing functions for analyzing and deriving category information of resources (e.g., via LDA topic modeling (Krestel et al., 2009)).

Utilizing Activation Processes in Human Memory. Activation processes in human memory describe the general and context-dependent usefulness of information. It was shown that these processes (especially usage frequency, recency and semantic context) greatly influence the reuse probability of tags (Kowald and Lex, 2016). Based on this, a set of time-aware tag recommendation approaches (see (Kowald et al., 2014; Kowald et al., 2015; Trattner et al., 2016)) was developed that utilize the activation equation of the cognitive architecture ACT-R (Anderson et al., 2004).

Therefore, TagRec was used to analyze the timestamps of tag assignments and to calculate tag co-occurrences for reflecting the semantic context of social tagging. Furthermore, TagRec enabled the hybrid combination of the components of the model (e.g., combining time-aware and context-aware recommendations).

Tag Recommendations in Informal Learning Settings. In the course of the European-funded project Learning Layers333http://learning-layers.eu/, which aims at supporting informal learning at the workplace, tag recommendations were used to support the individual user in finding descriptive tags and the collective in consolidating a shared tag vocabulary. These tag recommendations were used in two tools: (i) the Dropbox-like environment KnowBrain (Dennerlein et al., 2015), and (ii) the Sensemaking interface Bits & Pieces (Dennerlein et al., 2014).

To achieve this, TagRec was integrated as a tag recommendation library into the Social Semantic Server (Dennerlein et al., 2015), which was used as the technical back-end for KnowBrain and Bits & Pieces. This shows that TagRec cannot only be used as a standalone tool but also as a programming library (or toolkit) to include recommendation functionality in existing software. A similar approach will be followed in another European-funded project called AFEL444http://afel-project.eu/, which engages in design and development of analytics for everyday learning.

3.2. Resource Recommendations

Resource recommender systems suggest potentially relevant web items (e.g., movies, books, learning resources, URLs, etc.) to users. Most of these recommender systems are based on Collaborative Filtering (CF) techniques, which aim to calculate similarities between users to suggest the most suitable web resources to them (Sarwar et al., 2001). TagRec was applied to support and improve the development of CF approaches in tag-based online environments.

Mimicking Attention-Interpretation Dynamics. Seitlinger et al. (Seitlinger et al., 2015) introduced the first version of a CF-based recommendation approach that takes into consideration non-linear user-artifact dynamics, modeled by means of SUSTAIN. SUSTAIN (Supervised and Unsupervised STratified Adaptive Incremental Network) is a flexible network model of human category learning that is thoroughly discussed in (Love et al., 2004). It assumes that learning is a dynamic process that takes place through the encounter of new examples (e.g., Web resources). Throughout the learning trajectory, categories emerge and learners’ attention foci shift. In (Kopeinik et al., 2017), an advanced, adapted version of the initial approach was presented and analyzed in detail. The resulting approach $SUSTAIN+CF$, firstly applies CF to calculate the most suitable resources for a user, and secondly re-ranks this list depending on a user’s category learning model (i.e., SUSTAIN’s user model).

The algorithm has been implemented and developed within the TagRec framework. Features of the framework allowed for continuous evaluation and analysis of single factors of the model and with it, the associated change of recommendation performance. With this data, it was possible to gain deeper insight into the algorithmic approach and its parameters and thus, to further adapt the model to the requirements of our application area.

Resource Recommendations using Tag and Time Information. In (Lacic et al., 2014), the Collaborative Item Ranking Using Tag and Time Information (CIRTT) approach was presented. CIRTT uses Collaborative Filtering to identify a set of candidate resources and re-ranks these candidate resources by incorporating tag and time information. This is achieved via the Base-Level-Learning (BLL) equation, which is one component of ACT-R’s activation equation (Anderson et al., 2004).

Since TagRec contains a full implementation of the activation equation, it could be easily adapted for the task of resource recommendations as well. Apart from that, TagRec was used to compare CIRTTT to other related resource recommendation methods (e.g., (Zheng and Li, 2011)). Another study of the recency effects in Collaborative Filtering recommender systems was provided in (Larrain et al., 2015).

3.3. Recommendation Evaluation

One of the most challenging tasks in the area of recommender systems, is the reproducible evaluation of recommendation results (Herlocker et al., 2004). TagRec aims to support this process by providing standardized data processing methods, baseline algorithm implementations, evaluation protocols and metrics.

Evaluating Tag Recommendations in Real-World Folksonomies. Because of the sparse nature of social tagging systems, most tag recommendation evaluation studies were conducted using $p$-core pruned datasets. This means that all users, resources and tags, which do not appear at least $p$ times in the dataset, are removed. This clearly does not reflect a real-world folksonomy setting as shown by (Doerfel and Jäschke, 2013).

To overcome this problem, TagRec was used in (Kowald and Lex, 2015) to compare a rich set of tag recommendation algorithms using a wide range of evaluation metrics on six unfiltered social tagging datasets (i.e., Flickr, CiteULike, BibSonomy, Delicious, LastFM and MovieLens). The results showed that the efficacy of a recommendation algorithm greatly depends on the given dataset characteristics, and that cognitive-inspired approaches provide the most robust results, even in sparse data folksonomy settings.

Comparing Recommendation Algorithms in Technology Enhanced Learning Settings. Kopeinik et al. (Kopeinik et al., 2016) is another example of using TagRec for the evaluation of a variety of algorithms on different offline datasets. The paper focused on technology-enhanced formal and informal learning environments, where due to fast changing domains and characteristic group learning settings, data is typically sparse. The evaluation was divided in two settings, the performance of (i) resource recommendation strategies, and (ii) tag recommendation strategies. In both cases, the authors compared the recommendation accuracy of a number of computationally-inexpensive recommendation algorithms on six offline datasets retrieved from various educational settings (i.e., social bookmarking systems, social learning environments and massive open online courses). Investigated approaches are either state-of-the-art recommendation approaches, or strategies that have been explicitly suggested in the context of TEL systems.

To address the goals of this study, the TagRec framework already provided a wide range of required functionality such as the implemented data processing component, evaluation metrics and state-of-the-art algorithms. In the context of this research paper, it was further extended by a couple of algorithms that are considered particularly relevant to learning settings and by additional statistics, which were needed to interpret evaluation results properly.

3.4. Hashtag Recommendations

Over the past years, hashtags have become very popular in systems such as Twitter, Instagram and Facebook. Similar to social tags, hashtags are freely-chosen keywords to categorize resources such as Twitter posts (i.e., tweets). One of the biggest advantages of hashtags is that they can be easily used by integrating them in the tweet text. Unsurprisingly, this has led to the development of hashtag recommendation algorithms that aim to support users in applying the most descriptive hashtags to their tweets (Zangerle et al., 2011).

Temporal Effects on Hashtag Reuse. In (Kowald et al., 2017), a time-dependent and cognitive-inspired hashtag recommendation approach was proposed. In this paper, temporal effects on hashtag reuse in Twitter have been analyzed with the help of TagRec in order to design a hashtag recommendation approach, which utilizes the BLL equation of the cognitive architecture ACT-R (Anderson et al., 2004). Therefore, TagRec was extended with functions to access Apache Solr (see Section 2), which enables the content-based analysis of tweets using TF-IDF (see (Kowald et al., 2017)).

4. Related Work

In recent years, a multitude of recommendation engines have been created and made available either for commercial use or as open-source systems. In (Jenson, 2017), a well-structured overview of such approaches is presented. The author differentiates between Software-as-a-Service (i.e., SaaS), non-SaaS, open-source, academic and benchmarking recommender systems. We consider open-source and benchmarking frameworks that implement recommendation strategies fitting academic purposes as most relevant to our work.

A considerable contribution to this area is LibRec555http://wiki.librec.net/doku.php, a Java-based library that, so far, comprises around 70 resource recommendation algorithms and evaluation modules (Guo et al., 2015). Another Java-based, open-source framework is RankSys666http://ranksys.org/, which focuses on the evaluation of ranking problems and supports the investigation of novelty as well as diversity for academic research (Castells et al., 2015), which is reflected in its design (e.g., data input interfaces work with a triple of user, item and features).

Other examples of open-source recommender software are MyMediaLite777http://www.mymedialite.net/, an item recommender library that focuses on rating and ranking predictions in collaborative filtering approaches (Gantner et al., 2011), CARSKit888https://github.com/irecsys/CARSKit, a recommendation library specifically designed for context-aware recommendations, and Tag Recommender999http://www.libfm.org/tagrec.html, a software component that implements Tensor Factorization models for personalized tag recommendations in C++ (Rendle and Schmidt-Thieme, 2010).

However, to the best of our knowledge, an open-source recommender framework that implements a wide range of tag and resource recommendation algorithms (including a number of cognitive-inspired approaches) for the design and evaluation of personalized tag-based recommendation strategies was still missing.

5. Conclusion and Future Work

In this paper, we presented the TagRec framework as a toolkit for the development and evaluation of tag-based recommender systems. TagRec is open-source software written in Java and can be freely downloaded from Github. The framework consists of five components: (i) a data processing component, which processes data sources, (ii) a data model and analytics component, which enables access to the processed data, (iii) recommendation algorithms, which calculate recommendations, (iv) an evaluation engine, which evaluates the algorithms, and (v) recommendation results, which can be passed to client applications.

Apart from that, we summarized various use cases realized with TagRec from the fields of tag recommendations, resource recommendations, recommendation evaluation and hashtag recommendations. To date, TagRec supported the development and/or evaluation process described in 17 research papers. Specifically, our framework was used for the realization of recommendation algorithms based on models of cognitive science. In these papers, it was shown that the cognitive-inspired approaches provided the most robust results, even in sparse data folksonomy settings.

We believe that TagRec extends the already rich portfolio of recommender frameworks with a toolkit that is specifically tailored to fit tag-based settings. Furthermore, the presentation of TagRec’s use cases should be of interest for both researchers and developers of tag-based recommender systems.

Limitations & future work. Currently, one limitation of TagRec is that the data access is not standardized. Thus, social tagging data is accessed from folksonomy files, whereas resource-related metadata (e.g., tweet content) is accessed from Apache Solr.

Thus, our first plan for future work is to implement a mechanism that integrates all data into Apache Solr. Apart from that, we want to further work on the stability and code quality of the framework. For example, we want to enhance the build and dependency management of the software using Apache Maven101010https://maven.apache.org/.

6. Acknowledgments

This work was supported by the Know-Center Graz, the European-funded projects AFEL (GA: 687916) and Learning Layers (GA: 318209), and the Austrian Science Fund (FWF) project OMFix (Grant No P27709-G22). The Know-Center Graz is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency (FFG).