E-Sports Talent Scouting Based on Multimodal Twitch Stream Data

TL;DR

This paper explores a novel approach to identify e-sports talent by analyzing multimodal Twitch stream data, including video, audio, and chat, to predict gamer ranks using neural features and Bayesian modeling.

Contribution

It introduces a new multimodal data-driven method for e-sports talent scouting, combining neural feature extraction and hierarchical Bayesian modeling to estimate gamer skill.

Findings

High correlation between predicted intrinsic skill and actual gamer ranks

Effective integration of video, audio, and chat data for skill estimation

Demonstrated feasibility of automated e-sports talent prediction

Abstract

We propose and investigate feasibility of a novel task that consists in finding e-sports talent using multimodal Twitch chat and video stream data. In that, we focus on predicting the ranks of Counter-Strike: Global Offensive (CS:GO) gamers who broadcast their games on Twitch. During January 2019-April 2019, we have built two Twitch stream collections: One for 425 publicly ranked CS:GO gamers and one for 9,928 unranked CS:GO gamers. We extract neural features from video, audio and text chat data and estimate modality-specific probabilities for a gamer to be top-ranked during the data collection time-frame. A hierarchical Bayesian model is then used to pool the evidence across modalities and generate estimates of intrinsic skill for each gamer. Our modeling is validated through correlating the intrinsic skill predictions with May 2019 ranks of the publicly profiled gamers.

Tables

Table 1. Table 1: Modeling Data Statistics (Validation / Training)

Modality	Task	#Users	#Datapoints
Video	pretrain	186/ 3,531	433/ 8,221
Video	fine-tune	98/ 270	809/ 2,521
Audio	pretrain	1,822/ 4,094	1,994/ 5,013
Audio	fine-tune	91/ 261	613/ 1,727
Chat	pretrain	464/ 539	3,825/ 11,473
Chat	fine-tune	79/ 203	2,421/ 6,358

Table 2. Table 2: Results for Validation Set Gamers.

Loss	Pretr.	AUC	F1	Prec.	Recall
Audio Features
Cross-entropy	yes	0.339	n.a.	n.a.	n.a.
Cross-entropy	no	0.473	n.a.	n.a.	n.a.
Triplet+SVM	yes	0.585	0.526	0.417	0.714
Triplet+SVM	no	0.527	0.411	0.395	0.429
Text Chat Features
Cross-entropy	yes	0.482	0.539	0.471	0.632
Cross-entropy	no	0.508	0.506	0.467	0.553
Triplet+SVM	yes	0.551	0.600	0.484	0.789
Triplet+SVM	no	0.531	0.558	0.500	0.632
Temporal Chat Features
Cross-entropy	yes	0.576	0.564	0.550	0.579
Cross-entropy	no	0.553	0.549	0.472	0.658
Triplet+SVM	yes	0.784	0.709	0.683	0.737
Triplet+SVM	no	0.625	0.550	0.524	0.579
Spatial Video Features
Cross-entropy	yes	0.489	0.585	0.422	0.950
Cross-entropy	no	0.496	0.574	0.416	0.920
Triplet+SVM	yes	0.609	0.554	0.459	0.700
Triplet+SVM	no	0.606	0.571	0.462	0.750
Motion Video Features
Cross-entropy	yes	0.557	0.597	0.440	0.925
Cross-entropy	no	0.588	0.607	0.451	0.925
Triplet+SVM	yes	0.586	0.536	0.456	0.650
Triplet+SVM	no	0.708	0.603	0.461	0.875
Bayesian Pooling*
Best upstream models		0.797	0.754	0.672	0.860

Full text

e-sports, Twitch stream, multimodal learning, Bayesian pooling

1 Introduction

E-sports are formalized competitive computer gaming (Taylor, 2018). E-sports are gaining attention similar to that of the real life sports for their entertainment value (Raath, 2017). As a consequence, e-sports talent scouting is likely to become an economically valuable activity.

Many gamers share their real-time online game experience on Twitch,111https://www.twitch.tv which is an interactive video streaming platform. The video component of the broadcast typically contains game visuals, but could also include an image of a gamer him or herself (see Figure 1). The audio component of the broadcast could contain both, the sounds of the game as well as the commentary of the gamer. In turn, the broadcast audience can provide real time feedback on the game being played via the text chat. Twitch is now the fourth largest source of peak Internet traffic in the US (Deng et al., 2015).

We propose a novel multimodal task that consists in automated identification of talented online gamers based on their Twitch streamed video and audio recordings as well as their spectator chat messages. To this end, we focus on the Counter-Strike: Global Offensive (CS:GO),222CS:GO is a multi-player first-person shooter video game developed by Hidden Path Entertainment and Valve Corporation. It is one of the most popular games represented on Twitch. for which some of the gamers are publicly profiled by the E-Sports Entertainment Association (ESEA) League.333https://play.esea.net

Our research contributions are two-fold. First, we develop a high recall baseline model demonstrating feasibility of the e-sports talent scouting task outlined above. Second, we contribute two new multi-modal Twitch chat and video stream datasets (one for 425 publicly ranked CS:GO gamers and one for 9,928 unranked CS:GO gamers) collected during January 2019–April 2019.444The data acquisition and modeling code can be accessed at https://github.com/mug31416/E-sports-on-Twitch. We are working on making the data publicly available.

2 Related Work

Although multimodal research on e-sports is rather scarce, we found real-life sports research and research using Twitch platform data to be pertinent to our proposed task.

2.1 Research Related to Sports Talent Scouting

There is considerable literature focusing on the real life competitive sports performance analysis. In that, there are a few examples of multimodal research that has focused on mining for athletic talent using a variety of sources, including biomedical data and social media statistics (Schumaker et al., 2010; Radicchi & Mozzachiodi, 2016; Cumming et al., 2017).

More broadly, there are several extensively studied tasks that can be seen as pretext tasks for multimodal sports performance analysis. These tasks include multimodal sentiment detection (Cai & Xia, 2015; Borth et al., 2013; Campos et al., 2017; Thelwall et al., 2012; He et al., 2018), event detection (Ma et al., 2018b; Aslam & Curry, 2018; Ma et al., 2018a; Wang et al., 2018), and clustering (Wang et al., 2007). Tools and methods employed in this literature on intelligent multimedia modeling, including the use of neural features (Kung & Hwang, 1998) as well as work with incomplete and heterogeneous data (Miech et al., 2018), are directly relevant to our task.

2.2 Research using Twitch Platform Data

Relevant research employing Twitch data has largely focused on chat (Ruvalcaba et al., 2018; Ford et al., 2017; Nakandala et al., 2017) or mixed-mode (chat and audio transcripts) (Wirman, 2018) communication as well as on retrieval of streaming video highlights (Zhang et al., 2017; Liu et al., 2015), with some studies employing multimodal information (Jiang et al., 2019; Fu et al., 2017). Thus far, game play on Twitch explored a problem of identifying trolls in Pokemon games (Haque, 2019).

There are also many studies on Twitch as a social and economic phenomenon (Deng et al., 2015; Taylor, 2018; Zhu et al., 2017; Gros et al., 2018; Sjöblom et al., 2019), with papers ranging from qualitative exploration to quantitative analysis of platform participation activity and user surveys. However, optimization of Twitch content delivery thus far appears to have received most attention (Glickman et al., 2018; Provensi et al., 2017; Bilal & Erbad, 2017; Laterman et al., 2017; Deng, 2018).

To our knowledge, there is no research that combines Twitch chat and video stream data with an external supervision signal from a public gaming leaderboard to make inferences about comparative player performance.

3 Data

ESEA and Twitch data for modeling purposes have been systematically collected during January 2019–April 2019. Upon completion of modeling, we have obtained ad hoc validation data from May 2019 ESEA profiles.555Because of the changes in ESEA website security policy, the May 2019 rankings could not be obtained using the same scraping scripts as the ones used for the main data acquisition. Hence, we relied on a one-at-a-time manual rank look-up for selected gamers. We provide data collection details below.

3.1 ESEA data

The ESEA data comprise the supervision element of our study. The ESEA League is an online e-sports community that matches CS:GO players across different skill levels, maintains player statistics, and issues performance-based ranks. The ESEA leaderboard contains a ranked list of approximately 2,000 players who fall into four ranking sections—A (highest), B, C, and D—with each category containing approximately 500 players. The leaderboard rank sections are updated monthly. In addition, there are rank sections G (pre-professional) and S (professional). Elite gamers in these sections are tracked separately from the public leaderboard; there are approximately 500 players in this group.

Some ESEA player profiles contain Twitch stream identifiers. On two separate occasions (in January 2019 and in February 2019), we have scraped the ESEA leaderboard to obtain a roster of CS:GO players for tracking on Twitch. As a result, we extracted 748 CS:GO gamers with Twitch accounts for monitoring.

In addition, for each monitored gamer we extracted his or her rank section in January 2019, February 2019, March 2019, and April 2019. Because of the substantial leaderboard turnover, not all gamers have rank data each month (some get promoted to the elite sections, while others withdraw from competitive play). We also note that our monitored group has a higher share of rank A section players (40% vs 25% in general).

3.2 Twitch data

Twitch streamed video, audio, and text chat for CS:GO games comprise the feature space of our study. As explained in Section 4, we also employ the number of Twitch user followers as a weak supervision signal for pretraining.

The Twitch platform has a well-developed API666https://dev.twitch.tv/docs that can be used to detect specific streamer status (i.e., whether the streamer is online), extract all channels that are streaming CS:GO in general, and acquire channel-specific chat logs. Furthermore, Twitch provides an API, together with “streamlink” that allows us to record live streams with audio.

We have continuously collected Twitch chat and stream data during January 2019–April 2019 for the 748 ESEA players with available Twitch stream identifiers. In addition, for use in pretraining, we have also collected these data for all CS:GO games during March 2019–April 2019.

3.2.1 Followers

Meta-data for each Twitch user contains a number of followers (i.e. other users who are subscribed to this user’s channel). Twitch provides an easy API to acquire number of followers of a channel. We obtained an April 2019 snapshot of the number of followers for each targeted Twitch user. Note that for 2% of the Twitch users we tracked we have been unable to obtain the number of followers due to various exceptions (e.g., Twitch account anomaly, account deleted).

3.2.2 Video Stream

We have followed the guidelines in this tutorial,777https://www.godo.dev/tutorials/python-record-twitch/ to set up continuous monitoring for several groups of targeted Twitch users and capture the video stream whenever one of the users in the group appears online. For the unranked CS:GO players, we have set up a query service to obtain a list of users playing CS:GO every 15 seconds; this list has been passed on to the video stream recorder.

We have run multiple instances of this streamer monitor in parallel. Because we only record one stream at a time (within a group of users), it is possible that we have missed some streams. However, we do not see this as a major issue, because not many players are be online at the same time and we record the stream for a limited amount of time. Specifically, we have limited the stream recording duration to 5 minutes. We also have limited the number of clips per user per day to be one.

For the 748 ESEA streamers, we have collected 3,330 videos by 368 distinct gamers. The total size of these data is 741 GB. For the all other CS:GO gamers on Twitch without the ESEA profile, have collected 13,969 videos by 9,928 distinct users. The size of these data is 1.56 TB.

3.2.3 Text Chat

To gather the chat data from Twitch streams, we followed the pipeline in this GitHub repository.888https://github.com/bernardopires/twitch-chat-logger This pipeline automatically creates bots to listen for the chat log data of the target players. One bot is responsible for 20 channels. We have monitored 748 ESEA target gamers, which has required 38 bots in total.

For general CS:GO collection, we first used Twitch API to scrape live streams of CS:GO game and added the limitation of stream language to be English. After we got the names of those live streams, we used the same way of collecting chat logs for target gamers but fed in those live stream names as the monitor targets. Since the live streams are only active for a limited time period, we repeated the above data collection process once a day.

For the 748 ESEA streamers, there are 210,185 chat logs collected from 308 distinct players (channels). The total size of these data is 47 MB. For the general CS:GO gamers on Twitch without the ESEA profile, we have collected 1,020,687 chat logs from 660 distinct players. The size of those general CS:GO gamers is 187 MB.

4 Methods

Figure 2 provides an overview of our modeling pipeline. In that, we first apply discriminative learning techniques separately for each modality with an aim to predict whether or not a gamer is in the rank A section of the ESEA leaderboard at any point during January 2019–April 2019 (i.e., a binary classification task).999We did not use month-specific gamer ranks due to the data alignment challenges. To this end, we use a training set of the ESEA Twitch streamers, which, depending on the modality-specific data availability, comprises approximately 75% of all ESEA Twitch streamers.101010The split of the ESEA Twitch streamers has been implemented to preserve the share of the rank A section gamers of 40% across the training set and validation set. We experiment with two discriminative learning strategies, one using cross-entropy loss and another using triplet loss (Cai et al., 2007; Hermans et al., 2017). The setting of triplet loss is maxplus with square euclidean distance. The generated embeddings were then fed into an SVM of scikit-learn default settings; probabilistic predictions are generated using the Platt method (Platt et al., 1999).111111Our preliminary evaluation has shown that a non-linear SVM is an effective classifier for our data. It performed better than logistic regression, and random forest classifier.

As can be seen from Table 1, the number of unique Twitch streamers with available training data ranges between 203 and 270. Therefore, in addition to training our models from scratch, we experiment with fine-tuning these models after pretraining them on a related task of predicting the quantized number of Twitch followers.121212There is a statistically significant 8% correlation between rank A section membership and the number of followers. The quantization has been done according to the deciles of the empirical distribution for the number of followers; we have treated this pretraining target as a multiclass prediction problem.

In the second modeling stage, we use a hierarchical Bayesian model to pool the evidence from the upstream modeling tasks (i.e., predicted probability for a gamer to be in the rank A section) as well as the validation set rank section data. This modeling stage generates estimates of the intrinsic skill level for each gamer in our dataset. Below we provide additional details on the first-stage modality-specific data processing and modeling, as well as on the second-stage pooling model.

4.1 Visual Data

For the visual data we have used a classic two-stream modeling approach (Simonyan & Zisserman, 2014), in which spatial/image data and motion/optical flow data are modeled using two separate networks.

4.1.1 Processing

We have used ffmpeg (Ffmpeg, 2010) to cut all ESEA gamer videos to 60 sec, whereas videos used for pretraining have been cut to 20 sec. All videos have been also downsampled to 10 frames per second. OpenCV (Bradski, 2000) has been used to rescale all frames such that the smallest dimension is 256 pixels; the resulting JPEG quality has been set at 60%.131313We note that performance of our models for JPEG quality above 60% has not been materially better than performance reported in this paper. Optical flow has been estimated using a variational motion estimation method (Zach et al., 2007) from a GPU adaptation of OpenCV functionality from this GitHub repository.141414https://github.com/feichtenhofer/gpu_flow

4.1.2 Modeling

We adapted the two-stream network implementation from this GitHub repository.151515https://github.com/jeffreyhuang1/two-stream-action-recognition For both, spatial network and motion network, we trained the projection layer and the last block of the pretrained ResNet101 (He et al., 2016), available from torchvision (Marcel & Rodriguez, 2010). We use initial learning rates of 1e-3 and 5e-3 for the spatial model and the motion model, respectively. Each model has been fit for 10 epochs, using batch size four.

4.2 Audio Data

While we experimented with both, Mel-frequency cepstral coefficients (MFCC) and transcripts, we determined that audio transcripts are not usable because around 25% of Twitch audio streams are non-English. Hence, we focused only on MFCC features.

4.2.1 Processing

We have extracted of audio (WAV format) from videos (MP4) using ffmpeg (Ffmpeg, 2010). And extraction of MFCC features from WAV files using librosa. 161616https://librosa.github.io/ MFCC latent dimension is set to 20 and the audio is collected at a sample rate of 8K.

4.2.2 Modeling

Instead of clustering MFCCs into a codebook, we have trained a Convolutional Neural Network (CNN) architecture directly on MFCCs using keras.171717https://keras.io/ Three region sizes (2, 3, 4) are used, corresponding to bi-gram, tri-gram and 4-gram and for each region size, 16 filters. After a Global Max Pooling, there is a fully connected part with one hidden layer of 50 dimensions.

4.3 Chat Data

To represent information contained in the Twitch chat logs, we developed two models, one focusing on the chat content and another focusing on the chat density received.

4.3.1 Processing

For the first model, we grouped the chat messages based on the time-stamps: Messages sent within a certain time period (one hour in our case) are grouped together as one feature element. We first trained a model using fastText library 181818https://github.com/facebookresearch/fastText/tree/master/python using the train_unsupervised method provided with model set to skipgram and dimension set to 500. We then used the trained model to create sentence-level embeddings for every chat log. Embeddings for chat logs within one group were stacked together to form one feature point.

For the second model, we represented the temporal information of chat as a vector of chat logs received by each player per day. For each player, we counted the number of comments received every day and created a vector recording those numbers. The lengths of the chat density feature vectors are variable for each player since the coverage of stream days is different.

4.3.2 Modeling

In both pretraining and fine-tuning stage, the language classifier is an embedding layer followed by a concatenation of the unigram, bigram and trigram extractor with average pooling and max pooling. The model architecture is depicted in Figure 3. We use the fastText sentence embedding as the pretrained weight for the embedding layer. Since the chat density feature is also sequential, the same architecture is applied except that the input is the number of comments received instead of the actual sentence tokens. The number of hidden units for all three n-gram extractors is set to be 256. A dropout rate of 0.3 is applied before the final linear layer. The model uses 1 as the batch size to avoid feature padding issues. The learning rate is 1e-3 throughout the training process.

4.4 Pooling

The second modeling stage has focused on synthesis of the rank A section probability estimates from the upstream models. This stage has involved three modeling challenges. First, each upstream model has generated multiple predictions per gamer.191919For example, the visual data models produce video-level predictions, whereas the text chat model produces chat cluster-level predictions. Second, not all gamers have data for each modality and, hence, the corresponding upstream modeling result. Third, the validation set of ranked gamers is limited to 79–98 gamers, implying sampling uncertainty concerns.

The pooling challenges above have been addressed using a generative modeling approach. Specifically, we have implemented a hierarchical Bayesian pooling model summarized in Figure 4. In this model, we assumed that each gamer’s intrinsic/latent skill is a sum of modality-specific random effects $\eta_{im}$ drawn from a normal distribution with hyper-parameters $\mu_{m}$ (the grand mean) and $\sigma_{m}$ (the grand variance). To ensure that we obtain a meaningful representation of latent skill, we also assumed that gamer $i$-specific observed rank A status $r_{i}$ as well as logits $l_{itm}$ from each upstream model $m$ for gamer’s modality-specific data point $t$ are generated by a distribution whose mean is centered at $\eta_{im}$. In addition, we allow for model $m$-specific source of variability $\tau_{m}$ to affect $l_{itm}$. Finally, to avoid using the data twice, the observed ranks A standings of the gamers in the training set used for upstream modeling have not been included in this model (as such, for these gamers the evidence of skill is represented only by the logit estimates). Additional details on the priors for each hyper-parameter are given in Figure 4.

The samples from the posterior distribution for $\eta_{i}$ and other model parameters have been generated via Markov Chain Monte Carlo (MCMC) sampling using NUTS implemented in Stan (Carpenter et al., 2017). We took 20,000 samples from four independent chains. Convergence has been confirmed on the last 10,000 draws using the Rhat statistic (Gelman et al., 1992). We represent the posterior distribution for $\eta_{i}$ using 40,000 samples (10,000 warm-up samples on four chains).

5 Evaluation

For the intrinsic evaluation of our modeling we use standard metrics such as the area under the ROC curve (AUC), F1, precision, and recall. These metrics are evaluated on the validation set of the ESEA gamers, using gamer-level averaging of the modality-specific scores. The threshold for F1, precision, and recall has been selected to optimize the F1 measure on the training set of ESEA gamers (also using scores averaged at the gamer-level). Finally, we use the maximum a posteriori probability (MAP) measure to represent distributional intrinsic skill estimates for the evaluation.

Extrinsic evaluation of our modeling consists in validating our findings using the data from the May 2019 ESEA ranks. This evaluation is not exhaustive because of the restrictions placed by ESEA on the number of gamer profile queries. Specifically, we have checked three groups of gamers: (1) gamers in the top 10% of the estimated skill distribution (42 gamers); (2) gamers in the bottom 10% of the estimated skill distribution (42 gamers); (3) gamers for whom we estimated high intrinsic skill, yet whose rank section was below A (63 gamers). We have compared the preponderance of rank A or higher in each group. We also statistically tested the association between rank improvements for the first and the third group using Kendall’s test of rank correlation (Kendall, 1938).

6 Results

Table 2 shows the intrinsic evaluation results of our modeling on the validation set of ESEA gamers. In all cases, our target is to predict whether a player is in the rank A section. Models using the triplet loss have produced gamer-specific rank A section probability scores that are better than random (i.e., AUC above 0.5) for all feature sets. On the other hand, models using the cross-entropy loss often generated worse than random predictions on the validation set. Pretraining on a related task has been helpful for models using audio features, chat text features, and chat temporal features. However, models using video features have not been improved by pretraining.

Table 2 also shows that, across modalities, the chat temporal features have produced the highest AUC of 0.784, F1 score of 0.709, and precision of 0.683. The best recall of 0.925 has been produced by the video motion features with cross-entropy loss.202020A model using cross-entropy loss with pretraining on the video spatial features has generated an even higher recall of 0.950. Yet, this model has also produced a worse than random AUC. Hence, this recall result needs to be interpreted with caution. Finally, while audio features appear to capture some of the association with the rank A section (AUC of 0.585), they are never the best in any of the other evaluation categories (i.e., F1, precision, and recall).

The pooled model used the best performing model for each feature set. It generated the best overall AUC of 0.797 as well as the highest F1 of 0.754 and the second highest recall of 0.86 and precision of 0.672. Note, however, that the results of the Bayesian pooling are not directly comparable to the modality-specific results for two reasons. First, in contract to the upstream models that generate MLE estimates, this model produces a posterior density of gamer-specific skill and we use MAP to summarize it. Second, unlike the upstream models, the pooling model is estimated using the rank information of the validation set.

Figure 5 and Figure 6 show the extrinsic evaluation of our gamer-specific skill estimates against May 2019 ranks. Figure 5 shows the top 10% of the gamers (according to our estimates), while Figure 6 shows bottom 10% of the gamers. Each gamer is represented by a horizontal box plot that summarizes his or her posterior skill density. Notably, gamers with more abundant observational data have tighter credible intervals. The color labels correspond to the May 2019 rank. Across the figures, we can see that the top of our inter-gamer distribution is dominated by the gamers who are predominantly rank A section, with a few moving on to ranks G and S. On the other hand, the bottom of our inter-gamer distribution has only a handful of rank A section gamers.

Figure 7 shows the gamers who are ranked below A in our data, yet for whom we have generated relatively high skill predictions. Looking at the May 2019 ranks of these gamers, we can see quite a few conversions to rank A, with more of these conversions occurring at the top of this distribution. This pattern is statistically significant (Kendall $\tau$=0.297, p-value=0.00447). Finally, we have also tested whether conversions to rank G or S at the top of inter-gamer skill distribution (shown in Figure 5) are systematically correlated with the predicted skill, but did not find a statistically significant pattern.

7 Discussion

Modeling and evaluation results that we report in Section 6 imply the task of automated e-sports talent scouting based on Twitch chat and video stream data is likely a feasible one. Despite the fact that a gamer’s leaderboard rank is affected by a plethora of factors that have not been represented in our modeling, such as teammate performance and equipment quality,212121Note that this information is available for some of the ESEA profiled gamers. However, it is unlikely that it would be present in an actual e-sports scouting scenario involving unranked gamers. we have been able to produce better-than-random predictions. Our models also have high recall, which may be important for capturing a larger talent pool for the subsequent human evaluation. Furthermore, our estimates of the intrinsic gamer skill—as opposed to his or her current leaderboard rank that may be obsolete or subject to randomness extrinsic to the gamer—are statistically significantly correlated with the improvements in gamers’ leaderboard standings.

We have found that the triplet loss produces the best results on all feature sets. However, pretraining is not uniformly helpful (e.g., models using video features achieve better performance if trained from scratch). We have also found that there are differences across the explored feature sets in terms of their effectiveness as predictors. Stronger features sets include temporal chat features (best AUC, F1, and precision) and motion video features (best recall). Performance of the temporal chat features is likely indicative of a strong relationship between the popularity of the player and the players’ gaming skill. In turn, our qualitative assessment of the visual features suggests that their effectiveness is likely tied to the specific visual elements that appear and persist on the screen whenever the player is in the “time-out” phase.

In physical sports, player agility is an important indicator of skill; yet, we have found that the motion visual features are not uniformly better than the spatial visual features. It is possible that the variational motion estimation methods that we have used to estimate optical flow are not as good as the latest, neural methods (Ilg et al., 2017). However, it is also possible that success of a gamer is related to more context-dependent, episodic moves (e.g., attacking from an ambush) that could have been missed either at the data collection (we record at most 5 minutes of gaming per day) or at the video processing stage (e.g., we aggressively downsample the videos as well as degrade image quality). We believe that further work on the motion visual feature representation could be beneficial. Furthermore, additional video data cleaning would undoubtedly be helpful, because there is no guarantee that the gamer is actually broadcasting the game, even though Twitch indicates that this is a CS:GO channel.222222We have noted broadcasts of irrelevant activity, advertisements, as well as gamers waiting for his or her teammates to start the game.

Feature sets that had not performed as well as we have hoped include audio features and chat content-based features. For the audio features, we have discovered early on that transcripts are not usable due to the multi-lingual nature of the Twitch platform232323A quarter of the videos are not in English, judging from the title of the stream. and relied only on the MFCC features. We found these features to be weak, possibly due to excessive quality degradation caused by downsampling (8K from 22.5K); with additional computational resources improvements of the MFCC feature quality could be explored. However, we note that gamers do different things during their gaming broadcasts: Some play music, some narrate their own game, some chat with teammates. Given the diversity and size of the audio data, we expect it to be quite noisy. Therefore, one of the improvements in audio feature extraction could be to focus on detection of specific sounds or tonal features from the audio track (e.g., yelling as an indicator of frustration), for example using SoundNet (Aytar et al., 2016).

The result for the chat content-based features is hardly surprising because this is an extremely noisy information channel. Twitch chats contain plenty of irrelevant information, e.g. hyperlinks for YouTube videos, emojis, nonsense characters. Viewers repeat the same utterance multiple times for emotional emphasis as well as use many self-created game-specific words. Additionally, a significant portion of the chat content is non-English, despite our explicitly request for the stream language to be English at the data collection stage. All of these Twitch chat properties make it difficult for a traditional language classifier to effectively classify the chats. To make the textual chat features more effective, it would be helpful to implement additional data cleaning such that those obvious noises can be removed. Also, standard n-gram extractors do not work well due to the uncommon language structure used by the game viewers. Therefore, another improvement could be exploring other language representations that specifically work for the Twitch language.

Other limitations of our work are related to the relatively small number of unique players with ranking data (under 500). As such, we have been able to set aside fewer than 100 unique players for validation of the first-stage, modality-specific modeling. This has not been sufficient for application of discriminative learning techniques to develop a fusion model for modality-specific predictions. Instead, we relied on the Bayesian pooling approach that integrated the available evidence, yet did not explicitly model interactions across feature sets. A productive extension of our work could include development of a model that learns modality representations jointly, at the first modeling stage.

8 Conclusions

We have investigated feasibility of a novel task of e-sports talent scouting based on multimodal Twitch chat and video stream data. In that, we have focused on CS:GO games broadcasted on Twitch during January 2019–April 2019. We have built two novel Twitch stream collections: One for 425 publicly ranked CS:GO gamers and one for 9,928 unranked CS:GO gamers. We have also developed a multimodal baseline model to make inferences about the intrinsic skills of the gamers. Our modeling results have been validated against May 2019 gamer ranks. In that we found that our intrinsic skill estimates are statistically significantly correlated with the future rank improvements. From this, we conclude that the automated e-sports talent scouting may, indeed, be a feasible task.

Researchers wishing to build on our work could investigate the benefits of improving Twitch data and/or representation quality, including development tied multimodal representations in the first modeling stage, or focus validating our conclusions using another online game represented on Twitch.