External Correlates of Adult Digital Problem-Solving Behavior: Log Data Analysis of a Large-Scale Assessment

TL;DR

This study analyzes large-scale assessment log data to identify how adult digital problem-solving behaviors relate to demographic and cognitive factors, revealing that behavioral features outperform scores in predicting external characteristics.

Contribution

It introduces novel feature extraction methods from log data and demonstrates their effectiveness in predicting demographic and cognitive variables.

Findings

Behavioral features outperform scores in predictions.

Systematic associations between behaviors and demographics.

Features contain more information than final scores.

Abstract

Using the action sequence data (i.e., log data) from the problem-solving in technology-rich environments assessment on the 2012 Programme for the International Assessment of Adult Competencies survey, the current study examines the associations between adult digital problem-solving behavior and several demographic and cognitive variables. Action sequence features extracted using multidimensional scaling (Tang, Wang, He, Liu, & Ying, 2019) and sequence-to-sequence autoencoders (Tang, Wang, Liu, & Ying, 2019) were used to predict test-taker external characteristics. Features extracted from action sequences were consistently found to contain more information on demographic and cognitive characteristics than final scores. Partial least squares analyses further revealed systematic associations between behavioral patterns and demographic/cognitive characteristics.