# Evaluating the Information Security Awareness of Smartphone Users

## TL;DR

This paper introduces a new framework for evaluating smartphone users' information security awareness by combining subjective questionnaires with objective behavioral data from mobile agents and network traffic, validated through a long-term user study.

## Contribution

The paper presents a novel framework that integrates subjective and objective data sources to assess smartphone users' ISA against specific social engineering attack classes.

## Key findings

- Self-reported user behavior differs significantly from actual behavior.
- Objective data sources correlate highly with users' success in mitigating attacks.
- The framework effectively evaluates ISA in real-world social engineering scenarios.

## Abstract

Information security awareness (ISA) is a practice focused on the set of skills, which help a user successfully mitigate a social engineering attack. Previous studies have presented various methods for evaluating the ISA of both PC and mobile users. These methods rely primarily on subjective data sources such as interviews, surveys, and questionnaires that are influenced by human interpretation and sincerity. Furthermore, previous methods for evaluating ISA did not address the differences between classes of social engineering attacks. In this paper, we present a novel framework designed for evaluating the ISA of smartphone users to specific social engineering attack classes. In addition to questionnaires, the proposed framework utilizes objective data sources: a mobile agent and a network traffic monitor; both of which are used to analyze the actual behavior of users. We empirically evaluated the ISA scores assessed from the three data sources (namely, the questionnaires, mobile agent, and network traffic monitor) by conducting a long-term user study involving 162 smartphone users. All participants were exposed to four different security challenges that resemble real-life social engineering attacks. These challenges were used to assess the ability of the proposed framework to derive a relevant ISA score. The results of our experiment show that: (1) the self-reported behavior of the users differs significantly from their actual behavior; and (2) ISA scores derived from data collected by the mobile agent or the network traffic monitor are highly correlated with the users' success in mitigating social engineering attacks.