Abrupt and spontaneous strategy switches emerge in simple regularised neural networks

TL;DR

This study demonstrates that simple regularised neural networks can exhibit sudden insight-like strategy switches similar to human 'aha-moments', driven by noise, gating, and regularisation during gradual learning.

Contribution

It shows that insight-like behaviour can emerge naturally in simple neural networks through specific mechanisms, challenging the view that insights require complex cognition.

Findings

Humans show sudden strategy switches linked to insights.

Neural networks mimic human insight-like switches under certain conditions.

Gating, noise, and regularisation are crucial for emergent insight behaviour.

Abstract

Humans sometimes have an insight that leads to a sudden and drastic performance improvement on the task they are working on. Sudden strategy adaptations are often linked to insights, considered to be a unique aspect of human cognition tied to complex processes such as creativity or meta-cognitive reasoning. Here, we take a learning perspective and ask whether insight-like behaviour can occur in simple artificial neural networks, even when the models only learn to form input-output associations through gradual gradient descent. We compared learning dynamics in humans and regularised neural networks in a perceptual decision task that included a hidden regularity to solve the task more efficiently. Our results show that only some humans discover this regularity, whose behaviour was marked by a sudden and abrupt strategy switch that reflects an aha-moment. Notably, we find that simple neural networks with a gradual learning rule and a constant learning rate closely mimicked behavioural characteristics of human insight-like switches, exhibiting delay of insight, suddenness and selective occurrence in only some networks. Analyses of network architectures and learning dynamics revealed that insight-like behaviour crucially depended on a regularised gating mechanism and noise added to gradient updates, which allowed the networks to accumulate "silent knowledge" that is initially suppressed by regularised (attentional) gating. This suggests that insight-like behaviour can arise naturally from gradual learning in simple neural networks, where it reflects the combined influences of noise, gating and regularisation.