Beyond Deep Learning: Speech Segmentation and Phone Classification with Neural Assemblies

TL;DR

This paper presents a biologically inspired neural assembly framework for speech segmentation and classification that operates directly on continuous speech, achieving competitive results without traditional deep learning training.

Contribution

It introduces an AC-based speech processing system combining neural encoding, hierarchical architecture, and cross-area updates, enabling boundary detection and classification without weight training.

Findings

Detects phone boundaries with F1=0.69

Achieves 47.5% accuracy on phone recognition

Operates without weight training

Abstract

Deep learning dominates speech processing but relies on massive datasets, global backpropagation-guided weight updates, and produces entangled representations. Assembly Calculus (AC), which models sparse neuronal assemblies via Hebbian plasticity and winner-take-all competition, offers a biologically grounded alternative, yet prior work focused on discrete symbolic inputs. We introduce an AC-based speech processing framework that operates directly on continuous speech by combining three key contributions:(i) neural encoding that converts speech into assembly-compatible spike patterns using probabilistic mel binarisation and population-coded MFCCs; (ii) a multi-area architecture organising assemblies across hierarchical timescales and classes; and (iii) cross-area update schemes for downstream tasks. Applied to two core tasks of boundary detection and segment classification, our framework detects phone (F1=0.69) and word (F1=0.61) boundaries without any weight training, and achieves 47.5% and 45.1% accuracy on phone and command recognition. These results show that AC-based dynamical systems are a viable alternative to deep learning for speech processing.