Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning

TL;DR

Hecto introduces a modular sparse experts architecture combining different neural modules for specialized reasoning, improving interpretability and efficiency across diverse tasks without sacrificing performance.

Contribution

It presents a novel heterogeneous MoE architecture that combines a GRU and FFNN experts, demonstrating improved interpretability and specialization in reasoning tasks.

Findings

Hecto matches or exceeds homogeneous baselines in various benchmarks.

Experts specialize in distinct reasoning types, enhancing interpretability.

Performance improves at larger batch sizes due to architectural flexibility.

Abstract

Mixture-of-Experts (MoE) models enable conditional computation by routing inputs to specialized experts, but these experts rely on identical inductive biases, thus limiting representational diversity. This static computation pathway is inefficient for inputs that require different types of reasoning and limits specialization and interpretability. We propose Hecto, a lightweight MoE architecture that leverages architectural heterogeneity by combining a GRU expert for temporal reasoning and an FFNN expert for static abstraction under a sparse Top-1 gating mechanism. Evaluated on three reasoning benchmarks (AG News, SST-2, HotpotQA) and a regression task (STS-B), Hecto matches or closely trails homogeneous baselines in performance despite receiving isolated input representations, while achieving clear expert specialization, with each expert aligning to distinct reasoning types (temporal vs static). At larger batch sizes, Hecto exhibits improved performance, benefiting from relaxed computational constraints that allow its heterogeneous architecture to optimize more effectively. Ablation results isolate architectural diversity as the source of Hecto's stability and interpretability across diverse reasoning tasks. Overall, Hecto establishes itself as a new benchmark for conditional computation, offering a principled framework for specialized reasoning in low-resource regimes with its model strength derived from principled specialization.