Analysis of a certain polycyclic-group-based cryptosystem

TL;DR

This paper analyzes the security of a polycyclic-group-based cryptosystem, demonstrating that despite resistance to length-based attacks, it can be compromised by an efficient polynomial-time algorithm.

Contribution

It provides the first polynomial-time attack on the Anshel-Anshel-Goldfeld protocol using specific polycyclic groups, challenging its assumed security.

Findings

Length-based attack has low success rate

Deterministic polynomial-time attack can break the protocol

Polycyclic groups may not be secure for this cryptosystem

Abstract

We investigate security properties of the Anshel-Anshel-Goldfeld commutator key-establishment protocol used with certain polycyclic groups. We show that despite low success of the length based attack the protocol can be broken by a deterministic polynomial-time algorithm.