LLMs' morphological analyses of complex FST-generated Finnish words

TL;DR

This study evaluates how well large language models can perform morphological analysis on complex Finnish words generated by FST, revealing their limited ability to generalize to unseen morphological forms.

Contribution

The paper provides an empirical assessment of state-of-the-art LLMs' capacity to analyze complex Finnish morphology, highlighting their limitations in generalizing to unseen forms.

Findings

GPT-4-turbo shows some ability to analyze complex forms

GPT-3.5-turbo struggles significantly

Smaller models like Llama2-70B and Poro-34B fail nearly completely

Abstract

Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.