A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction

TL;DR

This study demonstrates that a domain-trained small language model can outperform large models in structured contract extraction, achieving high accuracy at significantly lower cost and operational risk.

Contribution

It introduces Olava Extract, a domain-specific Mixture of Experts model that outperforms large models in legal contract extraction with lower inference costs.

Findings

Olava Extract achieved the highest macro and micro F1 scores.

It reduced inference costs by 78% to 97% compared to large models.

It produced fewer hallucinations and unsupported extractions.

Abstract

This paper evaluates whether a domain trained Small Language Model (SLM) can outperform frontier Large Language Models on structured contract extraction at radically lower cost. We test Olava Extract, a self hosted legal domain Mixture of Experts model, against five frontier models. Olava Extract achieved the strongest aggregate performance in the study, with a macro F1 of 0.812 and a micro F1 of 0.842, while reducing inference cost by 78% to 97% compared with the frontier models tested. It also achieved the highest precision scores, producing fewer hallucinated and unsupported extractions, an important distinction in legal workflows where hallucinations create operational risk and downstream review burden. The findings shows that high performing, human comparable legal AI no longer requires the largest externally hosted models. More broadly, they challenge the assumption that commercially valuable enterprise AI capability must remain tied to ever larger models, massive infrastructure expenditure, and centrally hosted providers.