Can Smaller Large Language Models Evaluate Research Quality?

TL;DR

This study evaluates a smaller, downloadable LLM's ability to assess research quality, finding it correlates positively with expert scores, though less strongly than larger models, indicating smaller models can be useful for research evaluation tasks.

Contribution

It demonstrates that a smaller LLM can effectively estimate research quality scores, challenging the notion that only large models possess this capability.

Findings

Gemma-3-27b-it correlates positively with expert scores across fields.

Its correlation strength is 83.8% of ChatGPT 4o and 94.7% of ChatGPT 4o-mini.

Smaller LLMs do not significantly improve with score averaging or repetition.

Abstract

Although both Google Gemini (1.5 Flash) and ChatGPT (4o and 4o-mini) give research quality evaluation scores that correlate positively with expert scores in nearly all fields, and more strongly that citations in most, it is not known whether this is true for smaller Large Language Models (LLMs). In response, this article assesses Google's Gemma-3-27b-it, a downloadable LLM (60Gb). The results for 104,187 articles show that Gemma-3-27b-it scores correlate positively with an expert research quality score proxy for all 34 Units of Assessment (broad fields) from the UK Research Excellence Framework 2021. The Gemma-3-27b-it correlations have 83.8% of the strength of ChatGPT 4o and 94.7% of the strength of ChatGPT 4o-mini correlations. Differently from the two larger LLMs, the Gemma-3-27b-it correlations do not increase substantially when the scores are averaged across five repetitions, its scores tend to be lower, and its reports are relatively uniform in style. Overall, the results show that research quality score estimation can be conducted by offline LLMs, so this capability is not an emergent property of the largest LLMs. Moreover, score improvement through repetition is not a universal feature of LLMs. In conclusion, although the largest LLMs still have the highest research evaluation score estimation capability, smaller ones can also be used for this task, and this can be helpful for cost saving or when secure offline processing is needed.