Safurai 001: New Qualitative Approach for Code LLM Evaluation

TL;DR

Safurai-001 is a new conversational coding LLM that matches recent models in performance and introduces GPT4-based MultiParameters for comprehensive evaluation, showing improved metrics over GPT-3.5 and WizardCoder.

Contribution

The paper introduces Safurai-001, a novel coding LLM with conversational capabilities, and proposes GPT4-based MultiParameters as a new evaluation benchmark for coding models.

Findings

Safurai-001 outperforms GPT-3.5 by 1.58% in code readability.

Safurai-001 outperforms WizardCoder by 18.78% in code readability.

The GPT4-based MultiParameters benchmark provides comprehensive insights into model performance.

Abstract

This paper presents Safurai-001, a new Large Language Model (LLM) with significant potential in the domain of coding assistance. Driven by recent advancements in coding LLMs, Safurai-001 competes in performance with the latest models like WizardCoder [Xu et al., 2023], PanguCoder [Shen et al., 2023] and Phi-1 [Gunasekar et al., 2023] but aims to deliver a more conversational interaction. By capitalizing on the progress in data engineering (including latest techniques of data transformation and prompt engineering) and instruction tuning, this new model promises to stand toe-to-toe with recent closed and open source developments. Recognizing the need for an efficacious evaluation metric for coding LLMs, this paper also introduces GPT4-based MultiParameters, an evaluation benchmark that harnesses varied parameters to present a comprehensive insight into the models functioning and performance. Our assessment shows that Safurai-001 can outperform GPT-3.5 by 1.58% and WizardCoder by 18.78% in the Code Readability parameter and more.