Hyacinth6B: A large language model for Traditional Chinese

TL;DR

Hyacinth6B is a lightweight large language model for Traditional Chinese that balances performance and resource efficiency through parameter-efficient fine-tuning, aiming to maximize capabilities with lower hardware demands.

Contribution

The paper introduces Hyacinth6B, a resource-efficient LLM for Traditional Chinese, utilizing LoRA fine-tuning to achieve high performance with reduced hardware requirements.

Findings

Hyacinth6B achieves competitive performance on Chinese NLP tasks.

Parameter-efficient fine-tuning reduces training costs.

Model demonstrates effective balance between size and accuracy.

Abstract

This research's primary motivation of this study is to address the high hardware and computational demands typically associated with LLMs.Therefore,our goal is to find a balance between model lightness and performance,striving to maximize performance while using a comparatively lightweight model. Hyacinth6B was developed with this objective in mind,aiming to fully leverage the core capabilities of LLMs without incurring substantial resource costs, effectively pushing the boundaries of smaller model's performance. The training approach involves parameter efficient finetuning using the LoRA method.