Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks
Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo, Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, Jaewoo Kang

TL;DR
The paper introduces Meerkat, a family of open-source medical language models with up to 70 billion parameters, trained on textbooks and instruction data, achieving state-of-the-art accuracy and reasoning in medical benchmarks, surpassing prior models.
Contribution
Developed Meerkat, a new open-source medical language model family trained on synthetic reasoning data, significantly improving multi-step reasoning and benchmark performance over existing models.
Findings
Meerkat-7B surpasses USMLE passing threshold.
Meerkat-70B outperforms GPT-4 by 1.3%.
Models diagnose complex cases effectively.
Abstract
While recent advancements in commercial large language models (LM) have shown promising results in medical tasks, their closed-source nature poses significant privacy and security concerns, hindering their widespread use in the medical field. Despite efforts to create open-source models, their limited parameters often result in insufficient multi-step reasoning capabilities required for solving complex medical problems. To address this, we introduce Meerkat, a new family of medical AI systems ranging from 7 to 70 billion parameters. The models were trained using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. Our systems achieved remarkable accuracy across six medical benchmarks, surpassing the previous best models such as MediTron and BioMistral, and GPT-3.5 by a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
- 🤗RichardErkhov/dmis-lab_-_llama-3-meerkat-8b-v1.0-ggufmodel· 212 dl212 dl
- 🤗RichardErkhov/dmis-lab_-_llama-3-meerkat-70b-v1.0-ggufmodel· 69 dl69 dl
- 🤗RichardErkhov/dmis-lab_-_meerkat-7b-v1.0-ggufmodel· 266 dl266 dl
- 🤗RichardErkhov/dmis-lab_-_llama-3-meerkat-8b-v1.0-8bitsmodel· 1 dl1 dl
- 🤗RichardErkhov/dmis-lab_-_llama-3-meerkat-8b-v1.0-awqmodel
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsText Readability and Simplification · Topic Modeling
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · {Dispute@FaQ-s}How to file a dispute with Expedia? · 15 Ways to Contact How can i speak to someone at Delta Airlines · Attention Is All You Need · Label Smoothing · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Transformer · GPT-4 · Linear Layer
