RedSage: A Cybersecurity Generalist LLM
Naufal Suryanto, Muzammal Naseer, Pengfei Li, Syed Talal Wasim, Jinhui Yi, Juergen Gall, Paolo Ceravolo, Ernesto Damiani

TL;DR
RedSage is an open-source cybersecurity-focused LLM trained with domain-specific data and agentic augmentation, demonstrating improved performance on cybersecurity benchmarks and general reasoning tasks.
Contribution
The paper introduces RedSage, a cybersecurity domain-adapted LLM trained with a novel augmentation pipeline and benchmark, advancing open-source cybersecurity AI capabilities.
Findings
RedSage outperforms baseline models on cybersecurity benchmarks by up to +5.59 points.
Domain-aware pretraining improves general reasoning and instruction-following.
RedSage is publicly available for deployment and further research.
Abstract
Cybersecurity operations demand assistant LLMs that support diverse workflows without exposing sensitive data. Existing solutions either rely on proprietary APIs with privacy risks or on open models lacking domain adaptation. To bridge this gap, we curate 11.8B tokens of cybersecurity-focused continual pretraining data via large-scale web filtering and manual collection of high-quality resources, spanning 28.6K documents across frameworks, offensive techniques, and security tools. Building on this, we design an agentic augmentation pipeline that simulates expert workflows to generate 266K multi-turn cybersecurity samples for supervised fine-tuning. Combined with general open-source LLM data, these resources enable the training of RedSage, an open-source, locally deployable cybersecurity assistant with domain-aware pretraining and post-training. To rigorously evaluate the models, we…
Peer Reviews
Decision·ICLR 2026 Poster
The paper is well-motivated: to improve a domain-specific capability (cybersecurity), go and curate data for it and fine-tune an existing model. The paper is fairly well-written and easy to follow.
There is not really any methodological novelty, given this is similar to other methods like FineWeb-Edu. I would have liked to see evaluation of larger models (at least 32B) to see how well the methodology transfers to stronger models (one might worry that the gap will shrink). It would also be good to get a closed model (e.g., GPT-5 or Claude) to get a ceiling for the new benchmark. The abstract claims that the fine-tuned model improves on OpenLLM leaderboard tasks, but looking at Table 6, it s
- The paper clearly demonstrates that cybersecurity curated pre- and post-training improves performance significantly without catastrophic forgetting of general knowledge - The cybersecurity-specific curated dataset is incredibly useful for future training tasks
- The method for open-ended Q&A evaluation is not adequately described. Line 317 mentions "prefix exact match or regex matching" and points to Appendix C.1 however neither the text nor the appendix provide sufficient details or references to clearly understand this evaluation. - The method for generating the instruction-tuned variant is not explained or referenced. A diagramatic view of how each of the variants was derived would be helpful.
Clear presentation: This work integrates all stages (CPT, SFT, DPO) with substantial data at each phase, which is clearly shown in Fig. 1. Readers can have a very straightforward view on how RedSage was trained on the dataset selected. Comprehensive Training data coverage: The integration of CyberFineWeb, curated RedSage-Seed, and agentic augmentation provides strong coverage across cybersecurity subfields. Proposed New Benchmark: What I am interested in is that this work expands prior benchma
May need computational cost analysis: While "compute constraints" are briefly mentioned in page 4 (CyberFineWeb section), there's no breakdown of training time, GPU-hours, or carbon footprint across stages. Adding some analysis on computational cost would help readers to form a general impression on the scale of RedSage training process. Limited human validation: One of my concern on this work is the data part is heavily reliance on LLM-based verification, which could introduce subtle self-rein
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsWeb Application Security Vulnerabilities · Information and Cyber Security · Spam and Phishing Detection
