Review of human decision-making during computer security incident analysis
Jonathan M. Spring, Phyllis Illari
TL;DR
This paper reviews existing standards and practices for human decision-making in computer security incident response, highlighting strengths and identifying gaps in guidance for prioritization and reporting.
Contribution
It provides a comprehensive review of current standards and practices, emphasizing the need for improved guidance on decision prioritization and interpretation during incident analysis.
Findings
Existing advice covers many specific tasks
Gaps in guidance on task prioritization under time constraints
Lack of clear methods for interpreting and reporting results
Abstract
We review practical advice on decision-making during computer security incident response. Scope includes standards from the IETF, ISO, FIRST, and the US intelligence community. To focus on human decision-making, the scope is the evidence collection, analysis, and reporting phases of response. The results indicate both strengths and gaps. A strength is available advice on how to accomplish many specific tasks. However, there is little guidance on how to prioritize tasks in limited time or how to interpret, generalize, and convincingly report results. Future work should focus on these gaps in explication and specification of decision-making during incident analysis.
| RFC 2196, §5.4 only [53] | 27035-1:2016 [73] |
|---|---|
| RFC 6545 [106] | 27037 [69] |
| RFC 7203 [142] | 27041 [70] |
| RFC 7970 [39] | 27042 [71] |
| RFC 8134 [74] | 27043 [72] |
| NIST SP 800-61 r.2 [34] | [86] |
| NIST SP 800-86 [84] | [1] |
| NIST SP 800-83 r.1, §4 only [132] | |
| [56] | [109] |
| [68] (The kill chain) | [65] |
| [23] (diamond model) | [27] |
| [33] | [28, ch. 2 ] |
| [113] | [91] |
| [81, ch. 5 only ] | [140] |
| [30] | [104] |
| Criteria | |||||
|---|---|---|---|---|---|
| Document | 1 | 2 | 3 | 4 | 5 |
| IEC 31010:2009 | ✓ | ✓ | ✓ | ✓ | |
| ISO 13485:2003 | ✓ | ||||
| ISO/IEC 17799:2005 | ✓ | ✓ | |||
| ISO/IEC 27000:2009 | ✓ | ✓ | |||
| ISO/IEC 27001:2005 | ✓ | ✓ | |||
| ISO/IEC 27002:2005 | ✓ | ✓ | |||
| ISO/IEC 27004:2016 | ✓ | ✓ | ✓ | ||
| ISO/IEC 27005:2011 | ✓ | ✓ | ✓ | ||
| ISO/IEC 27006:2011 | ✓ | ||||
| ISO/IEC 27006:2015 | ✓ | ✓ | |||
| ISO/IEC 27033-1:2009 | ✓ | ✓ | |||
| ISO/IEC 27033-4:2014 | ✓ | ✓ | ✓ | ||
| ISO/IEC 27035:2011 | ✓ | ✓ | ✓ | ✓ | |
| ISO/IEC 27035-1:2016 | ✓ | ✓ | ✓ | ✓ | ✓ |
| ISO/IEC 27035-2:2016 | ✓ | ✓ | ✓ | ||
| ISO/IEC 27041:2015 | ✓ | ✓ | ✓ | ✓ | ✓ |
| ISO/IEC 27043:2015 | ✓ | ✓ | ✓ | ✓ | ✓ |
| ISO/IEC TR 18044:2004 | ✓ | ✓ | ✓ | ||
| ISO/IEC TR 20004:2015 | ✓ | ✓ | ✓ | ||
| ISO/NP TS 11633-1 | ✓ | ✓ | |||
| ISO/TR 11633-1:2009 | ✓ | ✓ | ✓ | ||
| ISO/TR 11633-2:2009 | ✓ | ✓ | ✓ | ||
| ISO/TS 19299:2015 | ✓ | ✓ | |||
| Criteria | |||||
|---|---|---|---|---|---|
| Document | 1 | 2 | 3 | 4 | 5 |
| [139] | ✓ | ✓ | ✓ | ||
| [4] | ✓ | ✓ | ✓ | ||
| [30] | ✓ | ✓ | ✓ | ✓ | ✓ |
| [140] | ✓ | ✓ | ✓ | ✓ | ✓ |
| [42] | ✓ | ✓ | ✓ | ||
| [36] | ✓ | ✓ | ✓ | ||
| [92] | ✓ | ✓ | |||
| [144] | ✓ | ✓ | ✓ | ✓ | |
| [63] | ✓ | ✓ | ✓ | ||
| [155] | ✓ | ✓ | ✓ | ✓ | |
| [104] | ✓ | ✓ | ✓ | ✓ | ✓ |
| [22] | ✓ | ✓ | ✓ | ✓ | |
| [21] | ✓ | ✓ | ✓ | ✓ | |
| [14] | ✓ | ✓ | ✓ | ✓ | |
| [99] | ✓ | ✓ | ✓ | ✓ | |
| [17] | ✓ | ✓ | |||
| [150] | ✓ | ✓ | ✓ | ||
| [77] | ✓ | ✓ | ✓ | ✓ | |
| [80] | ✓ | * | ✓ | ||
| [79] | ✓ | ✓ | * | ✓ | |
| [81, ch. 5 only ] | ✓ | ✓ | ✓ | * | ✓ |
| Directness per Phase | Scope | ||||||||
| per Goal | |||||||||
| Document | Col | Anz | Rep | Fix | Int | LE | General | Type | Formal |
| 27035-1:2016 [73] | C | C | C | B | Un | Ont | Qual | ||
| 27037 [69] | D | M | Un | Ont | Qual | ||||
| 27041 [70] | C | C | C | Likely | Instr | ||||
| 27042 [71] | D | D | M | Un | Instr | ||||
| 27043[72] | C | C | C | B | M | Un | Ont | Qual | |
| RFC 2196, §5.4 only [53] | C | D | M | B | Likely | Instr | |||
| RFC 6545 [106] | C | C | D | N | B | Un | Ont | Formal | |
| RFC 7203 [142] | C | C | N | N | N | Un | Ont | Formal | |
| RFC 7970 [39] | C | D | M | M | M | Un | Ont | Formal | |
| RFC 8134 [74] | C | B | B | Un | Study | ||||
| NIST 800-61 [34] | C | D | C | M | M | Un | Adv | Qual | |
| NIST 800-83 §4 [132] | D | C | B | Un | Ont | Qual | |||
| NIST 800-86 [84] | C | C | High | Study | Qual | ||||
| [56] (ENISA) | D | N | Un | Study | Qual | ||||
| [1] (CERT/CC) | C | B | Un | Ont | Qual | ||||
| [86] (CERT/CC) | C | D | C | M | Likely | Adv | Qual | ||
| [109] (CERT/CC) | C | C | C | B | High | Ont | Perf | ||
| [113] (JHU & US-CERT) | C | C | B | B | High | Ont | Qual | ||
| [68] (IC) | C | C | M | Un | Adv | Qual | |||
| [23] (IC) | C | D | M | High | Adv | Perf | |||
| [65] (CIA) | D | B | B | B | Wide | Instr | Qual | ||
| [81, ch. 5 only ] | D | M | High | Instr | Qual | ||||
| [28, ch. 2 ] | C | D | D | B | Wide | Ont | Qual | ||
| [104] | C | C | C | M | N | Un | Study | Qual | |
| [27] | D | C | D | M | Likely | Ont | Qual | ||
| [33] | C | C | C | B | Un | Ont | Perf | ||
| [91] | D | N | N | N | Likely | Instr | Formal | ||
| [140] | C | D | C | N | M | N | Likely | Study | |
| [30] | C | D | B | Likely | Study | ||||
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
datatype=bibtex]ucl.bib datatype=bibtex]mine.bib datatype=bibtex]NSPW2017.bib datatype=bibtex]work-20150722.bib datatype=bibtex]rfc.bib datatype=bibtex]cited-by-FIRST.bib datatype=bibtex]cited-by-IC-relevant.bib datatype=bibtex]cited-by-ISO.bib datatype=bibtex]csur-related.bib datatype=bibtex]nist-sp.bib datatype=bibtex]uniq-from-textbook.bib
Review of Human Decision-making during Incident Analysis
Jonathan M. Spring, Phyllis Illari
(April 2018)
Abstract
We review practical advice on decision-making during computer security incident response. Scope includes standards from the IETF, ISO, FIRST, and the US intelligence community. To focus on human decision-making, the scope is the evidence collection, analysis, and reporting phases of response. The results indicate both strengths and gaps. A strength is available advice on how to accomplish many specific tasks. However, there is little guidance on how to prioritize tasks in limited time or how to interpret, generalize, and convincingly report results. Future work should focus on these gaps in explication and specification of decision-making during incident analysis.
1 Introduction
The purpose of this literature review is to identify the structure of human decision-making during computer-security incident response (shortened as incident response if the usage is unambiguous) and identify the existing research programs that underpin the study of each element of the decision-making structure. The utility of such a literature review is to identify opportunities to expand research programs, as well as to identify loci of research programs best strengthened by interdisciplinary work or requiring interfield theories.
Incident response is an attractive topic because it anchors the whole field of information security. When information security is not responding to an incident, it is either preparing for one or learning from past incident responses.111Prevention of future incidents may fruitfully be discussed as independent from incident response, but in practice prevention techniques are almost all adapted from the lessons learned after responding to an incident. Preparation and learning are each complex and independent. However, their crux is incident response. We restrict our focus to human-centric decision making as opposed to automated or machine-learning techniques. Such automation is clearly crucial to incident response. However, deciding what automation to build and use remains a human decision, as does how to interpret and act on results.
Information security is important, yet breaches are usually detected months after they occur [151]. Professionals suggest judging whether an organization has a good security posture by their incident response capabilities [129, ch. 15].222See also: “The question you must accept is not whether security incidents will occur, but rather how quickly they can be identified and resolved.” https://www.gartner.com/smarterwithgartner/prepare-for-the-inevitable-security-incident/ The question we address in this review is: how do incident responders make decisions during an investigation?
Successful guidance on this question improves incident response, and thereby all of information security. However, the question is deceptively simple, and the use of ‘how’ is ambiguous. More specifically, we want a satisfactory process by which incident responders generate and prioritize questions to ask, a satisfactory process of answering these questions under time constraints, and a satisfactory process for how these answers lead to decisions. In summary, there is ample advice on what questions to ask and how to answer a given specific question, but little on how to prioritize questions or interpret results. Therefore, we identify a significant lacuna in the existing literature on decision making and evidence evaluation during incident response. We suggest some directions for filling this gap use mathematical modeling and philosophy of science.
The immediate challenge of this literature review is an explosion of scope. The scope is too broad for at least two reasons: the topic is of broad application with many sub-parts, and the academic literature is not the only relevant source. Practitioners are a necessary source if the review is to adequately capture the state of the art. Besides simply increasing the volume of publications to review, information security practitioners do not publish with the same reliability nor norms as customary for input to an academic literature review. This variability makes it difficult to evaluate contributions on a single appraisal strategy.
We approach this problem in the two broad parts. The first part covers scope and problem definition. Section 2.1 defines the term computer-security incident response, and which aspects are in scope for our discussion. Section 2.2 defines the scope in terms of publication venues included in the review. Section 3 provides evidence that no literature review with similar scope and goals has been conducted. Section 4 defines our method for reviewing the state of decision-making recommendations during computer-security incident response.
The second part covers results and proposed research. Section 5 provides the results of the incident response decision-making literature review. The first four subparts cover each of the four venues in scope, and Section 5.5 reports the evaluation of documents cited by those documents from the search results that are evaluated to be relevant. Section 6 critically analyzes the results of the reviews. Section 7 suggests gaps and some broad directions that might be followed to begin to fill them.
2 Scope
This section limits the scope of the literature review in two distinct aspects: the definition of the topic and the publication venues. We will restrict the definition of Computer Security Incident Response (CSIR) to three subtasks during investigation: evidence collection, analysis, and reporting. We will restrict publication venues to relevant international standards and academic literature that is referenced therein. Specifically, the search venues will be the Internet Engineering Task Force (IETF), the International Organization for Standardization (ISO),333Although ISO standards are only available for a fee, the terms and definitions as used in the relevant standards (the 27000 series) are freely available. Forum of Incident Response and Security Teams (FIRST), and documents understood to represent the United States of America (US) intelligence community (IC).
As far as possible, we will use standard definitions for terms, preferring global, consensus, freely-available definitions. Explicitly, in order of preference, the IETF, ISO, and FIRST. This ordering is based on the extent of consensus (the IETF membership is broader than FIRST) and the openness of the definitions. Otherwise, the choice of established definitions for jargon is primarily for clarity, and to compress the discussion; we assume familiarity with the terms in the IETF Internet Security Glossary [130].
The scope is not the traditional academic venues. The operational reason is to focus on what incident responders actually do. Ideally this would take the form of first-hand accounts; however, cybersecurity is a sensitive topic. Chatham House Rules444Chatham House Rules indicates a situation in which the information or content of a meeting, discussion, or presentation may be disclosed but the source of the information may not be identified, implicitly or explicitly. This request is made by the speaker prior to disclosing the information. and non-disclosure agreements abound in the discipline; these norms within the security community further frustrate any usual academic publication expectations. It would be impossible to evaluate the selection bias or outright deception within studies. The incident response standards at least form an honest baseline of what is expected of a competent practitioner. The assumption is that this review applies to competent practitioners, where competent is defined by consensus among practitioners and codified in the standards. We do not empirically address the extent to which competence is common. Due to the community norms of secrecy, norms documented by [141], a comprehensive evaluation seems impractical.
Section 3 provides evidence that the academic literature does not systematically cover our scope. The method to justify this is a search through two common sources of literature reviews, Association for Computing Machinery (ACM) Computing Surveys and the Institute of Electrical and Electronic Engineers (IEEE) Security and Privacy “systematization of knowledge” papers. In summary, no relevant surveys have been published on human decision-making during CSIR analysis. While the reason why is unclear, the task is not made easier by the natural secrecy of practitioners.
A further consideration in focusing on standards as the starting point is simply to prevent an explosion of documents to review. A cursory Google Scholar search for “computer security incident response” and “digital forensic investigation” each return about 2,000 results. Alternatively, searches in the ACM Guide to Computing Literature and IEEE Xplore databases for “computer security incident response” each return 20-25 results (both on Sep 1, 2017). Many of these results are obscure, and for those that are not it is challenging to evaluate their operational impact. Going straight to the standards bypasses this question of impact – the remit and authority of standards is explicit.
Finally, one author draws on six years of personal experience working at a research center interacting with a variety of practicing incident responders.
2.1 Scope—Topic
The first task is to situate computer-security incident response within a context of what it excludes and what falls under it. Incident response is a subspecies of business continuity planning or continuity of operations. Continuity planning may be a response to man-made or natural events, and either physical or digital events. A military invasion is a man-made, physical event; a hurricane is a natural, physical event. Computer-security incident response is only in response to security incidents that are primarily digital, where a security incident is something “contrary to system policy” [130]. Thus, accidents of all kinds are out of scope; though distinguishing apparent accidents from malicious acts is included. Intentional physical destruction of computing resources is also out of scope of computer-security incident response [19].
Narrowing the focus further, incident response is a task within incident management.555The term “incident management” does not appear in IETF documents consistently. [145] describes Incident Object Description Exchange Format (IODEF) [40] as a protocol for “exchange of incident management data,” but the term “incident management” does not appear again in [145], and not even once in [40]. [73] defines “information security incident management” as “exercise of a consistent and effective approach to the handling of information security incidents.” FIRST does not provide a definition itself, but [50] recommends the CERT® Coordination Center (CERT/CC) documentation on incident management. Trammell and Danyliw both worked at CERT/CC, so this is probably the source of the informal reference in the IETF documents. The CERT/CC phases are consistent with the [73] phases of plan and prepare; detection and reporting; assessment and decision; responses; and lessons learnt. We prefer the CERT/CC definitions as they are public (vice the ISO standards), and recommended by FIRST (thus in scope of using global, consensus-driven definitions). CERT/CC definitions of incident management [1, 109] locate incident response as independent from activities such as preparation, improving defences, training, financing, and lessons learnt. [109] surveys practices including those by CERT/CC and ISO; six tasks are included as part of incident response: monitoring, detection, evidence collection, analysis, reporting, and recovery. These six tasks form the core topic of this survey of incident response.
The human-centric decisions that are elements of these six incident response tasks vary in importance. Analysis, reporting, and recovery are almost wholly human-driven. Monitoring is almost wholly automated, while detection and evidence collection are a mixture. Where detection is automated, say in an intrusion detection system (IDS),666[130] refers to [10] for the definition and guidelines for IDS, which has been superseded by [127]. it is out of scope. Decisions about what detection rules to implement in an IDS are part of the preparation or improving defences phases of incident management, as a result of lessons learnt, and thus are also out of scope. Actual human intrusion detection is rare, and when it occurs is usually the result of analysis during incident response to some other, automatically-detected incident. Therefore, the focus on human-driven incident response investigation excludes monitoring and detection.
The IETF [130, 19] and CERT/CC define neither “investigation” nor “forensics” in relation to the incident management process. [72] places investigation as the centrepiece of incident management, where the principles of incident management are to “give guidance on the investigation of, and preparation to investigate, information security incidents” [73, §0 ]. In this way, ISO uses “investigation” as a near-synonym to “response” in the IETF and FIRST literature.
The use of ‘incident’ emphasizes that the investigation is oriented towards the violation of some policy, possibly but not necessarily a law. Thus, modelling or analysing online crime is an investigation, and so is IT staff looking into a usage policy violation. Incident response or investigation is entwined with cybersecurity because one essential aspect of a defence strategy is feedback from investigation to ‘preparation’ and ‘protection’ [1]. However, detailed discussion of preparation and protection is placed out of scope.
Incident response, per IETF and FIRST, explicitly includes remediation. ISO [72] treats remediation and response as separate from investigation. In determining scope, we follow ISO and exclude recovery. Note that both sets of standards agree that clear reporting is the proper output of incident analysis, and any recovery follows reporting. However, it does seem clear that recovery follows a different decision process than analysis, and the two should be treated separately. Within the six tasks identified within incident response, three are left in scope:
- •
evidence collection
- •
analysis
- •
reporting
These three seem too tightly coupled to separate, and are described consistently across the international standards organizations.
For each of these three topics, the concern is primarily with how an individual analyst makes decisions during these three phases. What tool or what language the analyst or investigator uses to make these choices is not germane and is out of scope. This is not a review of available security tools, platforms, or data exchange formats. The goal is to survey how analysts enumerate options, evaluate choices, generalize results, and justify these steps.
2.2 Scope—Publication Venues
Incident response and investigation includes professional and business aspects; therefore the scope of viable sources incident response practices cannot justifiably be limited to academic sources. As [137] documents, the science of security is an unsettled area of research rather than an area with anything like standards. In fact, traditional academic publication venues contain little if anything about day-to-day incident response practices; academics do not do incident response themselves. [141] seems to mark the first anthropological study of a Computer Security Incident Response Team (CSIRT)777CSIRT is the general term, and will be used unless referring to a specific organization. members and their attitudes, but this literature is not about the actual process of incident response; that is covered in the professional literature.
Therefore, to understand current incident response practices the scope of the review is internationally-relevant standards and whatever literature is referenced therein. The history of standards as its own industry is complex in its own right [138]. The Internet and IT standards are formed by heterogeneous processes by a wide variety of actors [112]. Security-relevant standards are beginning to be seen as having their own unique requirements, distinct from IT standards generally [87]. However, it is a separate project to analyse how CSIR standards have come to be. The standards in this review are taken as-is, with the understanding that any interpretations should be made cautiously because the standards may not cleanly fit in to existing studies of how and why other IT standards are created. More than other IT standards, CSIR standards are likely a codification of tacit practitioner knowledge [110].
The scope of publication venues is limited to ISO, IETF, FIRST, and the US intelligence community (IC). This choice is based on what organizations are relevant in influencing or describing international incident response practices, which is in turn based in the history of the governance of the Internet. We mitigate potential over-restriction of focus by including any documents cited by standards publications. The reasoning for selecting these organizations specifically is as follows.
ISO and the International Telecommunications Union (ITU) are the authoritative technology standards makers [112, p. 11]. The US federal government plays a dominant role in Internet development and standards, through the original Advanced Research Projects Agency (ARPA) development under the Department of Defense (DoD) and subsequent stewardship under the Department of Commerce.888Two important sub-parts of Commerce are Internet governance by the National Telecommunications and Information Administration and standards by National Institute of Standards and Technology (NIST).
ISO is de dicto where one looks for international standards. Each nation-state is allowed to have one member is ISO, namely the official national standards body representing all the industry-based bodies in each country. It is a federation of federations, representing a multitude of industries. ISO standardizes things like the two-letter country codes (which have been adopted as Domain Name System (DNS) top-level domains), paper sizes, and credit cards. The ITU and their CIRT program999http://www.itu.int/en/ITU-D/Cybersecurity/Pages/Organizational-Structures.aspx seems promising in name; however, their website publishes little besides an events list. It appears that content is provided by FIRST members, companies, or other consultancies; the ITU does not produce its own incident response materials or standards. This leaves only ISO in scope of the potential authoritative international standards bodies.
On the other hand, the IETF is the de facto place to go for international Internet standards because, for all intents and purposes, its standards are the Internet. The IETF “doesn’t recognize kings—only running code” and creates more pragmatic, open (freely-available) standards [112, p. 12]. Open standards happen to have won out on the Internet; IETF standards like Transmission Control Protocol / Internet Protocol (TCP/IP), DNS, and Border Gateway Protocol (BGP) underpin every Internet connection. For a background history of how the precursor to the IETF came to this dominant role, see [59]. The other main open-standards body is the World Wide Web Consortium (W3C), which standardizes Hypertext Transfer Protocol (HTTP) and Extensible Markup Language (XML), for example. W3C stays narrowly focused on web standards, and although this includes important web security considerations, W3C does not work on incident management, so we mark the group as out of scope.
FIRST is not part of this longer information and communications technology (ICT) standards history. It was formed in 1990 specifically to coordinate among and represent the interests of CSIRTs globally. FIRST’s mission includes developing and sharing best practices, as well as creating and expanding incident response teams [49]. FIRST is the one and only global organization representing those who do human-centric incident response tasks. FIRST’s work with United Nations (UN) agencies like the ITU also testifies to its global influence. It is naturally included as in-scope.
There are three organizations one might consider naturally in scope that are excluded. These are Agency for Network and Information Security (ENISA), NIST, and the US DoD. However, within the gray area between NIST and the US intelligence community, we identify a fourth set of de facto standards.
ENISA is specifically focused on CSIRTs and information security. The European Union (EU) makes available an independent evaluation of ENISA’s limited activities.101010“Annual ex-post evaluation of ENISA activities” https://www.enisa.europa.eu/about-enisa/annual-ex-post-evaluation-of-enisa-activities Its function is coordination and capacity building among EU-member CSIRTs and to provide advice on some EU policies. While EU directive 2016/1148 will increase ENISA’s statutory powers when it comes into effect in November 2018, at this point ENISA has little authority to force member states to take its advice. In the scheme of incident response practices, ENISA – founded in 2004 – is quite young. ENISA documents are informational, the one document interfacing with EU standards is an extended definition of the term “cybersecurity” and what EU work is done related to it [18]. Oddly, the document does not even mention incident response as an area within cybersecurity,111111Despite the fact that ENISA sponsored an informational document on evidence gathering aimed at increasing understanding between CSIRTs and law enforcement [5]. so it seems safe to leave ENISA out of scope.
NIST is a difficult organization to place in or out of scope. It is part of the Department of Commerce, and so has loose ties to the remaining Internet stewardship invested in the National Telecommunications and Information Administration. Strictly, NIST merely sets policies for how the US federal civilian government secures its IT infrastructure and responds to incidents [34]. This Federal Information Security Management Act (FISMA) policy responsibility is a part of NIST’s larger role of “advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life” [147]. Through this role, NIST standardized AES, which is the de facto global standard encryption algorithm. NIST documents and standards are also cited by the IETF, ISO, and FIRST, elevating certain NIST work from a national to international status. We shall consider NIST generally out of scope; however, many NIST publications will be considered as works cited by the international standards organizations.
There are two US federal government units that do not fall under NIST’s authority – the DoD and the Central Intelligence Agency (CIA). These two organizations have not published incident response standards as openly as NIST or these other standards organizations. The DoD does have other cybersecurity documents that are tangentially relevant. The Orange Book [16], which evaluates trusted computing platforms, is relevant background material for incident responders.
The questions the DoD and its sub-agency the National Security Agency (NSA) have raised around whether cybersecurity is, broadly, a science (see, e.g., [96, 54, 82]) could inform evidence evaluation in incidence response because evaluating evidence properly is a primary scientific activity. While these DoD projects ask the right questions about science to help with incident response, they have generally concluded security is not (yet) a science, and so there is little advice. [137] argues this conclusion is ill-founded and excessively pessimistic. However, the relevant point for this review is that the science of security literature does not advise CSIR.
While the main part of the DoD is out of scope, the intelligence community aspects of the US federal government do provide adequate documents. The DoD and CIA are generally not forthcoming with more conventional descriptions of their incident response practice. However, given that NIST is not authoritative over the IC, one would expect them to develop their own standard practices. Documents related to the practice of the US IC are occasionally published, with IC attribution either explicit or implicit. Three such documents are relevant to evidence collection and analysis in incident investigation, forming what is essentially a de facto standard. The first is a textbook published by the CIA and used to train intelligence analysts [65] whose methods are applicable to CSIR. The second is a pair of documents, the kill chain model of computer network attacks [68] and the diamond model of intrusion analysis [23]. Unlike the textbook, these documents are not explicitly acknowledged as standard analysis methods within the defence and intelligence communities. However, the diamond model paper is published by the DoD publisher, the Defense Technical Information Center.121212See http://www.dtic.mil/docs/citations/ADA586960. The diamond model builds on the kill chain. Given that Lockheed Martin, a US defence contractor, published the kill chain, it seems the papers are from overlapping communities. Although it is tenuous to term three documents a ‘standard,’ it is clear from the content that they come from a practitioner community and are one of the clearest available expressions of intrusion investigation methods publicly available. Therefore, it is necessary to place them in scope for discussion.
The US intelligence agencies exercise out-sized international influence. The US is part of an intelligence sharing alliance known as the five eyes, which includes Australia, Canada, New Zealand, and the United Kingdom. As the biggest partner in this alliance by far, what the US intelligence practitioners do is probably accommodated, if not directly copied, by the other countries’ services.
US military influence goes beyond the five eyes. The North Atlantic Treaty Organization (NATO) is the biggest alliance the US leads, with 28 other countries. NATO intelligence is also presumably influenced by five eyes, as Canada and the UK also play a big role. The US tends to supply logistics and intelligence support in its alliances, so intelligence standards are likely to influence allies. Other locations which cooperate extensively with the US include Israel, South Korea, Japan, and the Philippines. By virtue of these alliances, it is reasonable to assume that intelligence professionals in all these places are relatively closely aligned with US intelligence standards. These alliances end up including most of the global military and intelligence spending. Essentially only China and Russia are excluded, and the two of them account for 15-20% of global military spending. Thus, although there are rather few IC documents, and they are focused on the US, they should provide information about how a large swath of such practitioners make decisions.
In summary, this review will include the IETF, ISO, FIRST, and available intelligence community documents as in-scope publication venues for incident investigation standards of practice for evidence collection, analysis, and reporting. The review will exclude the ITU, W3C, ENISA and US federal civilian government departments and agencies as out of scope due to either limited applicable content or limited jurisdiction. The most borderline organization is NIST, which occasionally has standards canonized by the in-scope venues; the review will only include those NIST standards cited or adopted explicitly by the four in-scope venues.
Section 4.1 describes the method for determining which standards are relevant within these venues.
3 Novelty
This section provides a brief structured survey of the survey literature in order to demonstrate that our intended scope, as defined in Section 2, has not been previously surveyed. Lack of related surveys in two academic venues provide evidence: IEEE Security and Privacy Systematization of Knowledge (SoK) papers and ACM Computing Surveys (CSUR) journal. All 35 extant SoK papers (as of August 1, 2017) are appraised for relevance. For ACM CSUR we apply a keyword search to the corpus.
IEEE S&P has published 35 SoK papers since the venue initiated the SoK publication option in 2010. We make an exhaustive evaluation of relevance based on title and abstracts. Our basic relevance criterion in this case is if the SoK is about designing or evaluating investigations of maliciousness. Of the 35 SoK papers, only [64] and [123] are applicable to our project. I have done prior work explaining why each of these is insufficient for our purposes. [137] addresses the shortcomings in [64]. [61] expands and generalizes the good work of [123]. None of the SoK papers systematize knowledge of incident response, investigation, or analysis.
Our CSUR keyword search uses Google Scholar, limiting to publications in “ACM Computing Surveys” between Jan 1, 2007 and Aug 1, 2017. We use the keywords used in the main study, as described in Section 4.1. However, CSUR is a sufficiently different venue from our intended scope that we sometimes find different keywords more useful. The surveys returned by the following search terms are included in Table 1. Quotes are applied to the search as listed.
“computer security incident response” 2. 2.
“incident investigation” 3. 3.
“incident management” 4. 4.
“computer security” & “evidence collection” 5. 5.
“incident response” & analysis 6. 6.
“security incident” & investigation 7. 7.
“computer security” & incident investigation 8. 8.
“computer security” & incident analysis
These eight searches within CSUR return 22 unique results. Note in particular that the two most precisely relevant searches return no matches. As the search terms are expanded to include more general, related terms, we find a handful of possibly relevant results.
Two search terms were tried on CSUR but returned too many clearly irrelevant results to be considered useful. Namely, {“computer security” & analysis} with 79 results and {“computer security” & reporting} with 25. Any relevant papers appear to be included in the 22 captured by Table 1.
To determine whether any of these 22 surveys already adequately cover our topic of interest, we set out three relevance criteria. The survey must:
relate to reconstructing past events (i.e., forensics) rather than prediction; 2. 2.
focus on the technical- and knowledge-based decisions and processes, rather than management processes; 3. 3.
use our target level of abstraction to discuss the problem of incident response, investigation, or analysis (human decisions during the process), rather than tool development, without being so abstract as to make implementation impractical.
These criteria are marked in Table 1, based on each paper’s abstract. Some papers deserved a look beyond their abstracts.
[46] is the only survey that meets all three criteria, based on their abstract. However, their focus is quite different from our intended focus. They discuss automation of law-enforcement criminal investigation using computer science techniques. There may be overlap with computer-security incident response, in that some subset of law enforcement cases involve criminal action against computers. However, the focus of [46] is what [6, p. 3 ] call, quoting the European Commission, “traditional forms of crime… committed over electronic communication networks and information systems.” Incident response and investigation focuses on a different category, “crimes unique to electronic networks,” as well as organizational policy violations that are not illegal under the relevant jurisdiction. Finally, [46] focus on automation of police tasks, whereas our focus would be on the investigator’s decision process in, among other things, choosing which automation techniques to use and how to evaluate the evidence they provide. These various differences make a clear case that our intended survey topic is sufficiently distinct from [46].
[47, p. 1 ] aims to identify “techniques to assist human analysts in assessing … whether a given sample deserves closer manual inspection.” It is, in fact, a survey of software tools and their features, and does not discuss how an analyst should use them.
Many papers in Table 1 meet the technical criterion (#2) and fail the other two criteria. This pattern tends to be about some specific subset of network defense – for example, making better passwords, intrusion detection systems, or web browser defenses. These tools are certainly used and evaluated as part of security management, and are important considerations. However, these details are tangential to making decisions during incident response and investigation.
These reviews of the available survey literature demonstrate a lacuna; we lack a survey of incident response and investigation practices. This omission matters. Most security research and security management requires, directly or indirectly, “ground truth” evidence from incident response teams. Research and management need this “ground truth” evidence to evaluate any other security infrastructure, plans, defenses, or policy. However, it seems there is no systematic review of how this evidence should be collected, analyzed, and reported. One must understand these steps in order to properly interpret any such evidence. Therefore, although our topic is narrow, it has far-reaching impact on information security more generally.
4 Methods
The scope of our topic is restricted to evidence collection, analysis, and reporting in human-driven computer security incident response. We further restrict our review to internationally-recognized standards. This choice keeps us as in touch as possible to actual professional practice without violating confidentiality around incident response, which organizations justifiably do not often disclose in detail.
We explain our methodology in three parts. First, our search strategy. Secondly, our appraisal strategy for whether to include what we find in our synthesis. And finally, how we plan to synthesize the results into a coherent statement of current practice.
4.1 Search strategy
The major determining factor in our literature search strategy is the scope of publication venues, as justified in Section 2.2. Our search comes in two parts. First, we perform keyword searches in the relevant web archives. Secondly, we extract references from valid hits on these searches and include the referenced documents as sources to appraise.
The IC documents are selected by fiat, given the secretive nature of the community. Due to the idiosyncratic nature of the IC publication and publicity processes, there is no sense in a keyword search strategy. We have arrived at the three core documents we shall consider as the “standards” from this community essentially by word of mouth through interaction with practitioners.
Each of IETF,131313https://www.rfc-editor.org/search/rfc˙search.php FIRST,141414https://first.org/standards/ and ISO151515https://www.iso.org/standards.html have dedicated web pages. For IETF and ISO, we use their site-based search engines that cover their respective corpora of standards. FIRST has a smaller, more focused corpus of work, such that it does not have a site specific search engine; we use Google and prepend the term “site:first.org” to focus the search.
Quotes are applied to the search as listed. The keywords employed are:
“computer security incident response” 2. 2.
“incident investigation” 3. 3.
“incident management” 4. 4.
“computer security” & “evidence collection” 5. 5.
“computer security” & analysis 6. 6.
“computer security” & reporting
We also added or modified terms slight to accommodate each search venue. The IETF RFC search tool does not accommodate mixing quoted phrases with other terms, so for terms 4, 5, and 6 the quotes were removed. We added the following terms to the IETF search:
- •
“incident response”
We added the following terms to the ISO search, after it became apparent from searches 2 and 3 that the ISO documents do not use the term “computer security” but rather “information security”:
- •
“information security” & “evidence collection”
- •
“information security” & analysis
- •
“information security” & reporting
All the documents returned by this search strategy are appraised using the methods of Section 4.2. We then take a further search step and extract the references from those documents that pass the appraisal. We only include any cited documents which are publicly available (or, in the case of ISO, readily available). All the documents extracted from the references are appraised using the same methods to determine whether they are included in our review, independent of the document that cited it.
4.2 Appraisal strategy
The purpose of the appraisal is to determine whether each document is within the scope of evidence collection, analysis, and reporting for incident response. Our cutoff date for inclusion in the review is that the document must be published by July 1, 2017.
Furthermore, because standards follow an orderly progression, they may be superseded or amended by future work. Drafts are also commonly published for public comment before being finalized. We exclude any standard superseded as of July 1, 2017, and incorporate any amendments finalized by July 1, 2017. We note the existence of drafts on new topics, but exclude their content from the review.
Our specific inclusion criteria for whether the content is in-scope are the following:
- •
Target audience as expressed by author includes security professionals
- •
Topic is within scope, namely it applies to one of the following parts of computer-security incident response (or some clear synonym thereof)
- –
evidence collection
- –
analysis
- –
reporting
- •
Topic is on investigator practice (rather than implementation of software or managerial considerations related to CSIRTs)
- •
Document is finalized (not a draft) and not explicitly superseded as of July 1, 2017
- •
Document is available in English
A document must satisfy all of these criteria in order to be included in the review.
4.3 Synthesis methods
The input at this stage of the method will be all documents that are in scope; they will be documents for security professionals about investigator practice during evidence collection, analysis, and reporting. Our synthesis goal is to evaluation the nature and quality of advice that these documents provide about making decisions during these phases of incident response.
As a prelude to our synthesis, we classify advice on these topics in several ways: to which phases the document applies, the directness with which the document applies to each phase, the type, investigative goals supported, broadness of scope, generalizability of advice, and formalism. We use this initial evaluation to identify groupings of documents and get an overview what the literature search has found to be available. The following describes each evaluation in more detail.
Phases
indicates simply what combination of evidence collection, analysis, and reporting the document covers.
Directness
has two possible values: direct and constraints. Direct commentary on incident response explicitly talks about what an investigator should or should not do. Constraints provide only requirements for outcomes or outputs, and do not indicate how these properties should be achieved. Constraint-based advice is common when situating investigation within the larger context of incident response, and situating response within incident management.
Type
indicates what type and level of detail the document provides to decision making. Possible values are case study, ontology, advice, and instructions. At one end of the spectrum are case studies. Case studies report the facts of an individual case of investigation, without attempting to abstract up to general lessons. A categorization forms categories of useful actions (implicitly or explicitly from case studies), but gives no advice on how to apply these categories. Advice provides some ordering on what category of action should be taken, given certain conditions. Finally, instructions provide explicit decision-making instructions on how to evaluate options. Type also provides some rough guide to how much effort it will take to apply the document to practice, with case studies being the most difficult.
Goals
indicates what sort of investigation the advice targets. We distinguish three goals an investigator could have: fix the system, gather intelligence on adversaries, and make arrests. Certainly, there may be other goals, but these cover a wide degree of practical differences. Investigators need quite different information between these goals. For example, to fix a system, one needs to know everything that has been accessed by the adversary, but you need to know rather little about them. Whereas to make arrests, one cares very much about the adversary, but also is bound by several practical matters of what counts as admissible legal evidence of attribution and loss. When gathering intelligence on what an adversary may do next, these legal considerations fall away, but one also focuses on quite different aspects than fixing a system. For example, to gather adequate intelligence one need not enumerate all compromised systems.
Scope
reports how widely the document applies, as reported by the document. Options are narrow, medium, and broad. A narrowly-scoped document is intended to apply to only a small, non-representative group of people, and/or for a short period of time. Broad scopes are intended for most people within information security. Medium scope fits somewhere in between. Examples of medium scope are US-based law enforcement forensics specialists, or the operators of tier-three (that is, backbone) networks.
Generalizability
of advice indicates how likely the document can be relevant to contexts outside those for which it was specifically designed. Generalizability is explicitly level-set from the document’s scope. Thus, a document with broad scope but no generalizability may still be applicable to more people than a narrowly-scoped document that is generalizable. Whereas scope is a measure taken directly from the document being evaluated, generalizability is an evaluation of potential not explicit in the document. Indicators of generalizability include use of models or methods from other disciplines with well-established other uses or evidence from sources other than the document itself that the advice from the document applies more widely. Options for this criterion are coarsely set as unlikely , likely (in between unlikely and highly likely), and highly likely ; values represent essentially the evaluator’s prior belief on the document being applicable outside its stated scope. We have a final value, widely, indicating the document is certainly generalizable beyond its scope and is likely to be generalizable to a much broader scope.
Formalism
reports the degree of mathematical formalism present in the advice provided. Options are none, qualitative, formal, and perfunctory. Perfunctory indicates formalization is present, but essentially unused to advance the arguments or positions of the document. This rating does not mean the formalism is wrong; however, it does indicate it would take significant effort on the part of the reader to make use of the formalism beyond what qualitative models would provide. None only applies to narratives that make no attempts at abstraction. Both qualitative models and formal mathematical models have value in their own ways, and one should not be considered preferred over the other per se.
After this classification of the documents is complete, we undertake a more free form synthesis of the results. Our focus is on how investigators make decisions about evaluating the quality and importance of evidence, generalize from particular evidence to evidence of trends or patterns of behavior, and select what to report based on security constraints as well as what others will find most convincing. Although these align loosely with the three phases of investigation we have highlighted, we are not making a one-to-one connection between the three phases and our three focal points. For example, if an investigator knows some kind of evidence is particularly convincing to report, that should impact what they look for during the evidence collection phase. Section 6 reports the results of this classification, examination of focal points, and identification of preliminary gaps.
4.4 Limitations
While these methods have much to recommend them, there are of course limitations. Some of these are practical, such as the restriction to publications available in English. Some limitations are a function of restricting the scope to standards. Perhaps the most dangerous limitation is a result of the subject matter – security practitioners tend to be secretive about their methodologies.
The restriction to English will naturally limit the results. For example, an internet search for (information security incident response) returns a couple dozen results on Google as well as Google Scholar. This seems to be the preferred term in mainland China, searching for computer security incident response returns only a couple of Taiwanese sites.
The importance of this language choice on actually limiting documents available to us is less clear. EU and UN documents would be available in English as a matter of policy. The US government, which publishes in English dominates in this space, as do US companies. Countries that are not allied to the US and have developed computer security capabilities are relatively few; basically just the Russian Federation and the People’s Republic of China (PRC). While it is possible these countries have published comprehensive decision making procedures for incident response, it seems unlikely. It also seems likely that, given how much attention the US security establishment pays to Russia and the PRC, if such a thing were published it would be found and reported on, if not translated. For these reasons, we judge the impact of limiting our search to English documents is a low risk.
Focusing on standards, and what they cite, limits the scope but also creates other limitations. Indeed, reducing the scope to something manageable is one goal of focusing on standards, and this seems to function as intended. However, the type of information published in standards is different than that in academic journals and conferences, and this imposes some limits on the work. Specifically, standards are on a slower publication cycle than academic work. This delay would be a problem if our topic were covered much in the academic literature. However, as indicated by Section 3, academic publications do not appear to cover decision making during computer security incident response.
Creation of technology standards is itself a complex process, and as Section 2.2 touches on, the process has a complex history in its own right [138]. The way standards are made imposes its own limitations on our findings. Standards are rarely made purely for the dissemination of information; rather, they usually solidify a dominant business position. Security standards appear to be an outlier from this norm, as they have unique concerns about correctness and non-subversion [87]. Incident response standards are essentially unstudied within the academic standards literature; [87] mostly focus on cryptographic standards. This situation means we accept a risk in that we do not know what biases may be embedded in the creation of incident response standards. The standards literature provides evidence there will be a bias, but has not studied incident response standards in order to provide evidence for what that bias might be. We would like to know whose interests are best served by the creation of incident response standards, for example.
A closely related limitation of concern involves secrecy. Many incident response organizations may not wish to disclose their processes and procedures in detail, lest the adversary learn how to subvert or avoid them. Other areas of information security experience similar publication restrictions. Incident responders likely have a legitimate concern in this regard, and may also have a legal or regulatory requirement to keep certain information or processes private. Therefore, this limitation imposes a significant risk that relevant information is not public. Lack of access obviously limits the review. Our approach to reduce the impact of this limitation is to understand that we ought to read between the lines of available documents when we can justify expanding our interpretation of a document’s contents with circumstantial evidence from the context surrounding its publication. However, we must accept that there is an amount of information about our topic which simply is not public and we cannot hope to access for a public literature review. One could perhaps use news articles and audit reports to attempt to evidence the extent to which organizations in fact implement the available standards; we leave such investigations for future work. We first need a baseline understanding of the standards literature from which to begin.
5 Results
We present our results per search venue is Sections 5.1 through 5.4. Section 5.5 takes the results from these four venues, extracts the referenced documents from the results, and evaluates these cited documents.
In summary, Table 2 lists the documents that are relevant to our review scope and purpose, and will analyze them in depth. We have selected these 29 documents from roughly 350 possible documents returned from search results and references. Many tables documenting our the decisions that lead to these results results can be found in the Appendix.
5.1 IETF
Table 3 evaluates the results of our search procedure. We pass four documents on, RFCs 6545, 7203, 7970, and 8134.
The IETF documents break down into two clear broad categories, Best Current Practice (BCP) 21 on expectations for computer security incident response [19], and all the others, which are to do with Incident Object Description Exchange Format (IODEF), its expansion, and usage.
As an expectations document, BCP 21 focuses primarily on the services and support a CSIRT should provide to its constituency, who that constituency should include, and so on. These considerations are vital to CSIRT operations; however they are not directly relevant to our question at hand.
The IODEF projects in particular are feature CERT/CC staff heavily, Danyliw and Inacio during all their RFC authorship, and Trammell contributed heavily to SiLK (System for Internet-level Knowledge) while at CERT/CC before moving on. Because CERT/CC is also heavily involved in FIRST, it is unsurprising to see a sort of division of labor between the IETF documents and the FIRST documents. In particular, the IODEF format focuses almost exclusively on technical issues of data exchange and reporting format. The softer considerations, of how to collect, evaluate, and analyze the data contained within IODEF remain in the purview of FIRST.
Thus, as technical reporting formats are in our scope of reporting results, all RFCs related to IODEF are relevant. There do not appear to be any other IETF documents within our scope.
These IODEF documents may at first seem to be out of scope, as we specified that our scope is how investigators make decisions, not what tools or formats they use to document them. This topic recurs in Section 5.5.1. IODEF is essentially a language for talking about computer security incidents. However, because we are not interested in data formats, we are not interested in the language per se. IODEF is in scope because as a constructed language it makes judgments about what aspects of incidents are important, necessary, or possible to communicate. These judgments, at least implicitly, bear on what an investigator should choose to report. We therefore judge IODEF as in-scope. However, we stress that our intended scope is how to decide what information to report, not what language in which to report it. Therefore, data formats and languages for anything else remain out of scope.
5.2 ISO
Nine search terms return 31 total results, with 23 unique results displayed in Table 4. ISO 27035-1, 27041, and 27043 meet our criteria to carry through to the citation-harvesting and synthesis stage:
- •
Information security incident management, Part 1: Principles of incident management [73]
- •
Guidance on assuring suitability and adequacy of incident investigative method [70]
- •
Incident investigation principles and processes [72]
5.3 FIRST
FIRST is the smallest body surveyed, and it is not primarily a standards organization but rather a forum for organizations with a shared purpose – incident response.
On it’s “standards” web page, FIRST lists four standards it maintains:
- •
Traffic Light Protocol (TLP) on agreed-upon levels for marking information sensitivity
- •
Common Vulnerability Scoring System (CVSS) on describing the characteristics and severity of defects in software systems (not to be confused with Common Weakness Scoring System (CWSS) by the Mitre Corporation (MITRE))
- •
Information Exchange Policy (IEP) is a reporting format; in this regard it is another language for reporting, similar to IODEF or those listed in Table 7. IEP’s focus is on disseminating information responsibly and quickly during incident response.
- •
passive () (pDNS) is a formatting standard for DNS traffic analysis; the FIRST group is working on an IETF standard.
None of these standards meet our relevance criteria, because none are about investigator practice. They are all things a competent investigator should know how to interact with and interpret, but they do not help us understand what decisions an investigator should make in a given scenario. FIRST also notes it contributes to several ISO standards, which Section 5.2 covers (namely, 27010, 27032, 27035, 27037, and 29147).
More instructive than these standards are FIRST’s “Security Reference Index” that are “helpful” to the FIRST community.161616https://first.org/resources/guides/reference FIRST’s members are many, if not most, of the professionals and practitioners that we hope to come to understand. The documents Table 5 evaluates are listed as either best practices or standards in this reference index.171717Strictly speaking, [109] and [1] are not linked directly; they are the most relevant part of a suite of publications linked to by FIRST as https://www.cert.org/incident-management/publications/index.cfm. Five documents emerge as relevant to our review: [1, 34, 53, 56, 109].
The security reference index also links to the home pages of other security organizations; however, we cannot review the contents of everything these organizations have produced in full. In large part, the information is more about solving specific technical problems than our target for a general problem solving method. Such specific problems make for instructive cases when thinking about generalized methods, and so these organizations do provide an integral function to our topic. But they do not aim for the types of documents in scope of our review. The organizations identified are:
- •
Center for Applied Internet Data Analysis (CAIDA), www.caida.org
- •
CERT/CC (formerly Computer Emergency Response Team), www.cert.org
- •
Center for Internet Security (CIS) Benchmarking, http://www.cisecurity.org/
- •
Team Cymru, a security think tank, https://www.team-cymru.org/services.html
- •
Agency for Network and Information Security (ENISA), https://www.enisa.europa.eu/, including CSIRT services https://www.enisa.europa.eu/topics/csirt-cert-services
- •
Open Web Application Security Project (OWASP), https://www.owasp.org/index.php/OWASP˙Guide˙Project
- •
Microsoft Security Guidance Center https://technet.microsoft.com/en-us/library/cc184906.aspx
- •
Sysadmin, Audit, Network, and Security Institute (SANS Institute) reading room, https://www.sans.org/reading-room/
Although [43] targets management rather than practitioners, it provides links to training for practitioners. The two organizations the report lists are CERT/CC and the EU-funded TRANSITS.
5.4 Intelligence community
The canonical training course for CIA and other intelligence analysts is [65]. The book is essentially applied psychology. It covers topics such as analyzing competing hypotheses, which includes evaluating whether evidence has been planted to deceive, as well as overcoming human cognitive biases such as anchoring, vividness, and oversensitivity to consistency. Such methods, especially for evaluating evidence in the face of deception, have clear relevance to incident investigation.
The model of a computer attack as following a predictable ‘kill chain’ of steps from start to finish was published by Lockheed Martin incident responders [68]. The seven steps are reconnaissance, weaponization, delivery, exploitation, installation, command-and-control, and actions on objectives. These are the steps in one single attack – a single phishing email, a single drive-by download with a malicious advert, etc. Adversaries almost always compose a campaign out of multiple attacks; the objectives of one attack may be to obtain a platform from which further attacks are possible. The purpose of this model “is to capture something useful about the pattern all, or at least nearly all, attacks follow,” so the analyst can anticipate what to look for or expect next [134, p. 10].
[23] builds on attack ontologies, specifically the kill chain, and intelligence analysis to perform attribution in computer security incidents and analysis of whole campaigns. The method incorporates Bayesian statistics to model belief updates of the analyst. These statistical details are explicitly intended to help overcome analyst cognitive biases, such as those discussed in [65].
5.5 Referenced Documents
We harvest citations from the standards identified as relevant in the prior parts of Section 5. We summarize the results here; the evaluations are detailed in a subsection for each publication venue. The documents harvested from citations that are directly relevant to our review are:
- •
[27]
- •
[28, ch. 2 ]
- •
[33]
- •
[91]
- •
27037 [69]
- •
27042 [71]
- •
NIST SP 800-83 rev 1, §4 only [132]* *
- •
- •
[113]
- •
[86]
- •
[30]
- •
[140]
- •
[104]
- •
[81, ch. 5 only ]
5.5.1 IETF documents
The four relevant IETF standards reference 97 unique documents, excluding the IODEF-related standards already considered in Section 5.1. These documents fall into three broad categories: technical implementation requirements and dependencies; other related computer-security-incident report formats; and broader incident handling guidelines that describe the larger analysis and response context within which the reporting formats are used. This first category of implementation dependencies is not relevant to our project. Therefore, we focus on other reporting documents and broader incident handling guidelines. There are 22 cited reporting-related documents and exactly one related to broader incident handling and use of the reporting formats.
Table 7 lists the reporting formats and data sources cited by the IETF results in Section 5.1. Other report formats are primarily produced by NIST and MITRE – with funding from the US government including NIST. These projects also include the only referenced documents that are continuously updated data archives. The data formats for Common Vulnerabilities and Exposures (CVE), Common Weakness Enumeration (CWE), CWSS, Common Configuration Enumeration (CCE), Common Configuration Scoring System (CCSS), Common Platform Enumeration (CPE) are also continuously populated and published by NIST and MITRE as new vulnerabilities, platforms, etc. are discovered or developed. Thus these projects provide not only a format, but a standard reference dictionary of the possible terms with which the format may be populated. CVSS is perhaps the most important of these metrics which provide data and scoring; for a survey that relates CVSS to other security metrics, see [118].
These standard dictionaries are referenced by many of the other data formats which inherit the field essentially as a data type. For example, Malware Attribute Enumeration and Characterization (MAEC) may indicate which vulnerability a malware targets using its CVE number. Such dictionaries are useful background knowledge during incident response, and help reduce confusion by providing common reference tags. However, following the pattern of other documents surveyed, these reference dictionaries do not provide agreed-upon evidence collection, field population, or analysis guidelines for their contents.
The next largest group of cited work from identified RFCs are three more IETF documents related to IODEF that did not appear in the original search. Other documents are also related to IODEF. CVRF and XEP-0268 extend and implement IODEF, respectively. AirCERT is a proposed implementation architecture that uses IODEF in automated indicator exchange. Our conclusions about the IODEF results identified in Section 5.1 therefore apply equally to these other documents, and we do not need to pay them special attention.
The remaining documents fall loosely into the NIST-MITRE orbit. ISO 19770 for asset management is developed separately from, but is related to, CPE and CCE, NIST’s asset management for platforms and configurations, respectively. MMDEF is not directly related to MAEC; however, MAEC has adopted a significant component of the MMDEF schema.
The only citation related to actual decision making during an incident is NIST SP 800-61 [34]. This document is already included from the FIRST results, see Section 5.3. Thus, from the IETF citations, we add no new documents to the review.
5.5.2 ISO documents
We extract the references from the three relevant ISO standards, namely ISO/IEC 27035, ISO/IEC 27041:2015, ISO/IEC 27043:2015. ISO/IEC 27035 comes in two parts; although only the first part is in scope, we extract references from both parts. Although ISO charges for access to its documents, all the bibliographies are freely available, so we include all documents in this step.
There are 81 total references among the documents, with 64 unique references. Of these, 20 are elements of the ISO/IEC 27000 series of standards explicitly targeting information security. Four are the documents already referenced, and several others are already noted as not relevant in Table 4. However, from the references we add the following two 27000-series publications to our survey documents as relevant to our survey (27037 and 27042):
- •
Guidelines for identification, collection, acquisition and preservation of digital evidence [69]
- •
Guidelines for the analysis and interpretation of digital evidence [71]
Of the remaining 44 referenced documents, 13 are further ISO standards. Specifically:
- ISO 15489-1
- ISO 8601
- ISO 9000
- ISO/IEC 10118-2
- ISO/IEC 12207:2008
- ISO/IEC 17024:2012
- ISO/IEC 17025:2005
- ISO/IEC 17043:2010
- ISO/IEC 20000
- ISO/IEC 29147
- ISO/IEC 30111
- ISO/IEC 30121
- ISO/IEC/IEEE 29148:2011
All of these other ISO documents are out of scope. We can further remove the following as unavailable or already evaluated: ILAC-G19, which directly follows from ISO 17020 and 17025; RFC 5070 [40], see Section 5.1; one by Valjarevic and Venter that is not available but appears by title and timing to be a working group presentation discussing the other two papers by these authors. We also will not consider the Daubert 1993 US Supreme Court case, as we aim to be jurisdiction neutral. Removing these leaves the documents listed and evaluated for relevance in Table LABEL:tab:ISO-referenced. We pass the following documents on to the next stage of analysis: [27, 28, 33, 91]. And, like the IETF, ISO cites NIST SP 800-61 [34].
Also cited are two further MITRE data formats not covered in Section 5.5.1: STIX (Structured Threat Information eXpression) and TAXII (Trusted Automated eXchange of Indicator Information). These build on MAEC, CVE, and so on as formats for exchanging incident data. Like the other reporting formats already discussed, STIX and TAXII are not directly relevant to our incident response decision-making topic. They are a language in which to do reporting; reporting is in scope. However, we plan to discuss what to report in a language-independent way.
The ACPO (Association of Chief Police Officers) guidelines are representative of many of the documents in Table LABEL:tab:ISO-referenced. Their target audience is “UK law enforcement personnel who may deal with digital evidence” [154, p. 6]. It is primarily about the legal chain of custody necessary to bring digital evidence to court. This topic is about evidence collection, so it is in scope. But the target audience is law enforcement, not security practitioners. The guidelines are not transferable to our topic of interest. The extent of comment on the actual work of understanding what the digital evidence means is constrained to “it is not practically possible to examine every item of digital data and clear tasking is needed to ensure that the digital forensic practitioner has the best chance of finding any evidence which is relevant to the investigation” [154, p. 10].
Some relevant work will not be carried through because it is obsoleted in a rather round-about way. [148, p. 1 ] notes “an effort to
standardise the process has started within ISO, by the authors.” Thus we consider papers by these authors to be obsolete because the authors directly subsumed their ideas into the ISO process. The process classes and activities used by ISO are clearly derived from [149, p. 6 ], which also contains a matrix of how these reference terms relate to other common forensic investigation ontologies. This set of works cited181818Specifically, the overlapping works cited are [121, 11, 26, 13, 33, 37, 91, 154, 28] matches the ISO work remarkably closely, as would be expected since the primary authors are the same. Unfortunately, [149] gives absolutely no methodology for how they arrived at this list of resources. Their analysis method is also not discussed, so it is unclear how or why they arrived at their categories and classification.
These omissions are particularly strange in that [149, p. 3 ] quotes [35] as rightly concluding the next steps in reaching consensus on and improving the field of digital forensics are a review of the literature that can be used to accurately drive consensus. This task is clearly what’s been attempted, and as it has become an ISO standard it seems to have been accepted by a variety of practitioners. However, the lack of explanation of how these documents were selected as the correct set from which to drive consensus makes it hard to trace the authoritativeness of this source.
5.5.3 FIRST Documents
As Table 5 demonstrates, we pass five FIRST-related documents through the evaluation of results to harvest further citations. Three of these documents do not have any citations ready to harvest. RFC 2196 [53] does not have in-line citations, and only §5.4 is relevant, so the relevant citations to follow cannot be distinguished. Further, RFC 2196 is already 20 years old, and so following any citations would provide little modern benefit. On the other hand, [56] is a primary source – it is a survey of preventative practices at over 100 CSIRTs. [56] notes the tools that the respondents use, but it makes no citations to other incident response methodology documents. [1] is similarly a primary source, though on incident management from CERT/CC. The only reference we take from [1] is where it explicitly indicates further information on incident analysis is contained in another CERT document, namely [86]. Therefore, we harvest citations primarily from [34] and [109].
[34] references three classes of resources. First is a list of incident response organizations, secondly a list of NIST publications related to incident response, and finally a list of applicable data formats. The list of organizations includes many already discussed in Section 5.3. Those jointly listed by NIST and FIRST are CERT/CC, ENISA, and FIRST itself. NIST additionally lists the Anti-Phishing Working Group (APWG); Computer Crime and Intellectual Property Section (CCIPS); Government (GFIRST); High Technology Crime Investigation Association (HTCIA); InfraGuard; the Internet Storm Center (ISC); the National Council of Information Sharing and Analysis Centers; and Computer Emergency Readiness Team (US-CERT). These organizations are certainly involved in various aspects of incident response. However, organizations as such are out of the scope of our review.191919As a convenient sample, I have presented at APWG [136, 133] and attended InfraGuard meetings, and I do not expect there would be significant benefit in expanding the scope to include them. Likewise, I have interacted with several ISACs, and reviewed their available materials ( Research and Education Networking () (REN-ISAC) and Financial Services () (FS-ISAC) especially) and do not believe they have any documents of importance to our topic.
Table 8 lists and evaluates the relevance of the NIST publications referenced by [34]. All of these publications contribute to relevant background knowledge. For example, any incident response professional will need to know what a intrusion detection and prevention system is and how they are deployed (SP 800-94). But this topic is not about evidence collection, analysis, and reporting; it is merely necessary background knowledge. The two publications that are relevant are the guides to Malware Incident Prevention and Handling for Desktops and Laptops [132, §4 only] and Integrating Forensic Techniques into Incident Response [84].
The data exchange formats listed by [34] are quite similar to those NIST, IODEF, and MITRE formats extracted from the IETF documents in Table 7. The only difference is the addition of Asset Identification (AI), Asset Results Format (ARF), CVSS (from FIRST, see Section 5.3), and Cyber Observable eXpression (CybOX). As discussed in Section 5.5.1, these formats are languages for reporting results, but they do not directly discuss what to say. Our intention is to discuss what to say in results, while being language agnostic, which puts these various formats and languages just out of scope.
[109] cites 27 documents. They include several technical format documents for ontologies in the W3C Ontology Web Language (OWL), KL-ONE knowledge representation, knowledge graphs, process specification language (PSL), or the display tools used (Graphviz), which we will not discuss. There are also psychology and ontology that are obviously out of scope for our review [102, 9]. Further, four references we have already considered, namely ISO/IEC 27001, ISO/IEC 27002, [34], and [13]. These exceptions leave the eight documents evaluated in Table 9. The only documents that pass the evaluation are [113] and [86].
[96] on whether cybersecurity is a science is not within our current, relatively narrow scope. However, since incident response is an important subset of cybersecurity, whether security investigations are a kind of subcategory of scientific investigation clearly impacts our handling of what incident response is and how to link it to knowledge generation and evidence evaluation more generally. [137] address the relationship between cybersecurity and science and argue that cybersecurity as practiced is a kind of science.
[51] provides a difficult decision. It is one of the few attempts at formalization. However, its target is security knowledge, not security practice. This topic is closely allied to our hope to formalize incident response, as security knowledge would be instrumental to that project. So while not in scope for this review, this document may be useful for future related work.
5.5.4 IC Documents
[65] presents some challenges to adequate reference harvesting. The book contains no collected list of references, written in a traditional humanities style in which references are in footnotes intermixed with commentary, but this is not the central problem. As essentially a military intelligence and psychology book, its sources are quite wide-ranging. References range from World War II Nazi-propaganda analysis to behavioral economics. It is only through Heuer’s CIA experience that these disparate sources are converted into a useful guide on how to reason in adversarial situations. The other challenge is that [65] makes only passing reference to computers as tabulating machines. The closest he seems to get to computer science is via [131], as he discusses decision-making and satisficing. For these three reasons we consider [65] as essentially a primary source and do not trace citations from it. One should not be surprised it has many features of a primary source, as surely its main value is summarizing CIA analytic experience not otherwise publicly available.
[23] and [68] are more straightforward. There are 79 citations between the two, with no overlap, though [23] cites both [68] and [65]. The references in [23] are noticeably more strategy-focused over the tactically-focused [68], as one would expect from their different topics. [68] cites several vulnerability bulletins and company advisories as cases; it is more of a primary source, documenting the analysis methods used by Lockheed Martin incident response staff. We do not consider such advisories, software tool documentation, and news items, as they are not within our review topic. There are also several references already covered elsewhere: [34], STIX, CVE, and SANS. There is even yet a new reporting and data exchange format: Vocabulary for Event Recording and Incident Sharing (VERIS). We also exclude references that are merely the official definitions of terminology. These exceptions reduce the total referenced works to 50.
The kill chain [68] and the diamond model [23] are both attack ontologies. They model the possible routes an adversary may take when executing an attack. One big class of cited work is other attack ontologies. We consider the kill chain and diamond model as de facto standard ontologies, but we believe they reached that level of agreement within the IC because they also come with investigative norms for interpreting and filling in the ontologies. There are many other attack ontology works that do not come with such guidance, and so we will not consider them directly in scope for our review. We remove 16 references from consideration because they are attack-ontologies without guidance.
There are also several references that are clearly for background or motivation, such as [62], [95], Symantec’s analysis of the Duqu malware, and assessments of Chinese attack capabilities. We also remove how-to descriptions for conducting particular methods of technical analysis, namely on passive DNS analysis [7], crime-pattern analysis [115], and honeypots via the Honeynet Project. This leaves us with 22 documents in Table 10. Of these, four pass our relevance requirements: [30],202020Technically the citation is to this article’s republication in a popular textbook, [31, ch. 16 ]. However, as the rest of the textbook is not directly referenced, we reference just the original publication. [140], [104], and [81, ch. 5 only ].
6 Discussion
Table 6 classifies advice on these topics in several ways: to which phases the document applies, the directness with which the document applies to each phase, the applicability of the advice, investigative goals supported, broadness of scope, generalizability of advice, and formalism. We shall make some commentary on all the documents in the Table, in order of appearance, with one exception. Before we lose ourself amongst the trees of this discussion, we will make our general claim of the primary gap in the literature clear up front. Our evocative evidence for this claim will be the three NIST publications, as they make the point most clearly. After discussing these, we go on to discuss the ISO, IETF, etc. documents in the order they appear in Table 6.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Chris Alberts, Audrey Dorofee, Georgia Killcrece, Robin Ruefle and Mark Zajicek “Defining Incident Management Processes for CSIR Ts: A Work in Progress”, 2004
- 2[2] Christopher Alberts, Audrey Dorofee, Robin Ruefle and Mark Zajicek “An Introduction to the Mission Risk Diagnostic for Incident Management Capabilities (MRD-IMC)”, 2014
- 3[3] Tansu Alpcan and Tamer Başar “Network security: A decision and game-theoretic approach” Cambridge, UK: Cambridge University Press, 2011
- 4[4] Bernhard Amann, Robin Sommer, Aashish Sharma and Seth Hall “A lone wolf no more: supporting network intrusion detection with real-time intelligence” In Research in Attacks, Intrusions, and Defenses Springer, 2012, pp. 314–333
- 5[5] Philip Anderson “Electronic evidence – A basic guide for first responders”, 2015 URL: https://www.enisa.europa.eu/publications/electronic-evidence-a-basic-guide-for-first-responders/at˙download/full Report
- 6[6] Ross Anderson, Chris Barton, Rainer Böhme, Richard Clayton, Michel JG Van Eeten, Michael Levi, Tyler Moore and Stefan Savage “Measuring the cost of cybercrime” In Workshop on the Economics of Information Security , http://weis 2012.econinfosec.org/papers/Anderson˙WEIS 2012.pdf , 2012
- 7[7] M. Antonakakis, R. Perdisci, W. Lee, N. Vasiloglou II and D. Dagon “Detecting Malware Domains at the Upper DNS Hierarchy” In 20th Usenix Security Symposium , 2011
- 8[8] Sasikanth Avancha, Amit Baxi and David Kotz “Privacy in Mobile Technology for Personal Healthcare” In ACM Comput. Surv. 45.1 New York, NY, USA: ACM, 2012, pp. 3:1–3:54