Identification of Bugs and Vulnerabilities in TLS Implementation for Windows Operating System Using State Machine Learning

TL;DR

This paper uses protocol state fuzzing to identify vulnerabilities in the TLS implementation of Windows' SChannel, revealing flaws that could lead to serious security breaches.

Contribution

It introduces a protocol state fuzzing approach to analyze and uncover design flaws in the TLS implementation of Windows SChannel across multiple versions.

Findings

Discovered various flaws in SChannel's state machine

Identified potential attack vectors exploiting these flaws

Highlighted the importance of rigorous protocol implementation testing

Abstract

TLS protocol is an essential part of secure Internet communication. In past, many attacks have been identified on the protocol. Most of these attacks are due to flaws in protocol implementation. The flaws are due to improper design and implementation of program logic by programmers. One of the widely used implementation of TLS is SChannel which is used in Windows operating system since its inception. We have used protocol state fuzzing to identify vulnerable and undesired state transitions in the state machine of the protocol for various versions of SChannel. The client as well as server components have been analyzed thoroughly using this technique and various flaws have been discovered in the implementation. Exploitation of these flaws under specific circumstances may lead to serious attacks which could disrupt secure communication. In this paper, we analyze state machine models of TLS protocol implementation of SChannel library and describe weaknesses and design flaws in these models, found using protocol state fuzzing.

Tables

Table 1. TABLE I: Sink States and Self Loops in State Machine Models

Figure No.	State Machine Model	Sink States	States with Self Loops
3	Windows 7 RSA TLS 1.0 (Client)	2,4	1,4,5,6,7,9
4	Windows 8 RSA TLS 1.0 (Client)	2	0,1,2,3,4,5,6
5	Windows 8 and 10 RSA TLS 1.2 (Client)	2,3	1,3,4,5,6,7,8
6	Windows Server 2008 RSA TLS 1.0 (Server)	2,3	1,3,4,5,6
7	Windows Server 2012 (Server)	2	1,4,5,6,7
8	Windows Server 2016 (Server)	2	1,3,4,5

Full text

I Introduction

Transport Layer Security is the protocol responsible for secure communication over Internet. HTTPS, SFTP, SMTP and many other application layer protocols use TLS for secure communication. The protocol uses various cryptographic schemes like Asymmetric Key Encryption, Symmetric Key Encryption and Hashing to ensure confidentiality, authenticity and integrity of data. Vast use of TLS makes it a good target for security researchers and attackers. In the past, many attacks have been developed by the attackers which raised questions on security provided by the protocol, but with time the protocol has improved a lot. Most of these attacks target implementations of the protocol rather than the protocol itself. There are many implementations of TLS which are implemented by various programmers with their own understanding of the protocol. Many times individual understanding of the protocol specification do not cover all possible combinations of inputs and outputs, which leaves the protocol implementation vulnerable to attacks. To identify such vulnerabilities security researchers use many techniques. One of such technique is fuzzing which is widely use to find vulnerabilities in software implementations. In this paper we have used a technique called “protocol state fuzzing” which is used to identify undesired protocol state transitions in a specific protocol implementation. This technique is very useful in finding invalid inputs to a state, which may lead to a valid state with invalid transitions.

The paper is organized into 8 sections. Section II describes the related work in this domain. Section III gives an overview of TLS protocol which explains handshake mechanism of the protocol. Section IV discusses about learning procedure, SChannel implementation and experimental setup. Section V explains design of state machine models, types of bugs and vulnerabilities and attack scenario. Section VI presents learned models for various operating system and discusses analysis of these models. Section VII explain implications of flaws found in the learned models and the paper ends with concluding remarks in section VIII.

II Related Work

There have been many analysis of TLS protocol which revealed various vulnerabilities in the protocol. Most of such analysis are focused on use of weak parameters or vulnerability in software implementation. DROWN[1], FREAK[2], LOGJAM[3], SLOTH[4] are example of attacks which exploited weak parameters in the protocol while HEARTBLEED[5] is an example of vulnerability in software implementation.

Another dimension of TLS analysis is verification methods. There are mentions of verification methods which are used to verify correctness of protocol implementation with respect to protocol specification, but most of such literature provides abstract description rather than a practical implementation. Generally, verification methodology has an advantage that it not only verifies the correct path, but also finds the incorrect one. Such incorrect paths can be analyzed further to identify scope of attacks on protocol implementations. Such type of analysis is called ”Protocol State Fuzzing”, which provides all possible kinds of inputs to each state and traces the corresponding outputs. There are methods which apply such techniques to find bugs and vulnerability in the implementations. One such tool[6] has been designed using machine learning and tested against various open-source implementations of TLS. In this paper we have extended this work further and analyzed Windows TLS library called SChannel which is a closed source implementation.

III Overview of TLS Protocol

TLS is successor of SSL which was developed in 1994. SSL 2.0[7] was the first public version of the protocol, which was deprecated very soon and improved to SSL 3.0[8] which was the first stable version and supported many legacy systems. Later TLS 1.0[9], 1.1][10] and 1.2[11] succeeded version SSL 3.0 and formally defined by IETF.

TLS Protocol consists of 2 layers, handshake and record layer. Handshake is responsible for negotiation of various algorithm and parameters while record layer pack and unpack every message. The messages which are used by handshake and record layer are specified and defined in respective RFCs by IETF. Fig.1 describes various messages exchanged during a handshake.

As shown in Fig. 1 messages are exchanged between a client and server to start a secure and encrypted session. In this process some of the messages are optional which are exchanged, only if systems are configured to use specific parameters. The following description briefly explains role and control flow of these messages while establishing a secure session using TLS protocol.

ClientHello: The session is always initiated by the client with a ClientHello Message. It is the first message of the session and contains basic information for negotiation. This message specifies maximum version of the protocol supported, list of supported cipher suites (key exchange algorithm, encryption algorithm and hash function) and a random number which is used for key generation. Depending on the configuration, this message could have extensions too. 2. 2.

ServerHello: It is a reply message from the server in response to ClientHello Message. It is the first message from server which specifies TLS version and cipher suite to be used for secure session. This message contains a random number similar to ClientHello message and it is used in key generation algorithm. 3. 3.

Certificate: This message is sent from the server with a certificate owned by it. Client uses this certificate to authenticate the communicating server. RSA public key which is embedded in certificate is also used either in key exchange mechanism or for integrity check. 4. 4.

ServerKeyExchange[Optional]: This message is mostly used when Diffie-Hellman key exchange is used. This contains various parameter needed from server side to generate a common secret for both client and server. 5. 5.

CertificateRequest[Optional]: This message is uncommon and only used when server is configured to establish connection only with authenticated client. This message asks client to send a valid certificate to authenticate itself. 6. 6.

ServerHelloDone: This is simple message which inform the client that as per protocol specifications all required information has been sent from the server . 7. 7.

ClientKeyExchange: This message is sent from client to server with all necessary information to generate encryption keys. In case of RSA key exchange, a secret encrypted with RSA public key is sent to the server. 8. 8.

Certificate[Optional]: This message contain a certificate owned by client and sent to the server to authenticate the client.This message is a reply to CertificateRequest message sent by the server. 9. 9.

CertificateVerify[Optional]:This message is sent to provide explicit verification of the certificate by signing the handshake messages sent and received by the client. This message is only sent for the client certificate that has signing capability. 10. 10.

ChangeCipherSpec: This message is one byte message and it is used to indicate the receiving side that messages following this message are encrypted with encryption key which is generated using information shared previously. 11. 11.

*Finished: *This is first encrypted message sent from client to server. It is used to inform that handshake is finished and it also checks integrity of handshake messages to prevent session hijacking from MiTM(Man-In-the-Middle). This message contains hash of all handshake messages sent from sender of this message. The receiving side match this hash with the hash of handshake messages received from sender. 12. 12.

ApplicationData: This message contains actual data of user, encrypted with encryption key from sender which is decrypted with same key on receiving side. Encryption and decryption keys are generated on both sides using information shared in handshake messages.

IV Learning of TLS Protocol State Machine

State Machine of a system indicates the system behavior for every kind of inputs to the system. Although it is expected that system must be designed as per specifications, but this does not always happen. While developing a system all possible cases are not considered, which are later analyzed and then exploited by attackers.

TLS protocol is a system of messages which are used to establish a secure connection between two users in a network. There are many kinds of messages which are exchanged during the process of TLS handshake. Upon receiving and processing a message the TLS system(client or server) outputs a message (including empty response) and changes its state to receive next input. State-Machine of a TLS system is behavioral representation of responses of the system for various kinds of inputs. Each state of state-machine is designated for specific kind of task which is established by receiving and responding particular kinds of input and output pairs. On completing the designated task, the state transfers the control to the next state, depending upon the current input and output pairs.

Protocol State Fuzzing uses the technique of fuzzing in which all kinds of inputs are given to each state and output responses are analyzed to form state-machine diagram of protocol. These responses are analyzed automatically using machine learning technique[6][12]. SChannel[13] is a library which implements SSL/TLS protocol in Windows Operating System and it is stored in the system as schannel.dll. This library provides all the necessary functionalities for connection establishment, encryption and decryption of messages. This is not a open-source library so it works as black box without providing details of implementation. Most of the Microsoft services and third-party software for Windows use this library for secure Internet connection, therefore analysis of this library becomes more critical as it may affect a large number of systems around the world. Earlier development of learning system analyzed mainly open source libraries of TLS protocol and results were discussed[6]. In this paper, we have improved the earlier development by implementing ECDHE key exchange mechanism in addition of RSA and DHE. We have also made changes to the system to run for all kinds of Windows OS. Other than these changes TLS clients have been developed for each version of Windows Operating System which were used to obtain client state machines of TLS implementations. Virtual machines have been used to learn the models and the model have been verified using FlexTLS library which provides direct access to individual message of the protocol. Design of such state machine models, and the types of bugs and vulnerabilities found using this approach are discussed in next section.

V State Machine Models

In this section we will discuss about design of learned models of TLS library of Windows operating system. These models represents a visual representation of how state transitions happen and where it may go wrong. In next subsection, we will discuss design of these models which will provide better understanding the approach used in this paper.

V-A Design of State Machine Models

Models described in this paper are based on a typical state machine diagram where each state is labeled with number and arrows are used to specify the transition from one state to another. Every model (e.g. fig. 2) start with a green colored state labeled 0 and ends with red colored state which is labeled 2. In the context of TLS protocol, end(or sink) state is the state which closes the connection or absorbs all the inputs. Solid green arrows define a path of valid handshake while dotted red arrows indicate invalid or undesired transitions. Black arrows indicate transitions which terminate the ongoing connection.

In figure 2 green path ($0\rightarrow 1\rightarrow 3\rightarrow 4\rightarrow 5\rightarrow 6$) defines the valid handshake and for all the other messages connection is closed and control is passed to the end state, which is state 2. Figure 2 is the expected and ideal model of every TLS protocol implementation. In the next section, we will discuss practical behavior the state machines models which deviate from the ideal model. Due to deviation from expected model, bugs and vulnerabilities may arise in the system. Next subsection describes these bugs and vulnerabilities and discusses how these can effect the security of system and protocol.

V-B Types of Bugs and Vulnerabilities

Undesired transitions in a state machine are indication of bugs which may lead to critical vulnerabilities in the system. These vulnerabilities can be further exploited by the attacker to compromise security and privacy of the communication[14][15]. In context of TLS protocol, following types of bugs can exist in a state machine model, which are described in next section.

Additional end(or sink) state: End/sink state is the state which absorbs every input and doesn’t change the state. In TLS state machine model, there must be only one such state which corresponds to closing of ongoing connection. Additional sink state hangs the system because it just processes the given input, but doesn’t pass the control to any state. Such type of bugs may lead to denial-of-service vulnerabilities, where the sink state exhaust the CPU resources of the system to process the repeated crafted inputs fed by the attacker. 2. 2.

Self-Loops: Self-loops are the transitions in a state machine model which on a particular input remain in the same state. Self-loops are not expected in a state machine model unless it is specified explicitly. Similar to the sink state, existence of self-loops may lead to Denial of Service attacks where self loop exhausts CPU resources of the system by processing numerous crafted inputs provided by the attacker. 3. 3.

Alternate Paths: Alternate paths in a state machine model are the paths from a valid state to another valid state using invalid states or transitions. Existence of alternate paths may lead to critical vulnerability, where it is possible to reach to an important state, bypassing necessary transitions from previous states and thus violating protocol specifications. Such vulnerability may become even more critical, if it bypasses the key exchange or authentication mechanism of TLS protocol. CCS injection[16] is one of the such vulnerability which exploits an alternate path vulnerability and start encryption of messages without mandatory key exchange mechanism. 4. 4.

Undesired Replies: Undesired replies are indication of poor implementation where states are not checked against every possible input messages. In such cases, connection is closed for particular input but before closing the connection an unnecessary or undesired reply is sent. Although such replies do not affect the security of system, but they might introduce abnormal behavior in the system which violates the TLS protocol specifications.

V-C Attack Scenario

In this paper we have mentioned about attacks which can cause various kinds of damages to the system and the ongoing communication. By mentioning attacks we mean two types of attack scenarios. First type of attack scenario is Denial-of-Service where attacker acts as a client (or server) and establish many connections to its counterpart server (or client). Due to existing vulnerabilities these connections are accepted and processed further, which could result in heavy consumption of system resources and system may deny services provided by it.

Another type of attack scenario is Man-In-The-Middle(MiTM) where attacker comes in between of client and server and impersonate it as an other end of communication (client or server). Using this technique attacker can modify order and content of protocol messages and due to vulnerabilities described in next section, such crafted messages are accepted by the system which could weaken the security of communication.

VI Analysis of SChannel Based State Machine Models

We have designed and analyzed various state machine models for different combinations of Windows and TLS versions. Most of these models have implementation bugs except server model for Windows Server 2016. From the state machine models, it is clear that with newer versions implementation has been improved and number of undesired states has been reduced. Simple structure and less no. of states indicates a better and secure version of state machine model. In this section, we analyze state machine models of Windows SChannel Library for various version and discuss about bugs and vulnerabilities found in the system.

As we have discussed in last section that more than 1 sink state and self loops are the bugs which may lead to denial of service kind of vulnerabilities. Table-1 describes sink states and self-loops for each state machine model. Figure no. 3, 5 and 6 have extra sink states which may lead to denial of service attack, where these state absorbs every input without passing the control flow. Every model has some states with self loop which allows to absorb the inputs without changing the state which may again lead to critical denial of service attacks, if the connection does not close after a timeout period. Following subsections will discuss each model in detail.

VI-A *Windows 7 TLS 1.0 RSA

 [Client Implementation] (Figure 3)*

VI-A1 Valid Transitions (Handshake)

A valid handshake exists in the figure 3 which is shown using green arrows (path $0\rightarrow 1\rightarrow 5\rightarrow 6\rightarrow 7\rightarrow 8\rightarrow 9$). This path validates the transitions mentioned in discussed figure 2 which specifies expected behavior of protocol for a TLS client implementation.

VI-A2 Invalid/undesired Transitions

• •

In figure 3 there is an extra sink state 4, which is created by supplying Alert messages to the start state 0. This sink state absorbs all the inputs and disrupts the connection by owning the control flow forever. Other than additional sink state, states 1, 5, 6, 7, 9 have a self-loop for two inputs called ApplicationData and AlertWarning. These self-loops signify that for these mentioned two inputs, the state doesn’t change and it absorbs the inputs without any output. In this case too, states own the control for a particular message instead of closing the connection.

• •

Alternate path to reach state 1: In figure 3 transitions through state 3 create an alternate path to reach state 1. State 0 receives ApplicationData message and makes a transition to state 3 which on receiving ServerHelloRSA message changes the state to state 1. Existence of state 3 shows misunderstanding of programmer while implementing the protocol. Any input message for state 0, except ServerHelloRSA, must close the connection such that it should not lead to an alternate path to some valid state. Existence of alternate path may skip states of valid handshake which may lead to a handshake with invalid parameters. CCS Injection is one of such vulnerability which exploits existence of alternate path to a valid state.

• •

Undesired replies to messages in state 8 and 9: For three kinds of input messages types ServerHelloRSA, ServerCertificate and Finished abnormal behavior is detected in state 8 and 9 where instead of closing the connection ClientHello is sent and state is changed to 2. Although this ClientHello doesn’t initiate a new connection because after reaching to state 2, next input message closes the connection. These transitions again shows weak implementation, where states are not checked against every possible input messages.

VI-B *Windows 8 TLS 1.0 RSA

 [Client Implementation] (Figure 4)*

VI-B1 Valid Transition (Handshake)

Similar to figure 3, figure 4 also depicts a valid handshake for Windows 8 client implementation which is shown using green arrows (path $0\rightarrow 1\rightarrow 3\rightarrow 4\rightarrow 5\rightarrow 6$). This path validates the transitions mentioned in discussed figure 2 which signifies the existence of a successful handshake.

VI-B2 Invalid/undesired Transitions

• •

As we can see states 0, 1, 3, 4, 5 have a self-loop for two inputs called ApplicationData and AlertWarning. State 5 has a self-loop for another input called ApplicationDataEmpty. State 6 has a self-loop for ApplicationDataEmpty and AlertWarning message. As mention in section V, these loops may exhaust CPU resource if multiple connections are started frequently.

• •

Undesired replies to messages in state 6: For three kinds of input messages types ServerHelloRSA, ServerCertificate and Finished abnormal behavior is detected in state 6 where instead of closing the connection, ClientHello is sent and state is changed to 2. Although this ClientHello doesn’t initiate a new connection because after reaching to state 2, next input message terminates the connection. Existence of such state machine bugs show lack of understanding of protocol specification specified in RFCs.

VI-C *Windows 8 and 10 TLS 1.2 RSA

 [Client Implementation] (Figure 5)*

VI-C1 Valid Transition (Handshake)

In figure 5 a valid handshake is shown using green arrows (path $0\rightarrow 1\rightarrow 5\rightarrow 6\rightarrow 7\rightarrow 8$). Existence of this path validates the handshake transitions required for TLS 1.2. This handshake is validated for RSA and ECDHE key exchange mechanism and both resulted in similar state transitions for a successful handshake.

VI-C2 Invalid/undesired Transitions

• •

In figure 5, state 3 is an extra sink state which is created by receiving ChangeCipherSpec message at the start of connection. This state can also be reached from state 9 with same input. Once control is reached to state 3, it is never passed to any other state for any input and therefore sink state 3 is created. There are states (1, 3, 4, 5, 6, 7, 8) which have self-loops for mainly two inputs called ApplicationData and AlertWarning. These self-loop with no outputs result in idle transitions which only absorb given inputs and don’t play any meaningful role in the system of states.

• •

Alternate path to reach state 1: Upon receiving ApplicationData or AlertWarning messages at the start of connection instead of ServerHello, the connection doesn’t close and control goes to state 4 which creates an alternative path to the state 1. This alternate path is similar to the transition we have discussed in figure 3. State 4 passes the control to state 1 upon receiving ServerHello message and this behavior is similar to state 0. As this path doesn’t bypass any authentication or key exchange message it is not a security vulnerability but an implementation bug which deviates from the path of valid handshake.

• •

Undesired replies to messages in state 8 and 9: Similar to state 9 of figure 3, state 8 replies with ClientHello message and passes the control to state 9 upon receiving ServerHello or ServerCertficate or Finished message. In this case, expected behavior was to close the connection but ClientHello message is sent to the server. This ClientHelloRSA doesn’t initiate a connection because from state 9 all connection are closed for all inputs except ChangeCipherSpec. Therefore this bug doesn’t qualify for a security vulnerability. State 9 transfers the control to state 3 with no output upon receiving ChangeCipherSpec message but after state 3 every input is absorbed and system doesn’t output any message. Although these undesired replies and wrong transitions are not security issues, but must be fixed to prevent exploitation of such bugs in future.

VI-D *Windows Server 2008 TLS 1.0 RSA

 [Server Implementation] (Figure 6 )*

VI-D1 Valid Transition (Handshake)

In figure 6 a valid handshake is shown using green arrows (path $0\rightarrow 1\rightarrow 4\rightarrow 5\rightarrow 6\rightarrow 2$). This path validates the transitions mentioned in figure 2 for a valid handshake.

VI-D2 Invalid/undesired Transitions

• •

Similar to model described in figure 3 for Windows 7, a sink state is created when receiving ApplicationData or Alert message as the first message to communication. This sink state accepts all message with empty reply and loops the control flow of connection to itself. States 1, 4, 5, 6 have a self-loop each for two inputs called Alert-Warning and ApplicatonData(only for state 6). Upon receiving these inputs, mentioned states don’t respond and wait for next input without changing the state. These states receive such inputs till the connection timeout and then closed forcefully.

VI-E *Windows Server 2012

 [Server Implementation] (Figure 7)*

VI-E1 Valid Transition (Handshake)

Similar to state machine model for Windows Server 2008 green arrows (path $0\rightarrow 1\rightarrow 5\rightarrow 6\rightarrow 7\rightarrow 2$) in figure 7 depicts state transitions for a valid handshake. This path is observed for not only for RSA but also for ECDHE key exchange mechanism and in both the cases similar state machine model is learned.

VI-E2 Invalid/undesired Transitions

• •

In figure 7 states 1, 4, 5, 6, 7 have a self-loop for two inputs called ApplicationData and AlertWarning. These self-loops signify that for these mentioned two inputs state is not changed and the state absorb the inputs without responding any output.

• •

Alternate path to states 2 and 5: In figure 7 transitions through state 4 create an alternate path to reach state 5 from state 1. There is one more transition through state 3 to reach state 2 from state 0. State 0 receives AlertWarning/AlertFatal message and make a transition to state 3 which on receiving any input message changes the state to state 2. Both transitions are unexpected but doesn’t meet the requirement of security vulnerability as no security parameter is bypassed or changed using these transitions. Similar kind of behavior is observed in figure 5 where existence of alternate paths is described. This behavior shows that similar bugs which existed in client implementation, also exists in server implementation. Reuse of client code in server implementation could be a possible reason for such improper design. Therefore designers and programmers must understand and design control flow of protocol messages separately for client and server.

VI-F *Windows Server 2016

 [Server Implementation] (Figure 8)*

VI-F1 Valid Transition (Handshake)

Figure 8 describes a valid handshake which is shown using green arrows (path $0\rightarrow 1\rightarrow 3\rightarrow 4\rightarrow 5\rightarrow 2$). This path is almost similar to the one which is shown in figure 2 and have lesser number of undesired states than others. By comparing this model with other models it is quite clear that this model is more correct and simpler than previous state machine models. Most of the times simpler models are less vulnerable due to lesser complexity of state transitions which leaves lesser area for vulnerable attack surface. It is also clear that latest versions of SChannel are continuously improving by approaching towards ideal state machine model.

VI-F2 Invalid/undesired Transitions

• •

In figure 8 states 1, 3, 4, 5 have a self-loop for three types of inputs ApplicationData, ApplicationDataEmpty and Alert-Warning. Although this model is simpler and better but self-loops are still there which may put the system in unnecessary computation and wait condition, instead of closing the connection.

VII Implications

In this paper, we have discussed about various kinds of bugs and vulnerabilities present in state machines of various versions of Windows TLS library called SChannel. Windows operating system is one of the most popular operating system, therefore presence of a bug in Windows directly affects many computer systems around the globe. We have discussed in previous sections, how these bugs can cause denial of service attacks, which can shutdown a critical Windows based server of an organization to create panic to the users and financial loss to the organization.

To protect the system from such attacks, it is always necessary for developers as well as for testers to verify each possible transition of a protocol implementation. Leaving one loophole in implementation may cause severe vulnerability in the system. Understanding the protocol specification is also an important part of protocol implementation and it must be done as a group task so that individual understanding of the specification doesn’t influence the implementation.

VIII Summary

This paper presents state machine models for TLS implementation of Windows operating system. These models have been designed using query based machine learning technique. The paper describe various bugs and vulnerabilities present in these model due to incorrect implementation of TLS protocol in SChannel library. In this paper, we have analyzed a subset of messages that are exchanged between a client and a server but we have not included state machine behavior for many optional messages and various extensions of messages. Analysis of complete set of these messages could reveal more bugs in the implementation which could cause severe attacks on secure communication.