TCP Three-Way Handshake and SYN Flood Attack

TCP Three-Way Handshake and SYN Flood Attack

Problem Description
The TCP three-way handshake is the foundational process for establishing a network connection. However, malicious attackers can exploit its design flaws to launch SYN Flood attacks. Please explain the steps of the three-way handshake, and describe the principle of SYN Flood attacks as well as defense methods.

1. TCP Three-Way Handshake Process
TCP uses the three-way handshake to ensure both parties have normal communication capabilities. The steps are as follows:

Step 1: Client sends SYN packet

  • The client generates a random initial sequence number (e.g., x), sets the SYN flag bit in the TCP header to 1, and sends a SYN packet to the server.
  • The client then enters the SYN_SENT state, waiting for the server's response.

Step 2: Server replies with SYN-ACK packet

  • Upon receiving the SYN packet, the server allocates connection resources (e.g., buffers) and generates its own random sequence number (e.g., y). It sets both SYN=1 and ACK=1.
  • The acknowledgment number is set to x+1, indicating receipt of the client's sequence number x.
  • The server enters the SYN_RECEIVED state.

Step 3: Client sends ACK packet

  • The client verifies that the acknowledgment number is x+1. If correct, it sends a packet with ACK=1 and sets the acknowledgment number to y+1.
  • After the server receives this packet, both parties enter the ESTABLISHED state, and the connection is fully established.

Key Points: The three-way handshake ensures both parties can send and receive data, and the randomization of sequence numbers prevents interference from old connections.

2. Principle of SYN Flood Attack
Attackers exploit a flaw in the three-way handshake:

  • In Step 2, after receiving a SYN packet, the server must allocate resources and wait for the client's ACK (default wait time is tens of seconds).
  • The attacker sends a large volume of SYN packets with spoofed source IP addresses. The server continuously allocates resources but never receives the Step 3 ACK replies.
  • The server's resources become exhausted, making it unable to respond to legitimate connection requests, resulting in a Denial of Service (DoS).

Example:
The attacker sends SYN packets with randomly forged source IPs. The server sends SYN-ACKs to non-existent IPs, eventually filling up its resource pool (e.g., the half-open connection queue).

3. Methods to Defend Against SYN Flood Attacks
(1) SYN Cookie

  • In Step 2, the server does not immediately allocate resources. Instead, it uses an encryption algorithm (e.g., hashing) to generate a sequence number y, encoding the connection information within it.
  • Only after receiving and validating the legitimacy of the client's ACK packet does the server allocate resources. This way, attack packets do not consume actual resources.

(2) Increase the Half-Open Connection Queue Size

  • Adjust the size of the half-open connection queue via the operating system (e.g., Linux's tcp_max_syn_backlog parameter) to mitigate small-scale attacks.

(3) Firewall and Intrusion Detection System (IDS)

  • Set thresholds to monitor the frequency of SYN requests and automatically block suspicious IPs.
  • Use blacklists to filter known malicious sources.

(4) Load Balancing and Cloud Protection

  • Distribute traffic through CDN or cloud security services (e.g., AWS Shield), which filter malicious requests before forwarding them to the server.

Summary
The three-way handshake is the cornerstone of TCP reliability, but its security risks must be considered. SYN Flood attacks exploit the resource allocation mechanism in the protocol design to cause DoS through resource exhaustion. Defense requires a combination of protocol optimization (e.g., SYN Cookies) and external protective measures.