Detailed Explanation of TCP SYN Cookie Mechanism

Detailed Explanation of TCP SYN Cookie Mechanism

I. Mechanism Description
SYN Cookie is a technique to defend against SYN Flood attacks, proposed by Daniel J. Bernstein. When a server detects that it may be under a SYN Flood attack, it activates this mechanism. The core idea is: during the TCP three-way handshake, the server does not immediately allocate resources to store connection state. Instead, it generates a special sequence number (i.e., SYN Cookie) through calculation and uses it as the initial sequence number returned to the client. Only after receiving the client's ACK acknowledgment does the server allocate resources to establish the connection.

II. Detailed Working Mechanism

Step 1: Attack Detection and Mechanism Trigger

  • The server monitors the length of the half-open connection queue (SYN queue).
  • When the queue length exceeds a threshold (e.g., more than 50% of the maximum length), a possible attack is identified.
  • Automatically switches to SYN Cookie mode, ceasing to maintain the half-open connection queue.

Step 2: SYN Cookie Generation (Server Side)
Upon receiving a SYN packet, the server calculates the Cookie value using the following parameters:

Cookie = Hash(Source IP, Source Port, Destination IP, Destination Port, Random Seed) mod 2^24

Specific calculation process:

  1. Take the lower 6 bits of the current timestamp (accurate to the minute) as t.
  2. Use HMAC algorithm to encrypt the quintuple information to obtain an mss digest.
  3. Combine to generate the sequence number: seq = (t << 24) | (mss << 16) | HMAC_signature.

Step 3: SYN-ACK Response

  • The server uses the calculated Cookie value as the initial sequence number.
  • Does not allocate a TCB (Transmission Control Block) to store connection information.
  • Releases related resources immediately after sending the SYN-ACK packet.

Step 4: Cookie Verification (Upon Receiving ACK)
After receiving an ACK packet, the server verifies:

  1. Extract the acknowledgment number ack_num: ack_num - 1 is the original Cookie value.
  2. Check if the timestamp t is within the validity period (typically 2 minutes).
  3. Recalculate the HMAC signature and compare it with the signature in the Cookie.
  4. If verification passes, extract the negotiated MSS value from the Cookie.

Step 5: Connection Establishment

  • If verification passes: allocate TCB resources and complete connection establishment.
  • If verification fails: discard the ACK packet directly without any response.

III. Technical Characteristics Analysis

Advantages:

  • Complete defense against SYN Flood attacks: attackers cannot forge legitimate ACKs.
  • Stateless design: the server does not maintain a half-open connection queue, avoiding resource exhaustion.
  • Backward compatibility: clients can work normally without any modifications.

Limitations:

  • Does not support all TCP options: due to Cookie length limitations, some options cannot be negotiated in the SYN-ACK.
  • Computational overhead: requires additional hash calculations, potentially increasing CPU load.
  • Time precision requirement: server and client times cannot differ significantly.

IV. Practical Application
In Linux systems, it can be configured as follows:

# Enable SYN Cookie
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

# Adjust trigger threshold (enable when SYN queue length exceeds 128)
echo 128 > /proc/sys/net/ipv4/tcp_max_syn_backlog

This mechanism effectively defends against SYN Flood attacks while ensuring service availability.