Buffer Overflow Attacks and Protection in Operating Systems

Description
Buffer overflow is a common security vulnerability that occurs when a program writes data to a buffer, exceeding its pre-allocated memory boundaries, thereby overwriting adjacent memory regions. Attackers can exploit this vulnerability to alter a program's execution flow (e.g., hijacking control or executing malicious code). Understanding its principles and protection methods is crucial for system security.

Knowledge Point Explanation

1. Buffers and Memory Layout

   • A buffer is a contiguous block of memory space used by a program during runtime to store data (e.g., a character array).
   • In a program's memory layout, buffers are typically located in the stack area. The stack also stores function local variables, return addresses, and parameters. The return address, which determines where the program jumps after a function finishes executing, is a key target for attacks.
   • Example: If an array char buffer[8] is defined but 10 bytes are written into it, the extra 2 bytes will overwrite adjacent content on the stack.

2. Overflow Principle and Attack Steps

   • Step 1: Overwrite the Return Address
     The attacker crafts input data longer than the buffer size, carefully designing the overflow portion to overwrite the return address with the address of malicious code (e.g., the entry point of shellcode).
     • Example: Assuming the stack structure is [buffer][old frame pointer][return address], writing data beyond the buffer will sequentially overwrite the pointer and return address.
   • Step 2: Hijack Control Flow
     When the function executes the ret instruction, the CPU reads the location of the next instruction from the modified return address and switches to executing the attacker's injected code.

3. Protection Mechanisms

   • Non-Executable Stack (NX/DEP)
     Modern operating systems mark the stack as non-executable. Even if code is injected, the CPU will refuse to execute instructions within the stack.
   • Address Space Layout Randomization (ASLR)
     Each time a program runs, the system randomizes the base addresses of the stack and heap, making it difficult for attackers to predict the value to overwrite the return address with.
   • Stack Protection Technology (Canary)
     The compiler inserts a random value (canary) before the return address. Before the function returns, it checks whether this value has been modified. If corrupted, the program terminates immediately.

Summary
The core of buffer overflow is breaching memory boundaries through input data to tamper with control flow data. Protection requires a multi-layered approach combining hardware (NX), operating system (ASLR), and compiler tools (Canary) to form a defense-in-depth system.