Data Compression and Transmission Optimization in Distributed Systems

Problem Description
In distributed systems, nodes often need to transmit large volumes of data (such as logs, backups, state synchronization, etc.), but network bandwidth and storage resources are limited. Data compression and transmission optimization techniques reduce data volume, lowering network latency and storage costs. Interview questions may involve compression algorithm selection, trade-offs between compression and transmission, and optimization strategies under real-time requirements.

Detailed Knowledge Points

1. Why are data compression and transmission optimization needed?

   • Network Bottlenecks: Cross-data center transmission incurs high bandwidth costs and is prone to congestion.
   • Storage Efficiency: Compression reduces disk usage, especially suitable for cold data backups.
   • Energy Consumption Control: Reducing data transmission volume lowers overall system energy consumption.
   • Real-time Challenges: Compression/decompression requires computational overhead, necessitating a balance between compression ratio, speed, and resource consumption.

2. Classification and Selection of Compression Algorithms

   • Lossless Compression (e.g., GZIP, Zstandard, Snappy):
     • Applicable scenarios: Log files, database backups, code repositories, where complete data restoration is required.
     • Trade-off: Higher compression ratios (e.g., high levels of Zstandard) usually come with greater CPU overhead.
   • Lossy Compression (e.g., JPEG, MPEG):
     • Applicable scenarios: Multimedia streams, monitoring data, where partial precision loss is acceptable.
     • Example: Discarding multiple decimal places in monitoring metrics to save space.

3. Layered Compression Strategies

   • Transmission Phase Compression:
     • Real-time compression of data streams for network transmission (e.g., HTTP's Content-Encoding: gzip).
     • Technique: Choose algorithms based on data characteristics—text uses GZIP, binary data uses Snappy (faster).
   • Storage Phase Compression:
     • Compress data before writing to disk, such as database page compression or HDFS block compression.
     • Hot/cold data tiering: Use lightweight compression (LZ4) for hot data and high compression ratio algorithms (Brotli) for cold data.

4. Collaborative Optimization of Compression and Transmission

   • Incremental Transmission (e.g., rsync algorithm):
     • Transmit only the differential parts of files, relying on rolling hash (Rabin fingerprint) to split data blocks.
     • Process:
       1. The sender splits the file into blocks and calculates hash values.
       2. The receiver compares block hashes with existing files and identifies new blocks to transmit.
       3. Only differential blocks are sent to reassemble the new file.
   • Dictionary Preprocessing:
     • Pre-build dictionaries for frequently transmitted common data (e.g., JSON field names) and replace them with short identifiers.
     • Example: Serialization protocols like Avro/Protobuf have built-in Schemas to optimize field encoding.

5. Optimization Techniques in Real-time Systems

   • Pipeline Compression:
     • Split compression tasks into parallel stages (chunking → compression → sending) to avoid cumulative overall latency.
   • Adaptive Compression:
     • Dynamically adjust compression levels based on system load monitoring (e.g., use high compression ratio under low load, disable compression under high load).
   • Zero-copy Transmission:
     • Combine with Linux's sendfile system call to avoid multiple data copies between user space and kernel space.

6. Practical Case: Kafka's Data Compression

   • End-to-End Batch Compression:
     • The Producer compresses multiple messages in batches before sending, the Broker persists the compressed data, and the Consumer decompresses it.
     • Advantage: Reduces network I/O but requires balancing batch latency and throughput.
   • Compression Algorithm Comparison:
     • GZIP: High compression ratio, high CPU overhead;
     • Snappy: Fast compression, suitable for real-time scenarios;
     • LZ4: Balanced compression ratio and speed.

Summary
Data compression and transmission optimization must consider business scenarios (real-time requirements, data characteristics) and system resources (CPU/bandwidth ratio). The core principle is: Avoid compression for its own sake. Optimal strategies should be determined through performance testing, such as first monitoring actual data transmission bottlenecks and then selecting targeted algorithms or layered solutions.