Distributed Lock Implementation and Selection Strategy in Microservices

Distributed Lock Implementation and Selection Strategy in Microservices

Description
A distributed lock is a core mechanism in microservice architecture for coordinating exclusive access to shared resources among multiple service instances. When multiple microservices need to operate on the same resource simultaneously (e.g., a database row, a file, or a business entity), a distributed lock ensures that only one service can perform critical operations at any given moment, preventing data races and inconsistencies. Compared to single-machine locks, distributed locks must address challenges unique to distributed environments, such as network latency, node failures, and clock drift.

Problem-Solving Process

  1. Understand the Core Requirements of Distributed Locks

    • Mutual Exclusivity: At most one client can hold the lock at any time.
    • Fault Tolerance: The lock service should remain functional even if some components fail (e.g., when more than half of the nodes are alive).
    • Deadlock Prevention: Locks must have a timeout to prevent indefinite holding if a client crashes.
    • Reentrancy (Optional): The same thread can acquire the same lock multiple times.

  2. Database-Based Implementation

    • Pessimistic Locking Implementation: 
      -- Create lock table
CREATE TABLE distributed_lock (
    id VARCHAR(64) PRIMARY KEY,  -- Lock identifier
    holder_id VARCHAR(100),      -- Holder identifier
    expire_time TIMESTAMP        -- Expiration time
);

-- Acquire lock (atomic operation)
INSERT INTO distributed_lock VALUES ('order_lock_123', 'service-A', NOW() + INTERVAL 30 SECOND);
-- Insertion success indicates lock acquisition; requires unique index to prevent duplicate inserts
    • Optimistic Locking Implementation: Uses version numbers, suitable for low-conflict scenarios.
    • Drawbacks: Database performance bottlenecks, requires handling connection timeouts and deadlock detection.

  3. Redis-Based Implementation

    • Basic Implementation Using SETNX Command: 
      # Set key-value with timeout (atomic operation)
SET lock:order_123 <client_id> NX PX 30000
    • RedLock Algorithm (Fault-tolerant multi-node):
      1. Sequentially send lock requests to 5 independent Redis instances.
      2. If at least 3 nodes successfully acquire the lock, and the total time spent is less than the lock timeout, the lock is considered acquired.
      3. To release the lock, send deletion requests to all nodes.
    • Considerations: Evaluate the impact of clock drift; it is recommended to use mature clients like Redisson.

  4. ZooKeeper-Based Implementation

    • Ephemeral Sequential Node Mechanism:
      1. Create an ephemeral sequential node under the lock path (e.g., /lock/order_123/_node_001).
      2. Check if your node is the one with the smallest sequence number; if yes, the lock is acquired.
      3. Otherwise, watch for the deletion event of the preceding node.
    • Session Management: The client maintains a heartbeat with ZooKeeper; the lock is automatically released if the session expires.
    • Advantages: Strong consistency, no need to set tricky timeouts; Disadvantages: Lower performance compared to Redis.

  5. Selection Strategy and Practical Considerations

    • Consistency Requirements:
      • CP systems (ZooKeeper) ensure strong consistency, suitable for critical scenarios like finance.
      • AP systems (Redis) offer higher performance but may allow lock failure in extreme cases.
    • Performance Needs: Redis throughput can exceed 100k/sec, while ZooKeeper is around 10k/sec.
    • Operational Costs: ZooKeeper requires cluster maintenance, whereas Redis can use managed cloud services.
    • Hybrid Strategy: Use Redis for non-critical business and ZooKeeper + database redundancy checks for core transactions.

  6. Fault Tolerance and Fallback Plans

    • Lock Timeout Configuration: Dynamically adjust based on the maximum operation duration (e.g., 2-3 times the average duration).
    • Automatic Renewal Mechanism: A background thread periodically extends the lock timeout (watchdog pattern).
    • Fallback Strategy: Degrade to a local lock + alert when the distributed lock fails to ensure basic availability.
    • Lock Release Verification: Verify the holder's identity before releasing the lock to prevent accidental deletion of another client's lock.

Through the above steps, the most suitable distributed lock implementation can be selected based on specific business scenario requirements, team technical stack, and operational capabilities.