Coroutines and Asynchronous Programming in Python

1. Basic Concepts of Coroutines

1. Definition: A coroutine is a user-mode lightweight thread, controlled and scheduled by the program itself (rather than the operating system kernel).
2. Characteristics:
   • Execution can be paused (suspended) and later resumed when needed.
   • Switching between coroutines does not involve thread context switching, making it much less expensive than thread switching.
   • Multiple coroutines can run within a single thread, achieving concurrency.

2. Evolution of Coroutines

1. Generator Foundation (Before Python 3.4)

def simple_coroutine():
    print("Starting coroutine")
    x = yield  # Pause point, receives value from outside
    print("Received value:", x)

# Usage
coro = simple_coroutine()
next(coro)  # Start the coroutine, execute up to the first yield
coro.send(42)  # Send a value to the coroutine, resume execution

2. Introduction of the asyncio Library (Python 3.4)

import asyncio

@asyncio.coroutine  # Decorator to mark a coroutine
def old_style_coroutine():
    yield from asyncio.sleep(1)  # Delegate to other generators
    print("Execution completed")

3. async/await Syntax (Python 3.5+)

1. Keyword Definitions:

   • async def: Declares an asynchronous function (coroutine function).
   • await: Waits for an asynchronous operation to complete; the coroutine pauses but does not block the event loop.

2. Basic Syntax Example:

import asyncio

async def say_after(delay, message):
    await asyncio.sleep(delay)  # Asynchronous wait
    print(message)

async def main():
    # Sequential execution
    await say_after(1, "Hello")
    await say_after(1, "World")
    
    # Concurrent execution
    task1 = asyncio.create_task(say_after(1, "Task1"))
    task2 = asyncio.create_task(say_after(1, "Task2"))
    await task1
    await task2

# Run the coroutine
asyncio.run(main())

4. Operation Mechanism of Coroutines

1. Event Loop

   • Core scheduler that manages the execution of all coroutines.
   • Continuously checks for runnable coroutines and switches between them.
   • Provides infrastructure for asynchronous I/O, timers, etc.

2. Coroutine State Transitions:

   • Created: By calling an asynchronous function (but not yet executed).
   • Suspended: Pauses when encountering an await expression.
   • Resumed: Re-enters the ready queue when the awaited operation completes.
   • Completed: Function finishes execution or an exception occurs.

5. Key Points for Asynchronous Programming Practice

1. Error Handling:

async def risky_operation():
    try:
        await some_async_call()
    except Exception as e:
        print(f"Operation failed: {e}")

# Error handling for multiple coroutines
async def batch_operations():
    results = await asyncio.gather(
        task1(), task2(), task3(),
        return_exceptions=True  # Return exceptions as results
    )

2. Resource Management:

async def using_async_context():
    async with aiofiles.open('file.txt') as f:  # Asynchronous context manager
        content = await f.read()

6. Coroutines vs. Threads

1. Suitable Scenarios:

   • Coroutines: I/O-intensive tasks (network requests, file operations).
   • Threads: CPU-intensive tasks (computationally heavy operations).

2. Performance Characteristics:

   • Coroutines: Can handle tens of thousands of concurrent connections within a single thread.
   • Threads: Limited by the GIL; suitable for scenarios with high I/O but low CPU usage.

7. Practical Application Patterns

1. Producer-Consumer Pattern:

async def producer(queue):
    while True:
        item = await get_item()
        await queue.put(item)

async def consumer(queue):
    while True:
        item = await queue.get()
        await process_item(item)

Through this progressive learning approach, you can comprehensively grasp the core concepts of Python coroutines and asynchronous programming, from basic principles to practical applications.