A Quick Guide to the Python multiprocessing Module with Examples

Introduction

The multiprocessing module in Python allows you to create and manage processes, enabling you to take full advantage of multiple processors on a machine. It helps you achieve parallel execution by using separate memory spaces for each process, unlike threading where threads share the same memory space. Here's a list of commonly used classes and methods in the multiprocessing module with brief examples.

1. Process

The Process class is the core of the multiprocessing module, allowing you to create and run new processes.

from multiprocessing import Process

def print_numbers():
    for i in range(5):
        print(i)

p = Process(target=print_numbers)
p.start()  # Starts a new process
p.join()   # Waits for the process to finish

2. start()

Starts the process’s activity.

p = Process(target=print_numbers)
p.start()  # Runs the target function in a separate process

3. join([timeout])

Blocks the calling process until the process whose join() method is called terminates. Optionally, you can specify a timeout.

p = Process(target=print_numbers)
p.start()
p.join(2)  # Waits up to 2 seconds for the process to finish

4. is_alive()

Returns True if the process is still running.

p = Process(target=print_numbers)
p.start()
print(p.is_alive())  # True if the process is still running

5. current_process()

Returns the current Process object representing the calling process.

from multiprocessing import current_process

def print_current_process():
    print(current_process())

p = Process(target=print_current_process)
p.start()  # Prints the current process info

6. active_children()

Returns a list of all Process objects currently alive.

p1 = Process(target=print_numbers)
p2 = Process(target=print_numbers)
p1.start()
p2.start()

print(Process.active_children())  # Lists all active child processes

7. cpu_count()

Returns the number of CPUs available on the machine.

from multiprocessing import cpu_count

print(cpu_count())  # Returns the number of CPUs on the machine

8. Pool

A Pool object provides a convenient way to parallelize execution of a function across multiple input values. It manages a pool of worker processes.

from multiprocessing import Pool

def square(n):
    return n * n

with Pool(4) as pool:  # Pool with 4 worker processes
    result = pool.map(square, [1, 2, 3, 4, 5])

print(result)  # [1, 4, 9, 16, 25]

9. Queue

A Queue is a shared data structure that allows multiple processes to communicate by passing data between them.

from multiprocessing import Process, Queue

def put_data(q):
    q.put([1, 2, 3])

def get_data(q):
    data = q.get()
    print(data)

q = Queue()
p1 = Process(target=put_data, args=(q,))
p2 = Process(target=get_data, args=(q,))

p1.start()
p2.start()
p1.join()
p2.join()

10. Lock

A Lock ensures that only one process can access a shared resource at a time.

from multiprocessing import Process, Lock

lock = Lock()

def print_numbers():
    with lock:
        for i in range(5):
            print(i)

p1 = Process(target=print_numbers)
p2 = Process(target=print_numbers)

p1.start()
p2.start()
p1.join()
p2.join()

11. Value and Array

The Value and Array objects allow sharing simple data types and arrays between processes.

from multiprocessing import Process, Value

def increment(val):
    with val.get_lock():
        val.value += 1

shared_val = Value('i', 0)
processes = [Process(target=increment, args=(shared_val,)) for _ in range(10)]

for p in processes:
    p.start()

for p in processes:
    p.join()

print(shared_val.value)  # Output will be 10

12. Pipe

A Pipe provides a two-way communication channel between two processes.

from multiprocessing import Process, Pipe

def send_message(conn):
    conn.send("Hello from child")
    conn.close()

parent_conn, child_conn = Pipe()
p = Process(target=send_message, args=(child_conn,))
p.start()

print(parent_conn.recv())  # Receives data from the child process
p.join()

13. Manager

A Manager allows you to create shared objects, such as lists and dictionaries, that multiple processes can modify concurrently.

from multiprocessing import Process, Manager

def modify_list(shared_list):
    shared_list.append("New item")

with Manager() as manager:
    shared_list = manager.list([1, 2, 3])

    p = Process(target=modify_list, args=(shared_list,))
    p.start()
    p.join()

    print(shared_list)  # [1, 2, 3, "New item"]

14. Semaphore

A Semaphore allows you to control access to a resource, permitting only a certain number of processes to access it at a time.

from multiprocessing import Process, Semaphore
import time

sem = Semaphore(2)  # Only 2 processes can access the resource

def limited_access():
    with sem:
        print("Accessing resource")
        time.sleep(2)

processes = [Process(target=limited_access) for _ in range(5)]

for p in processes:
    p.start()

for p in processes:
    p.join()

Conclusion

The multiprocessing module in Python is designed to take full advantage of multiple processors on a machine. From creating and managing processes using Process, to controlling shared resources with Lock and Semaphore, and facilitating communication through Queue and Pipe, the multiprocessing module is crucial for parallelizing tasks in Python applications.