How to Perform a Parallel Processing in Python

1. Use the Processing Pool and Its Methods to Perform Multiprocessing in Python
2. Use the apply_async() Function to Perform Multiprocessing in Python
3. Use the map() and map_sync() Functions to Perform Multiprocessing in Python

This article explains how to execute multiple processes in parallel in Python.

Use the Processing Pool and Its Methods to Perform Multiprocessing in Python

To perform multiprocessing in Python, do the following steps.

• First, import the multiprocessing module.
• Use the Pool class to define the Python Processing Pool. This Pool represents a pool of worker processes. Take the code snippet below as an example.

Pool(processes=n)

Accordingly, the Pool class will allow the n number of processes to run simultaneously. This class contains methods that enable the delegation of tasks to the Pool’s worker processes.

The followings are some of the methods of the Pool class.

• Pool.map() - Use this blocking method of the Pool to block until the result is ready. This method chops the associated one iterable into several chunks. Then, it submits the chunks to the process pool as separate tasks.

• Use the following non-blocking methods of the Pool class to submit the processes simultaneously at once. Use the get() method to obtain the function’s results.

• Pool.apply_async()

• Pool.map_async()

To determine the number of processes that your machine can handle for multiprocessing, you should know the number of processors you have in your device; you can check this by using the following code.

import multiprocessing as mp

print("Number of processors: ", mp.cpu_count())

Output:

Number of processors:  4

Use the apply_async() Function to Perform Multiprocessing in Python

Here is an example demonstrating multiprocessing using the processing pool in Python while determining the square of a range on numbers from 0-9.

Note that here, we have even printed the Process worker’s name, which is optional.

from multiprocessing import Pool
from multiprocessing import Process, current_process
import time
from time import sleep
import random


def fnc_square(x):
    print(current_process().name)
    return x * x


if __name__ == "__main__":
    pool = Pool(processes=4)
    result = pool.apply_async(fnc_square, [6])
    print(result.get(timeout=1))
    print(pool.map(fnc_square, range(10)))

Output:

SpawnPoolWorker-1
36
SpawnPoolWorker-2
SpawnPoolWorker-1
SpawnPoolWorker-2
SpawnPoolWorker-1
SpawnPoolWorker-2
SpawnPoolWorker-1
SpawnPoolWorker-2
SpawnPoolWorker-1
SpawnPoolWorker-2
SpawnPoolWorker-1
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Use the map() and map_sync() Functions to Perform Multiprocessing in Python

The map_async function is non-blocking, whereas the map function is blocking. Below is an example that demonstrates the difference between these two commands.

from multiprocessing import Pool
import time


def f(x):
    print(x * x)


if __name__ == "__main__":
    pool = Pool(processes=1)
    pool.map(f, range(10))
    r = pool.map_async(f, range(10))
    print("First print")
    print("Second print")
    r.wait()
    print("Third print")

Here, we can observe that the pool.map() function will wait for the 10 operation calls to complete. Hence, we see the result printed in order.

Additionally, the pool.map_async() will execute the 10 function calls asynchronously. When the r.wait() function is called, it blocks the processor for execution. Hence, in the output, we see the print messages as First print and Second print in between, but the Third print will always be at the end.

Output:

0
1
4
9
16
25
36
49
64
81
First print
Second print
0
1
4
9
16
25
36
49
64
81
Third print