Python Threadpool

1. Thread Pool Definition
2. Differences Between multiprocessing.pool.ThreadPool and multiprocessing.Pool
3. multiprocessing.Pool in Python
4. multiprocessing.pool.Threadpool in Python

This tutorial will show the difference between Pool from multiprocessing and ThreadPool from multiprocessing.pool

Thread Pool Definition

A thread pool is a group of pre-instantiated, idle threads that stand ready to be given work. Creating a new thread object for each task to be executed asynchronously is expensive. With a thread pool, you would add the task to a task queue, and the thread pool assigns an available thread for the task. The thread pool helps to avoid creating or destroying more threads than would be necessary.

Differences Between multiprocessing.pool.ThreadPool and multiprocessing.Pool

multiprocessing.pool.ThreadPool behaves the same way as multiprocessing.Pool. The difference is that multiprocessing.pool.Threadpool uses threads to run the worker’s logic while multiprocessing.Pool uses worker processes.

multiprocessing.Pool in Python

The code below will spawn 4 different processes that will each run the function sleepy().

# python 3.x
from multiprocessing import Pool
import os
import time


def sleepy(x):
    print("Process Id", os.getpid())
    time.sleep(x)
    return x


if __name__ == "__main__":
    p = Pool(5)
    pool_output = p.map(sleepy, range(4))
    print(pool_output)

Output:

Process Id 239
Process Id 240
Process Id 241
Process Id 242
[0, 1, 2, 3]

multiprocessing.pool.Threadpool in Python

ThreadPool will generate 4 threads that run the sleepy() function instead of worker processes.

# python 3.x
from multiprocessing.pool import ThreadPool
import os
import time


def sleepy(x):
    print("Process Id", os.getpid())
    time.sleep(x)
    return x


if __name__ == "__main__":
    p = ThreadPool(5)
    pool_output = p.map(sleepy, range(4))
    print(pool_output)

Output:

Process Id 217
Process Id 217
Process Id 217
Process Id 217
[0, 1, 2, 3]