How to Fix Shared Memory Issues & Lock Shared Resources in Python

1. Use multiprocessing.Array() to Use Shared Memory in Python
2. Use multiprocessing.Lock() to Lock the Shared Resources in Python

This tutorial explains different aspects of multiprocessing shared memory and demonstrates how to fix issues using shared memory. We’ll also learn how to use the lock to lock the shared resources in Python.

Use multiprocessing.Array() to Use Shared Memory in Python

One of the most critical aspects of multiprocessing is sharing the data between the processes when you have multiple child processes.

One of the essential properties of the child processes you create using the ability of the processing module is that they run independently and have their own memory space.

It means that the child’s process will have some memory space. And, any variable tries to create or will be changed in its own memory space, not in the memory space of its parent.

Let’s try understanding this concept by taking an example and jumping into the code by importing the multiprocessing module.

We created an empty list called RESULT, and we have defined a function called Make_Sqaured_List(), which squares the elements of a given list and appends them to our global RESULT list.

The Procc_1 object is equal to the Process() class and sets the target as the Make_Sqaured_List function without brackets.

And, to the args parameter, we pass a separate list called Items_list that will be given as an argument to the Make_Sqaured_List() function.

Example Code:

import multiprocessing

RESULT = []


def Make_Sqaured_List(Num_List):
    global RESULT

    for n in Num_List:
        RESULT.append(n ** 2)
    print(f"Result: {RESULT}")


if __name__ == "__main__":
    Items_list = [5, 6, 7, 8]

    Procc_1 = multiprocessing.Process(target=Make_Sqaured_List, args=(Items_list,))
    Procc_1.start()
    Procc_1.join()

Let’s execute this child process, and we get the result according to our child process that the value of the global list.

Result: [25, 36, 49, 64]

But, if you try to print the RESULT list that is still empty, what is happening with the RESULT list?

print(RESULT)

Output:

[]

According to our main process, our parent process is still empty, whereas according to the child process, the RESULT list has some content. It means simply that both our different processes have different memory spaces.

We can understand this by a scenario in which we have process one, which is our main program where we initially have an empty RESULT list.

And, when we create a child process, it also has an empty initially, and then the Make_Sqaured_List() function is executed, so the RESULT list contains some items. But, since we are accessing RESULT from the parent process in this memory space, the changes are not visible.

Example Code:

import multiprocessing

RESULT = []


def Make_Sqaured_List(Num_List):
    global RESULT

    for n in Num_List:
        RESULT.append(n ** 2)
    print(f"Result: {RESULT}")


if __name__ == "__main__":
    Items_list = [5, 6, 7, 8]

    Procc_1 = multiprocessing.Process(target=Make_Sqaured_List, args=(Items_list,))
    Procc_1.start()
    Procc_1.join()
    print(RESULT)

Output:

Result: [25, 36, 49, 64]
[]

So, what is the solution for this? But, first, let’s see how we can solve it.

Solution to Fix the Issues With Sharing Data Between Multiprocessing

In the section, we will see the solution which helps us get the value of any changes and fix the issue with sharing data between the multiprocessing.

The solution is called shared memory. The multiprocessing module provides us with two types of objects called Array and Value which can be used to share the data between the processes.

The Array is an array allocated from the shared memory; basically, there is a portion in your computer memory that we can call shared memory or a region that multiple processes can access.

So in that shared memory, we create a new array or a new value. These added values are not our basic Python data structures; there is something different and defined in the multiprocessing module itself.

Now we declare an object called RESULT_ARRAY using multiprocessing.Array(). Then, in this array, we have to pass the data type. We pass i as a string, meaning we will put integer values inside it, and we have to give the size.

Example Code:

RESULT_ARRAY = multiprocessing.Array("i", 4)

It is related to a C programming style array so that we can give the size simultaneously. This way, we can store objects in the desired place.

Now we are creating a new value called OBJ_Sum is equal to multiprocessing.Value(), and it will store and enter the value.

Example Code:

OBJ_Sum = multiprocessing.Value("i")

Next, we will create an object called procc_1, which will be equal to multiprocessing.Process(), which we will call a function. We created a function called Make_Sqaured_List(), which will be taking three arguments:

1. A list
2. An array object
3. A value object.

We will pass these three arguments to our function using this Process argument called args. For example, look at the following code fence.

Example Code:

Procc_1 = multiprocessing.Process(
    target=Make_Sqaured_List, args=(Items_list, RESULT_ARRAY, OBJ_Sum)
)

Now in the Make_Sqaured_List() function, we are going to iterate Items_list using enumerate() function. So that we can get the index and value of Items_list.

It is a C-style array, so we will have to use indexing to assign the values to our array. We will also sum the values of an array, and the OBJ_Sum.value is a property of multiprocessing.Value().

Example Code:

def Make_Sqaured_List(Items_list, RESULT, OBJ_Sum):

    for i, n in enumerate(Items_list):
        RESULT[i] = n ** 2

    OBJ_Sum.value = sum(RESULT)

We have defined some variables in our main process, changing the function called by the child process. So our main agenda is whether we can get those changed values in our main process or not.

Now we can access the array reflected in the child process and get its sum using OBJ_Sum.value. For example, see the following code snippet.

Example Code:

import multiprocessing


def Make_Sqaured_List(Items_list, RESULT, OBJ_Sum):

    for i, n in enumerate(Items_list):
        RESULT[i] = n ** 2

    OBJ_Sum.value = sum(RESULT)


if __name__ == "__main__":
    Items_list = [5, 6, 7, 8]

    RESULT_ARRAY = multiprocessing.Array("i", 4)
    OBJ_Sum = multiprocessing.Value("i")

    Procc_1 = multiprocessing.Process(
        target=Make_Sqaured_List, args=(Items_list, RESULT_ARRAY, OBJ_Sum)
    )
    Procc_1.start()
    Procc_1.join()
    print(RESULT_ARRAY[:])
    print(OBJ_Sum.value)

Output:

[25, 36, 49, 64]
174

This way, we can make any changes to our objects defined in our parent process and those changes we are getting back from the child process. This is made possible by using the shared memory technique.

Use multiprocessing.Lock() to Lock the Shared Resources in Python

We will cover one crucial topic called lock; now, if you have taken computer science or operating system classes, you have already learned about the lock. However, the lock is a critical concept when it comes to multiprocessing and operating system concepts.

First, let’s consider why the lock is needed in real life; in our day-to-day life, some resources can not be accessed by two people simultaneously.

For example, the bathroom door has a lock because if two people try to access it simultaneously, it will create a pretty embarrassing situation. So that is why we protect the bathroom, a shared resource with a lock.

Similarly, in the programming world, whenever two processes or threads are trying to access a shared resource, such as a shared memory file or a database. It can create a problem, so you must protect that access with a lock.

What happens if we do not add that protection to our program, we will see that by running an example. Again, this is a banking software program, and here we have two processes.

The first process is depositing money into a bank using the MONEY_DP() function, and the second is withdrawing money from the bank using the MONEY_WD() function. And in the end, we are printing the final balance.

We are starting with a 200$ dollar balance in the MONEY_DP section. We are depositing 100$, we have a for loop which we traced through 100 times, and in each iteration, it will add 01$ to our bank account.

Similarly, in the MONEY_WD function, We have the same loop iterates 100 times and every time, it will deduct 1 dollars from our bank account.

Example Code:

import multiprocessing
import time


def MONEY_DP(B):
    for i in range(100):
        time.sleep(0.01)
        B.value = B.value + 1


def MONEY_WD(B):
    for i in range(100):
        time.sleep(0.01)
        B.value = B.value - 1

Now we are using a shared memory variable called value which we already learned about in the previous section. This multiprocessing value is a shared memory resource so let’s see what happens when we try to run this program.

if __name__ == "__main__":
    B = multiprocessing.Value("i", 200)
    Deposit = multiprocessing.Process(target=MONEY_DP, args=(B,))
    Withdrawl = multiprocessing.Process(target=MONEY_WD, args=(B,))
    Deposit.start()
    Withdrawl.start()
    Deposit.join()
    Withdrawl.join()
    print(B.value)

We will run it multiple times, and every time, it is just printing different values, but it should print 200$.

Output:

# execution 1
205
# execution 2
201
# execution 3
193

Why does this happen? It happens primarily when this process tries to read a variable called B.value in shared memory.

Let’s say B.value has a 200$ value, it will read it, and then it will add one, and then it will put back the same thing into the same variable.

Since B.value is 200$ and it is doing this addition operation at the operating system level, at the operating system level, it will be executing multiple assembly-line instructions.

So we have read the variable 200; it is added one and will assign back 201 to the B.value variable.

Example Code:

B.value = B.value + 1

Now while it is doing it at that same time, this instruction also got executed in the MONEY_WD() function.

Example Code:

B.value = B.value - 1

Although we are depositing first and then withdrawing, when the process tries to read B.value, it will still be 200 because the deposit process has not written back to the original variable.

Instead of reading B.value as 201 inside the MONEY_WD process, it will read B.value as 200, and after decreasing 1, it will have 199.

That is why we are getting inconsistent behaviour. First, let’s use Lock to lock the access; now, we create a variable called lock and use multiprocessing modules to use the Lock class.

Example Code:

lock = multiprocessing.Lock()

Now we will pass that lock to both processes and, inside both processes, we will call lock.acquire() to put a lock, and then to release a lock, we will call the lock.release() function.

These lock functions protect the code section while accessing the shared resource, called the critical section.

Example Code:

import multiprocessing
import time


def MONEY_DP(B, lock):
    for i in range(100):
        time.sleep(0.01)
        lock.acquire()
        B.value = B.value + 1
        lock.release()


def MONEY_WD(B, lock):
    for i in range(100):
        time.sleep(0.01)
        lock.acquire()
        B.value = B.value - 1
        lock.release()


if __name__ == "__main__":
    B = multiprocessing.Value("i", 200)
    lock = multiprocessing.Lock()
    Deposit = multiprocessing.Process(target=MONEY_DP, args=(B, lock))
    Withdrawl = multiprocessing.Process(target=MONEY_WD, args=(B, lock))
    Deposit.start()
    Withdrawl.start()
    Deposit.join()
    Withdrawl.join()
    print(B.value)

Now, this code is printing 200 every time.

Output:

200
PS C:\Users\Dell\Desktop\demo> python -u "c:\Users\Dell\Desktop\demo\demo.py"
200
PS C:\Users\Dell\Desktop\demo> python -u "c:\Users\Dell\Desktop\demo\demo.py"
200