How to Remove Duplicates From List in Python

1. Remove Duplicate From a List Using the set() Function in Python
2. Remove Duplicates & Maintain Order in a List Using OrderDict in Python

A List in Python is a data structure that is used to store data in a particular order. The list can store data of multiple types i.e. int, float, string, another list, etc. Lists are mutable, which means values once created can be changed later. It is represented by square brackets [].

myList = [2, 1, 2, 3, 0, 6, 7, 6, 4, 8]
print(myList)

Output:

[2, 1, 2, 3, 0, 6, 7, 6, 4, 8]

You can remove duplicate elements from the above list using a for loop as shown below.

myList = [2, 1, 2, 3, 0, 6, 7, 6, 4, 8]

resultantList = []

for element in myList:
    if element not in resultantList:
        resultantList.append(element)

print(resultantList)

Output:

[2, 1, 3, 0, 6, 7, 4, 8]

If you don’t want to write this much code, then there are two most popular ways to remove duplicate elements from a List in Python.

1. If you don’t want to maintain the order of the elements inside a list after removing the duplicate elements, then you can use a Set data structure.
2. If you want to maintain the order of the elements inside a list after removing duplicate elements, then you can use something called OrderedDict.

Remove Duplicate From a List Using the set() Function in Python

For removing duplicates from a list, we can use another data structure called Set. A Set is an unordered data type that contains only unique values. The order in which the set stores the values is different than the order in which you might have inserted the elements in a set. Sets are represented using curly braces {}.

So, whenever you print the elements stored inside the set then the order of the output will be different. That’s the reason why indexing cannot be performed on a set because it is unordered.

mySet = {80, 10, 50, 18, 3, 50, 8, 18, 9, 8}
print(mySet)

Output:

{3, 8, 9, 10, 80, 18, 50}

Note how the order of elements has changed after we have printed the elements.

In a set, no duplicates are allowed, which means if you have the same element multiple times, then the set will consider it as one element. That’s the reason why we use a set to remove duplicates elements from a list in Python.

original_list = [80, 10, 50, 18, 3, 50, 8, 18, 9, 8]

print("Original List is: ", original_list)

convert_list_to_set = set(original_list)
print("Set is: ", convert_list_to_set)

new_list = list(convert_list_to_set)
print("Resultant List is: ", new_list)

original_list = list(convert_list_to_set)
print("Removed duplicates from original list: ", original_list)

Output:

Original List is:  [80, 10, 50, 18, 3, 50, 8, 18, 9, 8]
Set is:  {3, 8, 9, 10, 80, 18, 50}
Resultant List is:  [3, 8, 9, 10, 80, 18, 50]
Removed duplicates from original list:  [3, 8, 9, 10, 80, 18, 50]

We convert a list to a set by applying the set() function. Then we convert the converted set, which has removed the duplicates, to a list by applying the list() function.

Remove Duplicates & Maintain Order in a List Using OrderDict in Python

The problem with using a set to remove duplicated elements is that it does not store elements in a particular order. So, if you don’t care about the order in which the elements are stored in the resultant list i.e. the list which you have created after removing duplicates, then in that case you can go with the Set data structure.

But if you want to maintain the order of the elements inside a list after removing the duplicates, then you can use OrderedDict in Python. The OrderDict preserves the order in which the elements have been inserted in the list. To use OrderDict you first have to import it from the collections module in Python from collections import OrderedDict.

from collections import OrderedDict

myList = [2, 1, 2, 3, 0, 6, 7, 6, 8, 0, 4, 8]

final_list = list(OrderedDict.fromkeys(myList))

print(final_list)

Output:

[2, 1, 3, 0, 6, 7, 8, 4]