Remove duplicates in a Python list

David Y.

—November 15, 2023

The ProblemThe Problem

How can I remove duplicate elements from a list in Python?

The SolutionThe Solution

The simplest way to do this is to convert our list into a set, a Python data structure that cannot contain duplicates, and then convert it back to a list. For example:

list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
list_without_duplicates = list(set(list_with_duplicates))
print(list_without_duplicates)
# will print [1, 2, 3, 4, 5]

As we can see, this method does not preserve the order of the list. If we want to remove duplicates while preserving the list order, we can use dict.fromkeys to create a new dictionary using the list items as keys, and then convert that dictionary back into a list:

list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
# remove duplicates while preserving list order
list_without_duplicates = list(dict.fromkeys(list_with_duplicates))
print(list_without_duplicates)
# will print [5, 2, 4, 1, 3]

This works because keys in a dictionary must be unique, and since Python 3.7, keys are ordered according to when they were inserted. However, it may be considered unclear. A more straightforward alternative would be to use a for loop:

list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
list_without_duplicates = []
for item in list_with_duplicates:
    if item not in list_without_duplicates:
        list_without_duplicates.append(item)

print(list_without_duplicates)
# will print [5, 2, 4, 1, 3]

This loop adds items from the first list to the second list, skipping any items that are already present.