Sentry Answers>Python>

Remove duplicates in a Python list

Remove duplicates in a Python list

David Y.

—

The Problem

How can I remove duplicate elements from a list in Python?

The Solution

The simplest way to do this is to convert our list into a set, a Python data structure that cannot contain duplicates, and then convert it back to a list. For example:

Click to Copy
list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
list_without_duplicates = list(set(list_with_duplicates))
print(list_without_duplicates)
# will print [1, 2, 3, 4, 5]

As we can see, this method does not preserve the order of the list. If we want to remove duplicates while preserving the list order, we can use dict.fromkeys to create a new dictionary using the list items as keys, and then convert that dictionary back into a list:

Click to Copy
list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
# remove duplicates while preserving list order
list_without_duplicates = list(dict.fromkeys(list_with_duplicates))
print(list_without_duplicates)
# will print [5, 2, 4, 1, 3]

This works because keys in a dictionary must be unique, and since Python 3.7, keys are ordered according to when they were inserted. However, it may be considered unclear. A more straightforward alternative would be to use a for loop:

Click to Copy
list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
list_without_duplicates = []
for item in list_with_duplicates:
    if item not in list_without_duplicates:
        list_without_duplicates.append(item)

print(list_without_duplicates)
# will print [5, 2, 4, 1, 3]

This loop adds items from the first list to the second list, skipping any items that are already present.

  • Sentry BlogPython Performance Testing: A Comprehensive Guide
  • Syntax.fmListen to the Syntax Podcast
  • Sentry BlogLogging in Python: A Developer’s Guide
  • Syntax.fm logo
    Listen to the Syntax Podcast

    Tasty treats for web developers brought to you by Sentry. Get tips and tricks from Wes Bos and Scott Tolinski.

    SEE EPISODES

Considered “not bad” by 4 million developers and more than 100,000 organizations worldwide, Sentry provides code-level observability to many of the world’s best-known companies like Disney, Peloton, Cloudflare, Eventbrite, Slack, Supercell, and Rockstar Games. Each month we process billions of exceptions from the most popular products on the internet.

learn more about sentryjoin our discord
© 2024 • Sentry is a registered Trademark
of Functional Software, Inc.