Remove duplicates in a Python list
David Y.
—The Problem
How can I remove duplicate elements from a list in Python?
The Solution
The simplest way to do this is to convert our list into a set, a Python data structure that cannot contain duplicates, and then convert it back to a list. For example:
list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5] list_without_duplicates = list(set(list_with_duplicates)) print(list_without_duplicates) # will print [1, 2, 3, 4, 5]
As we can see, this method does not preserve the order of the list. If we want to remove duplicates while preserving the list order, we can use dict.fromkeys to create a new dictionary using the list items as keys, and then convert that dictionary back into a list:
list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5] # remove duplicates while preserving list order list_without_duplicates = list(dict.fromkeys(list_with_duplicates)) print(list_without_duplicates) # will print [5, 2, 4, 1, 3]
This works because keys in a dictionary must be unique, and since Python 3.7, keys are ordered according to when they were inserted. However, it may be considered unclear. A more straightforward alternative would be to use a for loop:
list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5] list_without_duplicates = [] for item in list_with_duplicates: if item not in list_without_duplicates: list_without_duplicates.append(item) print(list_without_duplicates) # will print [5, 2, 4, 1, 3]
This loop adds items from the first list to the second list, skipping any items that are already present.
- Sentry BlogPython Performance Testing: A Comprehensive Guide
- Syntax.fmListen to the Syntax Podcast
- Sentry BlogLogging in Python: A Developer’s Guide
- CodecovPython - Codecov
- Listen to the Syntax Podcast
![Syntax.fm logo]()
Tasty treats for web developers brought to you by Sentry. Get tips and tricks from Wes Bos and Scott Tolinski.
SEE EPISODES
Considered “not bad” by 4 million developers and more than 100,000 organizations worldwide, Sentry provides code-level observability to many of the world’s best-known companies like Disney, Peloton, Cloudflare, Eventbrite, Slack, Supercell, and Rockstar Games. Each month we process billions of exceptions from the most popular products on the internet.