Python 5 Ways to remove duplicates from lists

Introduction

Removing duplicates in a Python list is a common task that can be achieved in multiple ways. In this article, we will explore four different methods to remove duplicates in a Python list.

We will go over the pros and cons of each method, as well as provide examples of how to use them.

Method 1: Using the set() function

The easiest and most straightforward way to remove duplicates in a Python list is by using the set() function. The set() function creates a new set object, which automatically removes duplicates.

To convert a list to a set, simply pass the list as an argument to the set() function.

original_list = [1, 2, 2, 3, 3, 4]
unique_list = list(set(original_list))
print(unique_list)

This will output: [1, 2, 3, 4]

The advantage of using the set() function is that it’s very simple and easy to understand. However, it also has the disadvantage of not preserving the original order of the elements in the list.

Method 2: Using a for loop and an empty list

Another way to remove duplicates in a Python list is by using a for loop and an empty list. This method involves iterating over the original list, and only appending elements that have not been seen before to the new list.

original_list = [1, 2, 2, 3, 3, 4]
unique_list = []
for i in original_list:
    if i not in unique_list:
        unique_list.append(i)
print(unique_list)

This will output: [1, 2, 3, 4]

The advantage of this method is that it preserves the original order of the elements in the list. However, it also has the disadvantage of being more complex and harder to understand than the previous method.

Method 3: Using list comprehension

A third way to remove duplicates in a Python list is by using list comprehension. This method involves creating a new list with a new list with only the unique elements from the original list using a list comprehension.

original_list = [1, 2, 2, 3, 3, 4]
unique_list = list(set(i for i in original_list))
print(unique_list)

This will output: [1, 2, 3, 4]

This method is similar to the first method in that it uses the set() function to remove duplicates, but it allows for a more concise and compact implementation.

However, it also preserves the order of the original list.

Method 4: Using the dict.fromkeys() method

A fourth way to remove duplicates in a Python list is by using the dict.fromkeys() method. This method creates a new dictionary where the elements of the original list are the keys and the values are set to None.

Since dictionaries only allow unique keys, this method effectively removes duplicates.

original_list = [1, 2, 2, 3, 3, 4]
unique_list = list(dict.fromkeys(original_list))
print(unique_list)

This will output: [1, 2, 3, 4]

The advantage of this method is that it preserves the order of the elements in the original list, similar to the second method, but it also allows for a more concise and compact implementation, similar to the third method.

Method 5: Using the OrderedDict method

Another way to remove duplicates in a Python list is by using the collections.OrderedDict.fromkeys() method.

This method is similar to the dict.fromkeys() method, but it allows for a more efficient and faster implementation as it is implemented in C. It creates an ordered dictionary where the elements of the original list are the keys and the values are set to None. Since dictionaries only allow unique keys, this method effectively removes duplicates and preserves the order of the elements in the original list.

Here is an example of how to use the collections.OrderedDict.fromkeys() method:

from collections import OrderedDict
original_list = 'abracadabra'
unique_list = list(OrderedDict.fromkeys(original_list))
print(unique_list)

This will output: ['a', 'b', 'r', 'c', 'd']

This method is not only the fastest but also the most memory efficient, especially for long lists. It is available in Python 3.5 and later versions.

It is important to note that the collections.OrderedDict.fromkeys() method is only available in Python 3.5 and later versions, therefore it cannot be used in earlier versions of Python.

Conclusion

In conclusion, there are various ways to remove duplicates in a Python list. Each method has its own advantages and disadvantages, and the choice of which method to use depends on the specific requirements of the task at hand. We have discussed five popular methods: using the set() function, using a for loop and an empty list, using list comprehension, using the dict.fromkeys() method, and using the collections.OrderedDict.fromkeys() method.

When deciding which method to use, it’s important to consider the requirements of your task, such as whether the order of the elements needs to be preserved or not and the performance of the method.

The set() function is the easiest and most straightforward method, but it does not preserve the order of the elements. The for loop and empty list method preserves the order of the elements, but is more complex. The list comprehension and the dict.fromkeys() methods are more concise and compact, while preserving the order of the elements. The collections.OrderedDict.fromkeys() method is a fast and efficient method for removing duplicates and preserving the order of the elements in Python 3.5 and later versions.

In summary, the set() function is the easiest method but doesn’t preserve the order of the elements. The for loop and empty list method preserves the order but is more complex. The list comprehension and dict.fromkeys() methods are more concise and compact, while preserving the order of the elements. The collections.OrderedDict.fromkeys() method is a fast and efficient method for removing duplicates and preserving the order of the elements in Python 3.5 and later versions.

It’s always a good idea to test and compare the performance of different methods for your specific case, and choose the one that is most suitable for your use case.