Python - How To Find Duplicates in a List in Python

100

This tutorial shows you how to look for duplicates within a list in Python.

Use the set() Function to Find Duplicates in a Python List

Python set() is a function to convert a list into a set. Based on the description of a set and a list, converting a list to a set would mean removing all the duplicates in the list.

However, what we want is to search for the entries within the list that are duplicates. To do that, we still use the set() function, but we only include those with more than one entry in the list, which means they are entries that have duplicates.

Declare a function that looks for duplicates within a list and store them as a set.

def listToSet(listNums): 	set([num for num in listNums if listNums.count(x) > 1]) 

Another approach, if you want to keep the result as a list instead of a set, is to use set() and add() to find duplicates within the list and re-convert it to a list during return.

def list_duplicates(listNums):   once = set()   seenOnce = once.add   twice = set( num for num in listNums if num in once or seenOnce(x) )   return list( twice ) 

This function adds the entry to once if the number appears for the first time, and doesn’t exist within the set once.

Both functions will return the same output, although one is a set, and another is a list. The result will output all the duplicate entries in listNums.

[1,3,5,8] 

Use the iteration_utils Module to Find Duplicate in a Python List

iteration_utils has two functions that can search for duplicates within a pre-defined function: duplicates and unique_everseen.

We’re going to use the same list definition listNums above.

Import duplicates from iteration_utils and use it to return a list of all the duplicates within listNums.

from iteration_utils import duplicates  listNums = [1,1,2,3,3,4,5,5,5,5,6,8,8]  def listDups(listNums):   return list(duplicates(listNums)) 

The output of the above function will return a list of all the duplicate entries within listNums.

[1,1,3,3,5,5,5,5,8,8] 

Although, the list also includes the iterations of the duplicate entries. This means it will return every instance of the duplicated entry.

• This is where unique_everseen comes in. This function processes the list to remove all the duplicate instances that duplicates() returns.

from iteration_utils import duplicates from iteration_utils import unique_everseen  listNums = [1,1,2,3,3,4,5,5,5,5,6,8,8] def listDupsUnique(listNums): 	return list(unique_everseen(duplicates(listNums))) 

This function will then return:

[1,3,5,8] 

In summary, there are 2 easy solutions to look for duplicates within a list in Python. The first is using set() and other utility functions of sets in Python to look for duplicates and store them in another variable. Another one is by the iteration_utils module by using duplicates and unique_everseen, which more or less does the same thing and produces the same output.