How to remove duplicates from a list of elements?

There are several approaches to remove duplicates from a list. The fastest and most Pythonic way for hashable types is to convert the list to a set, and then back to a list.

# Example list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Convert to set to remove duplicates
unique_elements_set = set(my_list)

# Convert back to list
unique_list = list(unique_elements_set)

# Print the unique list
# print(unique_list) # [1, 2, 3, 4, 5] (order not guaranteed)

If the order of elements is important and maintaining the original order is required, you can use a loop or dictionaries (for Python 3.7+):

# Example list with duplicates
my_list_ordered = [1, 2, 4, 2, 3, 4, 5]

# Option 1: Using a loop and a helper list
unique_list_ordered = []
seen = set()
for item in my_list_ordered:
    if item not in seen:
        unique_list_ordered.append(item)
        seen.add(item)

# print(unique_list_ordered) # [1, 2, 4, 3, 5]

# Option 2: Using a dictionary (Python 3.7+ preserves order)
unique_list_dict = list(dict.fromkeys(my_list_ordered))

# print(unique_list_dict) # [1, 2, 4, 3, 5]

For unhashable types (e.g., lists within a list), you need to iterate and store seen elements in a hashable representation (like a tuple or JSON string).

# Example list with unhashable elements
list_of_lists = [[1, 2], [3, 4], [1, 2], [5, 6]]

# Use a loop and convert sublists to tuples for hashing
unique_list_of_lists = []
seen_tuples = set()
for sublist in list_of_lists:
    sublist_tuple = tuple(sublist) # Convert list to tuple for hashing
    if sublist_tuple not in seen_tuples:
        unique_list_of_lists.append(sublist)
        seen_tuples.add(sublist_tuple)

# print(unique_list_of_lists) # [[1, 2], [3, 4], [5, 6]]

Comparison of methods:

Method	Preserves order	Performance (for hashable)	Suitable for unhashable
list(set(my_list))	No	High	No
Loop with set and append	Yes	Medium	No (requires conversion)
list(dict.fromkeys(...))	Yes (Python 3.7+)	High	No (requires conversion)

The choice of method depends on the requirements for order preservation and data type in the list. In most cases, converting through a set is sufficient for hashable data.