How were duplicates filtered out? Couldn't duplicates be identified by ID? Why enrich data from Redis?

Duplicates could be filtered out not only by ID, because the ID may not be unique or may be missing in some cases, especially when data comes from different sources. Therefore, additional criteria were used to identify duplicates, such as comparing key fields or hashing the content.

Enriching data from Redis was done to improve the quality and completeness of information: Redis is often used as a fast cache or session store, where additional metadata not stored in the primary source can be obtained. This helps avoid repeated queries to slow databases and speeds up processing, as well as ensuring data consistency and relevance when merging information from different systems.