Data Engineer
What databases have you worked with? How is MongoDB structured and how does it scale?
How do you determine that a record already exists and does not need to be recorded again?
Лига Цифровой ЭкономикиNULL plus 5 — what will it be?
Is it possible to read data in parallel from a single Parquet file in Spark, or only with one core in one task?
What is an RDD in Apache Spark and how does it differ from other Spark abstractions?
Have you performed statistical checks?
ОстровокIs it possible to create and apply a decorator without the @ syntax?
If the DAG schedule is @daily, what time does it actually run?
ОстровокWhat are decorators in Python?
Why are 'folders'/prefixes in S3 needed, besides convenience?
Лига Цифровой ЭкономикиHow was the parallel extraction of a large table from PostgreSQL to Spark technically implemented in the [context] case?
How to distribute storefront data unevenly across three shards: 50% on the first and 25% on each of the remaining two? How to implement this?
How to deal with data inconsistency?
ОстровокHow were tables created on the cluster, and how was the problem of missing replicas on a shard due to ZooKeeper solved?
How do you monitor data quality?
ОстровокWhat are the stages of a transaction?
ОстровокWhy did you decide to move ranking to a separate CTE? Why couldn't it be used directly in the first CTE?
Как работать с партициями через coalesce и repartition?
How to properly open and close files in Python using a context manager?
Are you familiar with ReplicatedQueue? Have you worked with it?