Data Engineer

Tell us about the replication setup task you worked on.

In which cases is normalization applied, and in which cases denormalization?

How to avoid/eliminate data skew in Spark?

If the query is still slow after adding several indexes - what diagnostic algorithm and what to do?

What is normalization and ER model, and why are they needed?

Tell about the basic concepts of Kafka (consumer group, partitions, topics).

What are the typical reasons for a database query to run slowly?

What non-obvious problems can arise when loading large tables (besides performance)?

Propose a solution for the task of regular incremental loading of a large table with flexible (changeable) update frequency from Postgres to Data Lake.

How are the sorting within the window function and the final ORDER BY sorting in the query related?

How to manage Spark application resources to avoid memory overuse when changing input data volume?

Как работать с партициями через coalesce и repartition?

What Docker images have you had to build, and why?

Are there any other shuffle management mechanisms besides coalesce/repartition?

Какие параметры Spark JDBC использовались для распараллеливания чтения?