Data Engineer

How familiar are you with Linux/Docker?

What is shuffle in Spark and why is it needed?

How can shuffle be managed in Spark (partitioning, broadcast join, etc.)?

What does the Iceberg format bring compared to regular Parquet?

Are you familiar with CTE (common table expressions)? Provide an example of its use.

How to find a substring in a specific column of a log file in Linux (e.g., date in the third column)?

What is the concept of statistics in the context of DBMS and query optimization?

[name] shared about his experience working with various DBMSs, how deep he had to understand optimization and internals.

Detail the algorithm for capturing an increment from a source so that nothing is lost when the load frequency changes.

How to organize reading the same message from Kafka by multiple consumers (fan-out)?

How to solve the problem when a reading process might see an intermediate (inconsistent) state of a table during multi-stage ETL (delete+insert) in Iceberg?

What is a "dead tuple"?

How are Spark partitions related to table partitions (e.g., in Iceberg)?

How to compare the original ("live") and target tables if the source is constantly changing?

Tell us about best practices for using indexes - when and how often to use them.

What is Spark JDBC and how to efficiently load a large table through it?

How to detect data skew in Spark?

What is a transaction log (WAL) in a DBMS and why is it needed?

Has the method of comparing hashes of records been used to calculate the delta between the source and target tables?

Why are cache and persist needed in Spark?