Is ACID maintained in Greenplum?
Data Engineer
Data Vault is a convenient thing, but there are quite a few objects. Can you explain the difference between hubs, links, and satellites?
What processes are triggered when deleting or changing records in ClickHouse?
You work with Data Science and big data processing — is there a replacement in this library written in Rust, multithreaded, unlike Pandas, very fast. Do you know such?
How do you break down a query into stages? What do we do for that?
How does CTE differ from a subquery?
Parquet, you uploaded data — what technologies were used to receive and upload the data, and what mechanisms were employed?
With dependencies between jobs and DAGs — did you use sensors to run several jobs one after another?
You uploaded data from Spark to S3 in Parquet format, and then from Parquet to Greenplum — how does the loading process occur?
Have you worked with Git? What did you store in Git?
Did you usually design the database schema manually in the database, or did you store scripts somewhere, or did you use Liquibase at all, what was the process?
How was data from an external table loaded into target tables?
Did you write Airflow DAGs and jobs manually or using templates?
Have you heard of Iceberg working with streaming? It's a trashy file storage.
If CTE is used only once in a query, does it make sense to use it?
This is real-time, how to implement some kind of connection between Kafka and ClickHouse Spark?
Have you worked directly with Spark? What is its disadvantage?
We will parse JSON — who will parse JSON? As far as I know, ClickHouse does not have functions for this.
We need a pipeline engine — a mechanism that allows very fast stream creation by providing contracts and parameters as input. How would you implement this at a high level?
Iceberg is an engine that allows you to turn S3 into a database, and it even complies with ACID. What is its downside?