Why does bool([""]) return True?
Middle
167
Data Engineer
671
0
What are the disadvantages of UDFs in Spark and what is their downside?
Middle
167
How to deal with data inconsistency?
Middle
166
Task #3 Find clients with a gap between their orders > 60 days. For these clients, output the maximum gap (gap_days). Orders table: | order_id | customer_id | order_dt | |----------|-------------|------------| | 1 | 1 | [phone] | | 2 | 1 | [phone] | | 3 | 1 | [phone] | | 4 | 1 | [phone] | | 5 | 2 | [phone] | | 6 | 2 | [phone] | | 7 | 3 | [phone] | Final result: | customer_id | gap_days | |-------------|----------| | 1 | 123 | | 2 | 120 |
Middle
166
What is the difference between repartition and coalesce in Spark?
Middle
166
What do you know about spill in Spark?
Middle
153
How would you describe in code a check that a value significantly differs from a normal (expected) distribution?
Middle
153
Why can data be lost? Provide some reasons.
Middle
153
Have you worked with context managers in Python? What are they and what are they used for?
Middle
153
If incorrect data has been loaded into the showcase or intermediate table, how can you prevent it from entering higher pipelines?
Middle
151
What methods do you know for transferring data between Airflow tasks?
Middle
149
What does the depends_on_past parameter do in Airflow?
Middle
148
How do you monitor data quality?
Middle
145
If the DAG schedule is @daily, what time does it actually run?
Middle
143
Is a dictionary in Python a mutable or immutable type? What are the requirements for its keys and values?
Middle
143
Have you performed statistical checks?
Middle
135
How to run a second Airflow DAG immediately after the first one successfully completes?
Middle
133
What is the algorithmic complexity of searching in a dictionary in the worst case?
Middle
132