Data Engineer

Umuman olganda, Data Vault sizga nima uchun kerak edi?

Ko'rsatilgan ekranda ikkita ma'lumotlar tuzilmasi mavjud; qaysi biri bo'yicha 2 qiymatini qidirish tezroq bo'ladi va nima uchun?

ClickHouse'da Table Engine Join bilan ishlash tajribangiz bormi?

PostgreSQL'dan ClickHouse'ga bir terabayt ma'lumot o'tkazish uchun konfiguratsiyani qanday baholash mumkin?

Apache Spark'da RDD nima va u Spark ning boshqa abstraktsiyalaridan qanday farq qiladi?

Qaysi SQL operatorlari guruhlarini bilasiz? Ular nimaga tegishli?

Middle

СБЕР

from interview.utils import get_clickhouse_client from airflow import DAG from airflow.operators.python import PythonOperator from airflow.sensors.external_task import ExternalTaskSensor from datetime import datetime import pandas as pd import clickhouse_driver import os CLICKHOUSE_CLIENT = get_clickhouse_client() default_args = { "start_date": datetime(2024, 1, 1) } with DAG( dag_id="datamarts.daily_revenue_per_country", default_args=default_args, schedule_interval="@daily", catchup=False ) as dag: transactions_sensor = S3KeySensor( task_id="transactions_sensor", bucket_key="data/transactions_{}.csv".format(datetime.now().strftime("%Y-%m-%d")), bucket_name="my-bucket", aws_conn_id="aws_default", timeout=600, poke_interval=30, mode="poke" ) def extract_from_s3(**kwargs): df = pd.read_csv("s3://my-bucket/data/transactions_{}.csv".format(datetime.now().strftime("%Y-%m-%d"))) kwargs["ti"].xcom_push(key="df", value=df.to_dict()) def load_to_raw_table(**kwargs): df = pd.DataFrame(kwargs["ti"].xcom_pull(task_ids="extract", key="df")) rows = [tuple(r) for r in df[["transaction_id", "user_id", "amount", "created_at"]].to_numpy()] CLICKHOUSE_CLIENT.execute( ... )

Natijaviy natija: | customer_id | gap_days | |-------------|----------| | 1 | 123 | | 2 | 120 | with cte as( select order_id, customer_id, order_dt, lag(order_dt) over(partition by customer_id order by order_dt) as prev_order_ft, datediff(day, prev_order_ft, order_dt) as gap_days from Orders ) select customer_id, max(gap_days) as gap_days from cte where gap_days > 60 group by customer_id

Middle

Островок

Python'da qanday tiplash turi qo'llaniladi: statik yoki dinamik?

Middle

МТС

Kafka'dan bir xil xabarni bir nechta turli iste'molchilar tomonidan o'qishni qanday tashkil qilish mumkin (fan-out)?

Qaysi jamoa va qaysi rahbarlik ostida ishlash sizga eng qulay?

To'rt kishilik jamoada roling nima edi va 3-4 yildan keyin o'z karerangni qanday ko'rasan, qanday maqsadlar qo'yasiz?

Middle

МВидео

Kafka'dan ClickHouse'ga real vaqt hisobotlari uchun streaming orqali ma'lumotlarni qanday yuklash mumkin?

Middle

МВидео

Spring Boot 2 dan Spring Boot 3 ga migratsiya qilish tajribangiz haqida gapiring

JSON ni qanday papkaga yozish (Airflow DAG kontekstida)

Airflow'da qanday qilib ko'p o'xshash vazifalarni parallel ravishda yaratish mumkin (masalan, ko'p o'xshash fayllarni yuklash)?

Qaysi hollarda normalizatsiya qo'llaniladi va qaysi hollarda denormalizatsiya?

Parquet ga ma'lumotlarni yuklash uchun qanday texnologiyalar ishlatildi va ma'lumotlarni olish va yuklash uchun qanday mexanizmlar ishlatildi?

Middle

МВидео

Semantik versiyalash nima? Qachon major, minor, patch ni oshirish kerak?

Oynash funktsiyalar bilan ishlagansizmi? Qaysi turdagi oynali funktsiyalarni bilasiz?

Middle

Aston