AWS Glue, Antenna va Spark bilan ish tajribangiz qanday?

Men AWS Glue bilan serversiz ETL vazifalarida ishladim, PySpark yordamida. Glue triggerlarini S3 voqealariga sozladim. AWS Data Catalogdan foydalanardim.

Antenna bilan tanish emasman.

Asosan, katta ma'lumotlarni qayta ishlash uchun PySpark orqali Apache Spark bilan ishladim. Spark SQL yordamida strukturali so'rovlar va RDD API yordamida pastki darajadagi operatsiyalarni amalga oshirdim. Sparkni joylashtirish bo'yicha tajriba: mahalliy klasterlar, YARN va Kubernetesda Spark.

AWS Glue uchun oddiy ETL skriptining misoli:

# glue_etl_script.py
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# S3 dan ma'lumotlarni o'qish
datasource0 = glueContext.create_dynamic_frame.from_options(
    format_options={"multiline": False},
    connection_type="s3",
    format="json",
    connection_options={"paths": ["s3://my-source-bucket/input_data/"], "recurse": True},
    transformation_ctx="datasource0"
)

# Ma'lumotlarni transformatsiya qilish (misol: ustunlarni tanlash va nomlarini o'zgartirish)
applymapping1 = ApplyMapping.apply(
    frame=datasource0,
    mappings=[
        ("id", "long", "user_id", "long"),
        ("name", "string", "user_name", "string"),
        ("value", "double", "amount", "double")
    ],
    transformation_ctx="applymapping1"
)

# Transformatsiya qilingan ma'lumotlarni S3 ga yozish
datasink2 = glueContext.write_dynamic_frame.from_options(
    frame=applymapping1,
    connection_type="s3",
    format="parquet",
    connection_options={"path": "s3://my-target-bucket/output_data/", "partitionKeys": []},
    transformation_ctx="datasink2"
)

job.commit()