Sobes.tech
Middle

Have you worked with Great Expectations and dbt tests for data quality verification?

sobes.tech AI

Answer from AI

Yes, I have worked with Great Expectations and dbt tests for data quality verification. Great Expectations allows creating declarative data checks (expectations) that can be run as part of a pipeline to ensure data meets specified requirements — for example, type checks, value ranges, uniqueness, etc.

Dbt tests are built-in or custom tests that verify data integrity and correctness at the dbt model level. For instance, you can check that a column contains no NULL values or that values are unique.

An example of using Great Expectations to verify that there are no negative values in the "age" column:

import great_expectations as ge

df = ge.read_csv('data.csv')
df.expect_column_values_to_be_between('age', min_value=0)

In dbt, you can add a test in schema.yml:

models:
  - name: users
    columns:
      - name: id
        tests:
          - unique
          - not_null
      - name: email
        tests:
          - not_null

Both tools complement each other well for ensuring data quality at different stages of the ETL/ELT processes.