Have you worked with Great Expectations and dbt tests for data quality verification?
sobes.tech AI
Answer from AI
Yes, I have worked with Great Expectations and dbt tests for data quality verification. Great Expectations allows creating declarative data checks (expectations) that can be run as part of a pipeline to ensure data meets specified requirements — for example, type checks, value ranges, uniqueness, etc.
Dbt tests are built-in or custom tests that verify data integrity and correctness at the dbt model level. For instance, you can check that a column contains no NULL values or that values are unique.
An example of using Great Expectations to verify that there are no negative values in the "age" column:
import great_expectations as ge
df = ge.read_csv('data.csv')
df.expect_column_values_to_be_between('age', min_value=0)
In dbt, you can add a test in schema.yml:
models:
- name: users
columns:
- name: id
tests:
- unique
- not_null
- name: email
tests:
- not_null
Both tools complement each other well for ensuring data quality at different stages of the ETL/ELT processes.