How can we test EA on data quality?

Question

Ed · Accepted Answer

There's a few different types of tests to consider:Quality Assurance (QA) - tests you run before you deploy code. Can test for problems you've never seen: numeric overflows, encodings, nulls, etc. This is simple to do with a python-based transformation process, but very difficult for SQL-based transformation processes.Quality Control (QC) - tests data after you've ingested it. Can test for uniqueness, nulls, ranges, business rules, etc. I've built my own frameworks for this, but am currently using DBT's - which is pretty good. Others also exist.Anomaly-Detection - tests for abnormal volumes and values in data, most typically after you've ingested it. These are available open-source and commercial, and frankly isn't difficult to build yourself.All three of these are part of a good quality solution. Given that your solution is currently mostly SQL I would initially focus on QC, and probably consider dbt for that. Next I'd consider anomaly-detection. Finally, I'd tackle QA. QA will be tough, given SQL transforms with a lot of tables

How can we test EA on data quality?

1 Answer

1 Answer

How can we test EA on data quality?

1 Answer

1 Answer

Welcome back

Create Account

Reset password