In my project I'm using two databases, one called transformations where all data is cleaned/modelled and mart where data is presented to the visualization layer. Once data has been modelled and cleaned it is loaded to the mart database and I have defined assertions for all tables i the mart database. All of this is handled by dataform.
It is important to me that the data users are able to easily take advantage of the automatically generated assertion views in the mart database. But i realized that even if i define assertions on tables in the mart database, the views are materialized in the transformations database. I assume this is due to the fact that I have defined transformations as my default database in the dataform.json file.
I have tried the following in order to solve this issue but without success.
- Create tables in mart that are filled with data from the automatically generated assertions views.
Issue: When a assertion fails, the table is sqlx file defining the table is not run. - Create views in mart that read from the automatically generated assertions views.
Issue: In order to for the data users to be able to read from these views thay also must have reader access in the transformation database and that is not acceptable for me. - Define mart as my default database.
Issue: This lead to compilation error for all my sqlx files defining tables in the mart database saying "Cannot read properties of undefined (reading 'get')". I have not been able to solve this error nor understand it yet.
I have been able to solve this for my custom assertions by assigning a database in the config block. Ideally I'd like to be able to assign a default database for all my built-in assertions as well.
LinkedIn
Twitter
