I am using argument settings to populate as many settings in my pipeline as possible. The pipeline consumes files from GCS, processes with Wrangler, and writes out to GCS and BigQuery.
I am able to use the macros everywhere EXCEPT in the source GCS. If I try to set a macro for "format" or "schema", I get a null pointer exception error. It is no problem in the sink GCS steps. It definitely appears to be a bug.
Here is the relevant settings in my arguments JSON:
It seems that it responds to publicly accessible objects. Can try setting your GCS in to public? For more information you can view the documentation here:
I created a new bucket and made it public. I uploaded my arg settings files and pointed the data pipeline arg settings plugin to the new location. I still get the same results.
Understand that the arg settings were working fine when the bucket wasn't public. The problem is only when attempting to use arguments/macros on the GCS source, and only on the "format" and "schema".
User | Count |
---|---|
4 | |
1 | |
1 | |
1 | |
1 |