This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Hello Google Cloud Community,I am working on migrating an app from AWS
to GCP.The main objective is to design a real-time dashboard system
intended to operate within a latency window of 7-15 seconds from data
event (Kafka) to visualization (custom ap...
Hello,I am seeking guidance on an issue I've encountered while working
with Google BigQuery and Google Cloud Storage (GCS).Overview:I've set up
an external table in BigQuery, which sources data from a Parquet file
stored in GCS. The specific GCS path...
Hello Community,I am currently working on a model, PURE JS, that
iterates through a configuration to publish incremental tables. My
objective is to introduce a variable that captures partitions for
efficient partition pruning but only when incrementa...
Hello everyone,My team is grappling with the challenge of scaling our
Dataform code both robustly and as automatically as possible. We manage
an extensive number of tables (over 300) which can be grouped into
common categories, and we apply nearly id...
Hello everyone,I am currently facing a situation where I am trying to
implement a custom MERGE (INCREMENTAL TABLE) with DataForm. Here, I have
the primary keys along with two metadata columns: LOAD_BATCH and
DRIVER_ACTION_CODE.My objective is to impl...
Hello @ms4446 ,First and foremost, I would like to extend my heartfelt
gratitude for your prompt response yesterday; it was incredibly helpful.
I have a few clarifications and questions to further understand some
concepts.Regarding DataForm, at least...
Thank you @ms4446 for your fast and detailed response regarding the
use of Dataform, particularly in the context of JavaScript models and
incremental tables. Your step-by-step guide, from partition pruning to
differentiating operations, is immensel...
Hello, here's the basic structure for invoking DATAFORM from airflow :
from datetime import datetime, timedelta from airflow import DAG from
google.cloud.dataform_v1beta1 import WorkflowInvocation from
airflow.providers.google.cloud.sensors.dataform <!-- --><!-- -->...
