Moving structured data directly to BigQuery skipping GCS

NJ
New Member

Data : Sales table, Customer table, products table....20 such tables, TB's of data

Relational and Structured Data

I came across this case study.

End Goal: Query the data, create ADS for ML model

Questions:

What are the benefits of ingesting the tables to Google Cloud Storage first and than moving those to Bigquery ?

Can I directly move my data to Bigquery skipping the Google Cloud Storage? 

0 2 259
2 REPLIES 2

You can import directly though I strongly recommend against it for the amount of data you're dealing with.

The advantages of staging via GCS are:

  1. GCS connections will be more stable than anything coming from outside Google Cloud
  2. Importing from GCS is a native solution, see https://cloud.google.com/bigquery/docs/batch-loading-data 
  3. If you need to clear the tables for any reason and reimport it'll be much faster and more convenient

The disadvantage is GCS will incur charges until you delete the import files.

If the source data is inside Google Cloud then a direct import would be more realistic, but I would still use GCS to stage the import data.