Solved: BigQuery Network Requirements

hyf_je · 12-18-2023 10:13 AM

Disclaimer: I'm a humble system administrator, not a software dev/architect or data engineer.

Looking into an issue re: BigQuery and we suspect that network issues (firewall, packet forwarding, etc) could be part of the problem.

The problem is - I can't find any documentation that spells out what network endpoints/protocols are required in order for communications (assume APIs, machine-to-machine) to work.

Google must have this documented somewhere, but I came up short. Hoping someone can point me in the right direction.

kolban

Howdy. Could you elaborate on the symptoms/nature of the issue? BigQuery is a Google Cloud service. EVERY Google Cloud service exposes itself as set of endpoints that can be called via REST and gRPC. BigQuery is a managed OLAP database. It's existence is managed by Google.

Every Google Cloud service can be reached at the googleapis.com domain. This is likely the documentation you are looking for. You probably came up empty trying to find BigQuery endpoint/protocol information as the "general" story is that ALL Google APIs have the same story and BigQuery is merely (in this instance) an example of a Google service that is exposed through that common story. For BigQuery specifically, see here. There we find that BigQuery's endpoint is bigquery.googleapis.com and is being accessed via HTTPS (port 443). If you are considering network issues, contemplate the path from where you are making BigQuery API calls to that endpoint. If you are invoking BigQuery from WITHIN the Google Cloud environment (eg. Compute Engines or GKE) then your Google Cloud environment admins may have configured restrictions in the Google Cloud VPC.

View solution in original post

kolban

Howdy. Could you elaborate on the symptoms/nature of the issue? BigQuery is a Google Cloud service. EVERY Google Cloud service exposes itself as set of endpoints that can be called via REST and gRPC. BigQuery is a managed OLAP database. It's existence is managed by Google.

Every Google Cloud service can be reached at the googleapis.com domain. This is likely the documentation you are looking for. You probably came up empty trying to find BigQuery endpoint/protocol information as the "general" story is that ALL Google APIs have the same story and BigQuery is merely (in this instance) an example of a Google service that is exposed through that common story. For BigQuery specifically, see here. There we find that BigQuery's endpoint is bigquery.googleapis.com and is being accessed via HTTPS (port 443). If you are considering network issues, contemplate the path from where you are making BigQuery API calls to that endpoint. If you are invoking BigQuery from WITHIN the Google Cloud environment (eg. Compute Engines or GKE) then your Google Cloud environment admins may have configured restrictions in the Google Cloud VPC.

hyf_je

The second link is what I was after, thanks.