LinkedIn
Twitter
I'm using Terraform to create a DataStream that connects to a private MySQL instance in Cloud SQL. However, I'm encountering connection errors despite seemingly correct configurations.
My Terraform provisions the following resources:
- Google Cloud Networking resources (likely VPC and subnets)
- Cloud SQL Private MySQL instance
- Google DataStream Private Connection
- Compute Engine VM instance
I don't understand why it is not able connect MySQL server:
Solved
3 ACCEPTED SOLUTIONS
I have created resources using Terraform for DataStream to MySQL private connectivity. Previously, I created a source connection profile and used an IP address instead of a hostname with same configuration. Here is an example:
But now I want to create it using a hostname instead of an IP address. I have written Terraform code for that; however, I'm facing issues while creating it using a hostname.
Could you please help me with this issue?
Error:
Error: Error waiting for Updating ConnectionProfile: {"@type":"type.googleapis.com/google.rpc.ErrorInfo","domain":"datastream.googleapis.com","metadata":{"message":"We can't connect to the data source using the hostname that you provided. Make sure that the hostname is correct.","originalMessage":"(2003, \"Can't connect to MySQL server on '192.168.59.52' ([Errno 111] Connection refused)\")","time":"2024-04-11T12:42:25.686357Z","uuid":"0ff7fc6f-cd38-42f0-b76f-498e0e3bbfa1"},"reason":"WRONG_HOSTNAME"}
│ {"code":"VALIDATE_CONNECTIVITY","description":"Validates that Datastream can connect to the source database.","message":[{"code":"WRONG_HOSTNAME","level":"ERROR","message":"We can't connect to the data source using the hostname that you provided. Make sure that the hostname is correct.","metadata":{"original_error":"(2003, \"Can't connect to MySQL server on '192.168.59.52' ([Errno 111] Connection refused)\")"}}],"state":"FAILED"}
│ with google_datastream_connection_profile.source_connection_profile,
│ on main.tf line 143, in resource "google_datastream_connection_profile" "source_connection_profile":
│ 143: resource "google_datastream_connection_profile" "source_connection_profile" {
Terraform :
resource "google_compute_instance" "custom_hostname_instance" {
name = "custom-hostname-instance-name"
project = var.project
machine_type = var.proxy_machine_type
zone = var.zone
# Set a custom hostname below
hostname = "xxxxxx.com"
boot_disk {
initialize_params {
image = "debian-cloud/debian-11"
}
}
network_interface {
network = google_compute_network.network.name
subnetwork = google_compute_subnetwork.private-1.self_link
access_config {
// Ephemeral public IP
}
}
resource "google_datastream_connection_profile" "source_connection_profile" {
display_name = "Source connection profile"
location = var.region
connection_profile_id = "source-profile"
project = var.project
mysql_profile {
hostname = google_compute_instance.custom_hostname_instance.hostname
username = var.user_name
password = var.user_password
}
private_connectivity {
private_connection = google_datastream_private_connection.private.id
}
# depends_on = [google_datastream_private_connection.private]
}
I have configured multiple instances in Datastream and created multiple DataStream's for each one.
In my case, I already have multiple datasets and tables in BigQuery.
Now, I want to replicate newly added data into BigQuery in the same dataset, but I am not able to replicate it. I am facing the following issues:
Is datastream not able to replicate data in same dataset in same table or it is creating new dataset and table for each?
resource "google_datastream_connection_profile" "source_connection_profile1" {
display_name = "mysql-bq-source1"
project = var.gcp_project_id
location = var.gcp_region
connection_profile_id = "source-profile1"
mysql_profile {
hostname = google_compute_instance.private.network_interface.0.network_ip
username = data.google_secret_manager_secret_version.app_user_secret.secret_data
password = data.google_secret_manager_secret_version.app_pwd_secret.secret_data
}
private_connectivity {
private_connection = google_datastream_private_connection.private.id
}
depends_on = [
google_compute_address.static,
data.google_secret_manager_secret_version.app_pwd_secret,
data.google_secret_manager_secret_version.app_user_secret
]
}
resource "google_datastream_connection_profile" "destination_connection_profile1" {
display_name = "mysql-bq-destination1"
location = var.gcp_region
connection_profile_id = "destination-profile1"
project = var.gcp_project_id
bigquery_profile {}
}
# Create Datastream Stream
resource "google_datastream_stream" "mysql_to_bigquery1" {
display_name = "mysql_to_bigquery1"
location = var.gcp_region
project = var.gcp_project_id
stream_id = "mysql_to_bigquery1"
desired_state = "NOT_STARTED"
source_config {
source_connection_profile = google_datastream_connection_profile.source_connection_profile1.id
mysql_source_config {
include_objects {
mysql_databases {
database = "ss_v2_db"
mysql_tables {
table = "transaction_monitors"
}
}
}
}
}
destination_config {
destination_connection_profile = google_datastream_connection_profile.destination_connection_profile1.id
bigquery_destination_config {
source_hierarchy_datasets {
dataset_template {
location = var.gcp_region
# kms_key_name = "bigquery-kms-name"
}
}
}
}
backfill_none {
}
}
