Repository could not be reached, Dataform connected to Gitlab

[Chanan]

Hi

I have implement data transformation pipeline on Google Cloud Composer, by using

1. "DataformCreateCompilationResultOperator" to create comilation on Dataform
2. "DataformCreateWorkFlowInvocationOperator" to create invocation to run models on Dataform 

which process by order 1 >> 2

The error happen when "DataformCreateCompilationResultOperator" try to create compilation on Dataform which connected to Gitlab repository. It raised this error 

google.api_core.exceptions.InvalidArgument: 400 Remote repository 'xxx.git' could not be reached.

After that I add number of retries to 2 (1 min interval) to DAG config, it was  retried and it can be proceed but it not stable, sometimes they need 5 retries.

ref : Google's troubleshooting link

After that I try to find solution on Google's troubleshooting, the google inform that "There is no need to take any action. Unless GitHub or GitLab issues persist, the subsequent scheduled releases can be successful"

The question is, what is the cause of this error and how to solved it, with appropriate action,  the "retry" method is not stable, maybe the error will happen more often in the future.

I'm really troubled by this issue because the customers are really worry

Can you please give me the solution, Thank you very much

[ms4446]

The error message "Remote repository 'xxx.git' could not be reached" indicates that Cloud Composer was unable to connect to the GitLab repository. There are a few possible causes for this error:

• The GitLab server may be experiencing downtime or is temporarily unreachable.
• The firewall settings within your Google Cloud environment or VPC might be blocking access to the GitLab server.
• There could be a misconfiguration in the connection settings within Cloud Composer.

To troubleshoot this error, consider the following steps:

1. Check the GitLab Server Status: Visit the GitLab website or use services like DownDetector (https://downdetector.com/status/gitlab) to check for reported outages. However, remember that DownDetector is based on user reports and might not capture all outages.
2. Verify Network Access: Ensure that the firewall settings in your Google Cloud environment or VPC are configured to allow access to the GitLab server. You can do this by navigating to the VPC Networks page in the Cloud Console and selecting the network that contains your Cloud Composer environment. Then, click the Firewall Rules tab and make sure that there are rules that allow traffic to the GitLab server.
3. Review Connection Settings in Cloud Composer: Navigate to the Cloud Composer UI and go to the Connections page. Ensure that the connection settings for the GitLab repository are correctly configured. The connection settings should include the GitLab server URL, your username, and your password.

If you have tried the above steps and you still encounter the error, consider reaching out to Google Cloud support for further assistance.

Additional Tips:

• Opt for a reliable and regularly maintained GitLab server.
• Regularly review and update firewall settings to ensure uninterrupted access to essential services.
• Always double-check connection settings in Cloud Composer before initiating new pipelines.

[Chanan]

thank you for your advice, now i solved it by change Dataform connection to Gitlab method from HTTPS to SSH and it work properly