Sample Integration : Upload and Download to/from GCS to/from SFTP

[shaaland]

Hi All,

This sample in GitHub shows how you can copy files from SFTP to Google Cloud Storage (GCS) and vice ....  For example, if you have a partner that only uses SFTP, but your process uses GCS, you can use Application Integration to help bridge the gap.  Please use the Readme file for instructions.

For convenience, I put both of these flows in one Integration.  The GCS to SFTP flow has two types of triggers. The PubSub Trigger is enabled for a notification to PubSub feature that GCS supports: when you drop a new object in the configured bucket, pubsub is notified, and then the Integration executes!  Event driven Files!  The API Trigger allows you to easily test the flow without having to drop more files into the GCS bucket. You can cut/paste the payload from the logs from the PubSub trigger execution and set it to the default value of the input variable ($PubSubPayload$) so that you can re-test with the same GCS object over and over until you have everything finalized.

You can edit the .json files to replace your project-id, region, bucket-name, pub-sub-topic, sftp-connection-name, and gcs-connection-name and then upload to your Application Integration environment in the GCP console.  Or, you can just upload it and then configure all of these settings in the Connection Tasks, Trigger tasks, and in the default value of the input variables. (search and replace in a text editor may be easier to do in the .json file).  You can also use the IntegrationCLI tool to do the replacements.  Here is what it looks like: 

Please see this community post for some details about how to do binary files 

Please see these docs for how to set up the GCS Notifications: https://cloud.google.com/storage/docs/reporting-changes#command-line 

Enjoy.  Please comment with any questions.

[umkothari]

Hi,

I am able to download and move .csv and .txt file but but integration got failed for .xlsx and .png and other files, Can you please help me with that.

[Madhuvandhini]

Kindly verify if binary mode is set as true while uploading  file to SFTP or GCS for xlsx and png

SFTP : Upload binary file example  can be helpful

GCS : https://cloud.google.com/integration-connectors/docs/connectors/cloudstorage/configure

[avindia]

Great this is what exactly I am looking for . Need to setup file transfer from my GCP platform ( GCS Storage Bucket) to Third Party SFTP Server exposed on Public Internet? Can I use this solution . Please advice and share the relevant link 

[shaaland]

Sorry for the delayed response.  The solution is in the orginal post as a sample.  You can use Application Integration with a few easy steps to set up in GCP Console.  Here is the link to provision Application Integration.