Changing cloud composer variables with ci/cd

[Cayo_dias]

I have a Cloud Composer environment running with the 1.20.11 Composer version and 2.3.4 Airflow, but I have some difficulties with changing the airflow variables using the ci/cd pipeline in GitHub actions. I tried to use the command below and I am getting 'ERROR: (gcloud.composer.environments.run) PERMISSION_DENIED: The caller does not have permission' and I can't find the right permission to grant to the GitHub actions service account to run this.

gcloud composer environments run $COMPOSER_ENVIRONMENT --location $LOCATION variables import -- gcsfuse/variables.json

 I don't know if we have some other options to update the variables to use in the scripts as you see below:

internal_project = Variable.get("bq_project")

[ms4446]

To update Airflow variables in a Cloud Composer environment, you need to grant the GitHub Actions service account the Composer Environment Admin role. This role encompasses a wide range of permissions related to Cloud Composer environments, including the ability to run commands in the environment and update variables.

Airflow REST API

Google Cloud recommends using the Airflow REST API to update Airflow variables. The Airflow REST API is available in Airflow 2.3.4, but it is important to ensure that it is enabled and properly secured before using it.

Cloud Functions Airflow Trigger

The term "Cloud Functions Airflow Trigger" is not standard terminology in Google Cloud. However, it is possible to trigger a Cloud Function based on certain events, which can then interact with Airflow to update variables. This would be a custom solution and would require more detailed setup and security considerations.

GitHub Actions YAML

Here is a revised GitHub Actions YAML snippet for updating Airflow variables:

- name: Update Airflow variables
  run: gcloud composer environments run $COMPOSER_ENVIRONMENT --location $LOCATION variables import -- $AIRFLOW_VARIABLES_JSON

Replace the following variables:

• $COMPOSER_ENVIRONMENT: The name of your Cloud Composer environment.
• $LOCATION: The location of your Cloud Composer environment.
• $AIRFLOW_VARIABLES_JSON: The path to the JSON file containing the Airflow variables that you want to update.

Service Account Authentication in GitHub Actions

You need to securely store the service account key as a GitHub secret and use it in the GitHub Actions workflow to authenticate with Google Cloud. Directly embedding service account keys in code or configuration files is a security risk.

Alternative CI/CD Tools

There are a number of third-party CI/CD tools that can be used to update Airflow variables, such as CircleCI and Jenkins. The setup and configuration for updating Airflow variables would differ for each tool. Please consult the documentation for your specific CI/CD tool for more information.

Final Step in GitHub Actions

Once you have made the changes to your GitHub Actions workflow, commit your changes to your GitHub repository and push them to the remote repository. GitHub Actions will then run your workflow and update the Airflow variables in your Cloud Composer environment.

[tmdedge]

Hi, I'm facing the same issue since I upgraded my Cloud composer image to version composer-2.6.0-airflow-2.6.3.

I tried to add the Composer Environment Admin to the service account but the role simply does not exist. There is a similar Composer Administrator role, which does not contain the required permission.

Therefore it does not work. By the way, I wasn't able to find the required permission at all.

[ms4446]

Dealing with Cloud Composer and Airflow, especially after those recent updates, can feel a bit like navigating a maze. Let's simplify things and get you through the permissions puzzle:

The Core Issues:

• Evolving IAM Roles: Changes in Google Cloud's IAM roles for Cloud Composer can lead to confusion. Roles like "Composer Environment Admin" might not exist or function as expected in newer setups.

• Permission Gaps: The "Composer Administrator" role may lack specific permissions required for executing gcloud composer environments run commands, indicating a gap in permissions for effective variable management.

• Documentation Updates: The pace at which Cloud Composer features are updated can result in documentation that lags behind actual changes, complicating the search for current information on required permissions.

Strategic Troubleshooting Steps:

1. Comprehensive Role Review:

   • Inspect all roles assigned to your service account in the IAM & Admin section of the Google Cloud Console, with a focus on roles related to Composer and environment management.

2. Detailed Permissions Analysis:

   • Use gcloud iam roles describe to examine the permissions of roles you've assigned or consider relevant, specifically looking for capabilities related to environment command execution.

3. Log Investigation:

   • Explore detailed logs in Cloud Composer's "Logging" section for error messages that specify the missing permissions, offering clues for resolution.

4. Temporary Broad Permissions:

   • Temporarily increase permissions (e.g., assigning a project-level Editor role) for diagnostic purposes to identify missing permissions. This approach should be used with caution and reversed once the issue is identified.

Additional Considerations:

• Leverage the Airflow REST API: The Airflow REST API provides a more granular and secure option for managing Airflow variables, offering precise control over permissions.

• Craft a Custom IAM Role: Creating a custom IAM role with the exact permissions needed ensures adherence to the principle of least privilege.

• Stay Informed on Version Changes: Regularly review Cloud Composer version updates and their implications for permissions and functionality to maintain compatibility and security.

Reflecting on Security and Workflow Preferences:

• Assess Security Requirements: Align your strategy with your organization's security standards, balancing the need for access against minimizing risk.

• Choosing Between REST API and gcloud: Evaluate which integration method best suits your workflows, considering the REST API for its flexibility and security or gcloud for its simplicity.

Effectively managing the permissions landscape of Cloud Composer and Airflow demands a proactive and informed approach. By thoroughly reviewing roles, permissions, and documentation, and selecting the tools that best meet your needs, you can maintain a secure and efficient environment. Always prioritize security and the principle of least privilege, adapting your strategy as Cloud Composer evolves. For unresolved issues, consider seeking assistance from the Cloud Composer community or platforms like StackOverflow for specific error resolutions.

[petar-popovic]

Hi @ms4446 Does your example change Composer env var or Airflow env var running within composer?
It looks like @Cayo_dias is accessing those variables inside dags.

---

@Cayo_dias wrote:

internal_project = Variable.get("bq_project")

I have the same issue and I want to automate it with github actions but currently I am using Airflow webserver UI to manually upload json containing variables.

[ms4446]

The example provided in the previous responses is indeed for updating Airflow variables within a Cloud Composer environment, not the environment variables of the Composer itself. The Airflow variables are those that you can access within your DAGs using Variable.get("variable_name"), like in your example with internal_project = Variable.get("bq_project").

To automate the updating of these Airflow variables using GitHub Actions, you can follow these steps:

1. Prepare Your JSON File:

• Create a JSON file containing the Airflow variables you want to update.
• Ensure the format complies with Airflow's CLI expectations for variable imports.

2. Store the JSON File:

• Place the JSON file in a location accessible by your GitHub Actions workflow. This could be within the repository or a cloud storage bucket.

3. GitHub Actions Workflow:

• Create a workflow in GitHub Actions to automate the update process.
• Include a step to authenticate with Google Cloud using a service account with the necessary permissions.
• Use the gcloud composer environments run command with the variables import flag to import variables from the JSON file into your Airflow environment.

Example Workflow:

name: Update Airflow Variables

on:
  push:
    branches:
      - main

jobs:
  update-airflow-variables:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@master
        with:
          service_account_key: ${{ secrets.GCP_SA_KEY }}
          project_id: ${{ secrets.GCP_PROJECT }}
      - name: Update Airflow variables
        run: |
          gcloud composer environments run ${{ secrets.COMPOSER_ENVIRONMENT }} --location ${{ secrets.LOCATION }} variables import --path/to/your/variables.json

Remember:

• Replace path/to/your/variables.json with the actual path to your JSON file.
• Set the following secrets in your GitHub repository: GCP_SA_KEY, GCP_PROJECT, COMPOSER_ENVIRONMENT, and LOCATION.

This workflow will trigger on pushes to the main branch and update your Cloud Composer environment's Airflow variables using the Google Cloud CLI.