See GCS and BQ Storage Sizes Across Multiple projects

[Dovile]

I need to monitor general GCP statistics across multiple projects. Like

1. Total GCS Storage
2. Total BQ Storage
3. Total BQ Compute

What are a ways of getting this data? 

• UI Option? In-build dashboard? Billing Reports are at Service or SKU level which is too high level or to low level
• Billing Export is very detail and requires building custom query / dashboard.

So how you monitor things across projects at summary level?

[ms4446]

Monitoring general Google Cloud statistics across multiple projects, such as total GCS storage, BigQuery storage, and BQ compute, can be challenging due to the granularity of data provided by Google Cloud native tools. However, there are several approaches you can take to get a summary-level overview across projects:

1. Cloud Monitoring Metrics Scopes:

   • Approach: Create a dedicated "monitoring" project and add your other projects as monitored projects. Access metrics across projects from this central monitoring project.
   • Pros:
     • Integrated into the Google Cloud.
     • Ability to create custom dashboards and alerts.
   • Cons:
     • Requires setting up and managing an additional project.
     • Necessary permissions must be set up in each monitored project.

2. Cloud Asset Inventory (CAI):

   • Approach: Use CAI for a comprehensive view of your assets across projects, and export this data to BigQuery for custom SQL queries.
   • Pros:
     • Centralized asset view.
     • Flexible querying with BigQuery.
   • Cons:
     • More setup required for exporting and querying.
     • Additional costs for BigQuery usage.

3. Custom Scripting with GCP APIs:

   • Approach: Utilize GCP APIs to develop scripts for aggregating data across projects, and output the results to a database or visualization tool.
   • Pros:
     • Highly customizable.
     • Integration with other tools.
   • Cons:
     • Development and maintenance effort required.
     • Need to ensure scalability and efficient API usage.

4. Third-party Monitoring Solutions:

   • Approach: Use specialized tools like Datadog or New Relic for cross-project resource usage views.
   • Pros:
     • Comprehensive dashboards and advanced features.
     • Simplified setup and management.
   • Cons:
     • Additional costs.
     • May offer cost optimization features.

5. Billing Reports and BigQuery:

   • Approach: Export billing data to BigQuery and create custom queries for detailed analysis.
   • Pros:
     • Detailed cost monitoring and resource usage analysis.
   • Cons:
     • Requires SQL knowledge and initial setup.

6. Data Studio for Visualization:

   • Approach: Use Looker Data Studio to create visual dashboards from BigQuery or other data sources.
   • Pros:
     • User-friendly visualizations.
   • Cons:
     • Requires setup and design of dashboards.

Choosing the Right Method:

• Ease of Use: Cloud Monitoring Metrics Scopes offer a straightforward solution.
• Granular Control: Custom scripting with GCP APIs provides flexibility in metrics and aggregation.
• Cost: Cloud Monitoring Metrics Scopes and CAI are generally cost-effective. Third-party tools may add costs but can also offer cost optimization.
• Advanced Features: Third-party solutions are ideal for robust visualizations and integrations.