Skip to main content

8 posts tagged with "google cloud"

View All Tags

Google Provider Update - August 2026

· 3 min read
Jeffrey Aven
Technologist and Cloud Consultant

We've released an update to the StackQL Google provider, regenerated from the latest Google API discovery documents. The google provider now covers 187 services, 2,183 resources and over 9,100 operations - up from 179 services, 1,966 resources and 8,423 operations in the previous release. The companion providers in the google family (googleworkspace, googleadmin and firebase) were regenerated in the same pass.

New Services​

Eleven services are new in this release:

ServiceDescription
agentregistryCentralized catalog to store, discover and govern MCP servers, tools and AI agents within Google Cloud
agentidentityIdentities and authorization for AI agents - auth providers, authorizations and access summaries
agentidentitycredentialsShort-lived credential issuance for agent identities
cesGemini Enterprise for Customer Experience - agents, apps, deployments, conversations, guardrails and tool schemas
hypercomputeclusterCluster Director - deploy, manage and monitor AI/ML and HPC clusters
databasecenterOrganization-wide database fleet health monitoring across projects and folders
metastoreDataproc Metastore - services, backups, federations, metadata imports and migrations
threatintelligenceGoogle Threat Intelligence - alerts, findings, documents and configurations
cloudnumberregistryIP address management - realms, registry books, custom and discovered ranges
developerknowledgeProgrammatic access to Google developer documentation
healthGoogle Health API (v4) - health and fitness metrics, data points, devices and subscriptions

Expanded Coverage in Existing Services​

  • compute - the largest expansion in this release, 73 new resources: capacity planning (future_reservations, reservation_slots, reservation_blocks and reservation_sub_blocks versions), instant_snapshot_groups and regional snapshot resources, cross_site_networks and wire_groups, organization_security_policies, network_firewall_policies, rollouts and rollout_plans, plus per-resource IAM policy resources across addresses, firewalls, health checks, routes, target proxies and more
  • aiplatform (Vertex AI) - agent engine build-out: agents, memory_banks, sandbox_environments (with templates and snapshots), online_evaluators, evaluation_metrics, responses and semantic governance policy resources
  • oracledatabase - GoldenGate integration: deployments, deployment environments/types/versions, connections and connection assignments, plus autonomous database refreshable clones
  • dataplex - universal catalog governance: data_domains, data_products, data_assets, metadata_feeds and change_requests
  • redis - token-based auth: token_auth_users, auth_tokens and acl_policies
  • networksecurity - dns_threat_detectors and Secure Access Connect realms and attachments
  • cloudkms - single_tenant_hsm_instances, key deletion proposals and retired_resources
  • observability - log analytics buckets, datasets, links, views and scope settings

Another 50+ services picked up incremental resources and operations, including discoveryengine, contactcenterinsights, netapp, artifactregistry, assuredworkloads and dataproc.

Removed Services​

Google has retired several APIs since the last release, and they are removed from the provider accordingly:

  • datalabeling - AI Platform Data Labeling was shut down
  • integrations - Application Integration no longer publishes a discovery document
  • lifesciences - Cloud Life Sciences was shut down
  • dataplex Explore resources (content, environments, sessions) were removed upstream

The duplicate *_aggregated helper resources in compute were also consolidated - cross-zone queries are served by the primary resources, so a zone-less SELECT fans out across all zones:

SELECT name, status, machineType, zone
FROM google.compute.instances
WHERE project = 'my-project';

Example: Open Firewall Audit​

Ingress rules open to the entire internet, and what they allow:

SELECT
  name,
  direction,
  sourceRanges,
  allowed
FROM google.compute.firewalls
WHERE project = 'my-project'
AND sourceRanges LIKE '%0.0.0.0/0%';

Get Started​

Pull the latest provider from the public registry:

stackql registry pull google;

Authenticate with a service account key in the GOOGLE_CREDENTIALS environment variable (or interactively via gcloud auth login), then explore:

SELECT
  json_extract(config, '$.name') AS api,
  state
FROM google.serviceusage.services
WHERE parent = 'my-project'
AND parentType = 'projects'
AND filter = 'state:ENABLED';

Provider docs, including getting-started queries for each provider in the family, are at google-provider.stackql.io, googleworkspace-provider.stackql.io, googleadmin-provider.stackql.io and firebase-provider.stackql.io. Visit us on GitHub and let us know how you're using it.

Google Provider Update - December 2025

· 3 min read
Jeffrey Aven
Technologist and Cloud Consultant

We've released a major update to the StackQL Google provider with a new service, enhanced AI/ML capabilities, and improvements across 177 service files.

New Service: Speech-to-Text v2​

The speechv2 service brings Cloud Speech-to-Text API v2 to StackQL with 6 resources:

ResourceDescription
recognizersManage speech recognition configurations with create, list, get, patch, delete, undelete, recognize, and batch_recognize methods
custom_classesCreate custom vocabulary classes for improved recognition accuracy
phrase_setsDefine phrase hints to boost recognition of specific terms
configManage location-level Speech-to-Text configuration
locationsQuery available service locations
operationsTrack long-running operations

Key features include support for multiple audio encodings (WAV, FLAC, MP3, OGG, WebM, MP4/AAC), translation capabilities, denoiser config, and KMS encryption support.

Vertex AI / AI Platform​

The largest update in this release with 87,000+ line changes introduces powerful new RAG and evaluation capabilities:

  • RAG Resources: rag_corpora, rag_files, rag_engine_config for Retrieval-Augmented Generation
  • Conversational AI: New chat resource
  • Model Evaluation: evaluation_sets and evaluation_items for systematic model assessment
  • New Resources: science, invoke, and openapi resources
  • Performance: Enhanced cache_config for caching configurations

Discovery Engine​

Major enhancements (50,000+ line changes) for search and conversational AI:

  • New assistants resource
  • New sitemaps resource for site search
  • New custom_models resource
  • Enhanced sessions and answers for conversational search
  • New authorized_views and authorized_view_sets for access control

Contact Center AI Insights​

Quality assurance and analytics improvements (20,000+ line changes):

  • New qa_questions and qa_question_tags for quality assurance workflows
  • New analysis_rules resource
  • New segments resource
  • New authorized_views with IAM policy support
  • New datasets and views resources

BigQuery​

Enhanced governance and access control (18,000+ line changes):

  • New routines_iam_policies for stored procedure/function IAM
  • Enhanced row_access_policies

Healthcare API​

Expanded metrics and data mapping (15,000+ line changes):

  • New data_mapper_workspaces_iam_policies
  • Enhanced metrics: hl7_v2_store_metrics, dicom_store_metrics, series_metrics, study_metrics
  • New instances_storage_info resource

Cloud Spanner​

Backup and security enhancements (14,000+ line changes):

  • New backup_schedules with IAM support
  • New databases_split_points resource
  • New database_roles with IAM policies

Cloud SQL Admin​

New integration and management features (12,000+ line changes):

  • New instances_entra_id_certificate for Microsoft Entra ID integration
  • New instances_disk_shrink_config
  • New instances_latest_recovery_time

GKE On-Prem​

Enhanced IAM across VMware and Bare Metal clusters (9,000+ line changes):

  • Enhanced VMware cluster resources with IAM policies
  • Enhanced Bare Metal cluster resources with IAM policies
  • New vmware_node_pools and bare_metal_node_pools with IAM

Developer Connect​

Git integration improvements (3,500+ line changes):

  • New git_repository_links_git_refs resource
  • New users_self and users_access_token resources
  • New token resources: read_token, read_write_token

Text-to-Speech​

Enhanced voices and text resources with new capabilities.

Get Started​

Update to the latest Google provider:

stackql registry pull google

Let us know your thoughts! Visit us and give us a star on GitHub.

Exploring the Google Cloud Asset API

· 6 min read
Jeffrey Aven
Technologist and Cloud Consultant

The Cloud Asset API has recently gone GA, this is an exceptionally useful service which stores the history and inventory of cloud resources in your GCP org. Using the Cloud Asset API via StackQL you can enumerate all of the services and resources in your GCP org, including billable resources such as Cloud Storage buckets or Compute Engine instances, as well as other objects such as billing accounts, folders, projects, firewalls, service accounts and much more. All of this can be done using SQL!

Let’s start by exploring the available fields in this service:

Explore the API​

Use the DESCRIBE or DESCRIBE EXTENDED to see the fields available in the google.cloudasset.assets resource as shown here:

DESCRIBE EXTENDED google.cloudasset.assets;

As you can see there is some very interesting stuff here, including where the asset fits in the organization hierarchy as well as whether the asset is included in a service perimeter.

Run some queries!​

To start querying you just need to supply a root node from which you want to start enumerating assets, this can be at an org level, folder level or project level.

A simple query to group and count all of the different types of assets in a GCP project is shown here:

SELECT assetType, COUNT(*)
FROM google.cloudasset.assets 
WHERE parent = 'projects/123123123123'
GROUP BY assetType;

or to see the most recent assets to be deployed or modified you could run:

SELECT name, updateTime
FROM google.cloudasset.assets 
WHERE parent = 'organizations/12312312312' 
ORDER BY updateTime DESC
LIMIT 3;

You can go nuts from here with other reports or drill into detail as to anomalies or stray assets, have fun!

Preventing Public Access for GCS Buckets

· 2 min read
Jeffrey Aven
Technologist and Cloud Consultant

Its easy enough for anyone to deploy a Cloud Storage bucket in google, this can be done through the console, gcloud, terraform or stackql as shown here: Deploying and Querying GCS Buckets using StackQL. It is also easy to inadvertently allow users to set public ACLs on a bucket, therefore making its contents publicly visible by default. There is an easy way to prevent this from happening by Using public access prevention.

Let's work through a real life scenario using StackQL.

Step 1 : Run a query to find buckets which do not have public access prevention enforced​

Run the following StackQL query from the shell or via exec:

SELECT name, 
JSON_EXTRACT(iamConfiguration, '$.publicAccessPrevention') as publicAccessPrevention
FROM  google.storage.buckets
WHERE project = 'myco-terraform';
/* returns
|-------------------|------------------------|
|       name        | publicAccessPrevention |
|-------------------|------------------------|
| myco-tf-nonprod   | unspecified            |
|-------------------|------------------------|
| myco-tf-prod      | enforced               |
|-------------------|------------------------|
*/

We can see from the query results that the myco-tf-nonprod bucket does not have public access prevention enforced, lets fix it...using StackQL.

Step 2 : Configure public access prevention for a bucket​

Run the following StackQL procedure to enforce public access prevention:

EXEC google.storage.buckets.patch 
@bucket = 'myco-tf-nonprod'
@@json = '{
    "iamConfiguration": {
      "publicAccessPrevention": "enforced"
    }
}';

Step 3: Confirm public access prevention is enforced​

Run the first query again, and you should see that the desired result is in place.

SELECT name, 
JSON_EXTRACT(iamConfiguration, '$.publicAccessPrevention') as publicAccessPrevention
FROM  google.storage.buckets
WHERE project = 'myco-terraform';
/* returns
|-------------------|------------------------|
|       name        | publicAccessPrevention |
|-------------------|------------------------|
| myco-tf-nonprod   | enforced               |
|-------------------|------------------------|
| myco-tf-prod      | enforced               |
|-------------------|------------------------|
*/

Easy!

Enable Logging for Google Cloud Storage Buckets and Analyzing Logs in Big Query (Part II)

· 4 min read
Jeffrey Aven
Technologist and Cloud Consultant

In the previous post, we showed you how to enable usage and storage logging for GCS buckets. Now that we have enabled logging, let's load and analyze the logs using Big Query. We will build up a data file vars.jsonnet as we go and show the queries step by step, at the end we will show how to run this as one batch using StackQL.

Step 1 : Create a Big Query dataset​

We will need a dataset (akin to a schema or a database in other RDMBS parlance), basically a container for objects such as tables or views, the data and code to do this are shown here:

INSERT INTO google.bigquery.datasets(
  projectId,
  data__location,
  data__datasetReference,
  data__description,
  data__friendlyName
)
SELECT
  '{{ .projectId }}',
  '{{ .location }}',
  '{ "datasetId": "{{ .datasetId }}", "projectId": "{{ .projectId }}" }',
  '{{ .description }}',
  '{{ .friendlyName }}'
;

Step 2 : Create usage table​

Let's use StackQL to create a table named usage to host the GCS usage logs, the schema for the table is defined in a file named cloud_storage_usage_schema_v0.json which can be downloaded from the location provided, for reference this is provided in the Table Schema tab in the example provided below:

/* create_table.iql */

INSERT INTO google.bigquery.tables(
  datasetId,
  projectId,
  data__description,
  data__friendlyName,
  data__tableReference,
  data__schema
)
SELECT
  '{{ .datasetId }}',
  '{{ .projectId }}',
  '{{ .table.usage.description }}',
  '{{ .table.usage.friendlyName }}',
  '{"projectId": "{{ .projectId }}", "datasetId": "{{ .datasetId }}", "tableId": "{{ .table.usage.tableId }}"}',
  '{{ .table.usage.schema }}'
;

Run the following to execute the StackQL command with the input data shown:

stackql exec -i ./create_table.iql --iqldata ./vars.jsonnet

Step 3 : Load the usage data​

We have a Big Query dataset and a table, lets load some data. To do this we will need to create and submit a load job, we can do this by inserting into the google.bigquery.jobs resource as shown here:

/* bq_load_job.iql */

INSERT INTO google.bigquery.jobs(
  projectId,
  data__configuration
)
SELECT
  'stackql',
  '{
    "load": {
      "destinationTable": {
        "projectId": "{{ .projectId }}",
        "datasetId": "{{ .datasetId }}",
        "tableId": "{{ .table.usage.tableId }}"
      },
      "sourceUris": [
        "gs://{{ .logs_bucket }}/{{ .object_prefix }}"
      ],
      "schema": {{ .table.usage.schema }},
	  "skipLeadingRows": 1,
      "maxBadRecords": 0,
      "projectionFields": []
    }
  }'
;

Run the following to execute:

stackql exec -i ./bq_load_job.iql --iqldata ./vars.jsonnet

Clean up (optional)​

If you want to clean up what you have done, you can do so using StackQL DELETE statements, as provided below:

NOTE: To delete a Big Query dataset, you need to delete all of the tables contained in the dataset first, as shown in the following example

-- delete table(s) 

DELETE FROM google.bigquery.tables 
WHERE projectId = '{{ .projectId }}' 
AND datasetId = '{{ .datasetId }}' 
AND tableId = '{{ .table.usage.tableId }}';

-- delete dataset

DELETE FROM google.bigquery.datasets 
WHERE projectId = '{{ .projectId }}' 
AND datasetId = '{{ .datasetId }}';