97 posts tagged with "stackql"

We've released an updated StackQL Datadog provider covering the Datadog v1 and v2 REST APIs together: 18 services, 597 resources and 1658 operations, up from 16 services and 575 operations in the previous release.

What's new​

The previous provider was built from the v2 API alone. Datadog's most-used resources - monitors, dashboards, synthetics, SLOs, hosts, log indexes and pipelines - only exist in the v1 API, so this release merges the two specs into one provider. The v2 surface has also grown considerably since the last build. In summary:

• The v1 API: monitors (list, search, create, replace, validate, delete), dashboards and dashboard lists, synthetics tests (API, browser and mobile), locations, private locations and global variables, SLOs and SLO corrections, hosts, host totals and host tags, notebooks, log indexes and pipelines, the Azure, PagerDuty, Slack and webhook integrations, and usage metering.
• New v2 surfaces: cases and case projects, on-call schedules, escalation policies and paging, status pages, incident configuration and responders, feature flags, deployment gates, LLM Observability (projects, datasets, experiments, prompts, annotation queues), Fleet Automation, cloud cost budgets, commitments and tag pipelines, security findings automation, static analysis and SCA, agentless scanning, SIEM historical detections, RUM replay and product analytics, reference tables, org groups and personal access tokens, among others.
• Site from the environment: the provider addresses https://api.{site}, and site is resolved from DD_SITE when it is set (datadoghq.eu, us5.datadoghq.com, ap2.datadoghq.com, ...), the same convention as the Datadog Agent and API clients. Queries carry no site clause; a WHERE site = '...' still wins for one statement.
• Pagination and pushdown: cursor-paginated lists (audit events, container images, spans, RUM events, CI events, security signals and findings) are traversed transparently, and a SQL LIMIT is sent as the API's page-size parameter.
• snake_case surface: columns and WHERE / INSERT keys are snake_case throughout; the few camelCase wire names are aliased.
• Terraform-aligned authentication: DD_API_KEY and DD_APP_KEY, unchanged.

Service highlights​

Authentication​

Export an API key and an application key; set DD_SITE if your organization is not on datadoghq.com:

export DD_API_KEY=...
export DD_APP_KEY=...
export DD_SITE=datadoghq.eu     # optional, defaults to datadoghq.com

Monitors​

Every monitor with its state:

SELECT id, name, type, overall_state, tags
FROM datadog.monitoring.monitors;

Only alerting monitors, using the API's own filter:

SELECT id, name, overall_state
FROM datadog.monitoring.monitors
WHERE group_states = 'alert';

Monitor search, with the same syntax as the Manage Monitors page:

SELECT id, name, status, type
FROM datadog.monitoring.monitor_search_results
WHERE query = 'type:metric status:alert';

Dashboards, SLOs and synthetics​

SELECT id, title, layout_type, author_handle, modified_at
FROM datadog.dashboards.dashboards;

SELECT id, name, type, target_threshold, timeframe
FROM datadog.service_management.slos;

SELECT public_id, name, type, status, locations
FROM datadog.monitoring.synthetics_tests;

Users, roles and keys​

v2 resources return the JSON:API row shape - id, type, attributes, relationships - so attributes are one json_extract away. A user audit:

SELECT id,
       json_extract(attributes, '$.email') AS email,
       json_extract(attributes, '$.status') AS status,
       json_extract(attributes, '$.disabled') AS disabled,
       json_extract(attributes, '$.created_at') AS created_at
FROM datadog.organization.users;

API keys by age, the input to a rotation policy:

SELECT id,
       json_extract(attributes, '$.name') AS name,
       json_extract(attributes, '$.created_at') AS created_at,
       json_extract(attributes, '$.last4') AS last4
FROM datadog.organization.api_keys
ORDER BY created_at;

Infrastructure and logs​

Hosts reporting to Datadog, and the log indexes with their retention:

SELECT host_name, up, is_muted, apps, last_reported_time
FROM datadog.infrastructure.hosts;

SELECT name, num_retention_days, daily_limit
FROM datadog.logs.indexes;

Audit log​

The audit event list is cursor-paginated and takes the time window as a query parameter:

SELECT json_extract(attributes, '$.timestamp') AS timestamp,
       json_extract(attributes, '$.attributes.evt.name') AS event,
       json_extract(attributes, '$.attributes.usr.email') AS actor
FROM datadog.organization.audit_logs
WHERE "filter[from]" = 'now-1d';

Security monitoring rules​

Which detection rules are enabled, and who last changed them:

SELECT id, name, type, is_enabled, is_default, updated_at, update_author_id
FROM datadog.security.monitoring_rules
WHERE is_default = false;

Provisioning​

Mutations use the same SQL grammar. v1 resources take their fields as columns; v2 resources take the JSON:API data document. A monitor end to end - validate the definition, create it, replace it (the v1 monitor API updates with PUT), delete it:

EXEC datadog.monitoring.monitors.validate_monitor
  @type = 'metric alert',
  @query = 'avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 90',
  @name = 'High CPU on prod hosts';

INSERT INTO datadog.monitoring.monitors (name, type, query, message, tags)
SELECT 'High CPU on prod hosts',
       'metric alert',
       'avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 90',
       'CPU above 90% on {{host.name}} @slack-ops',
       '["team:web", "managed-by:stackql"]';

REPLACE datadog.monitoring.monitors
SET name = 'High CPU on prod hosts', type = 'metric alert',
    query = 'avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 95'
WHERE monitor_id = 12345678;

DELETE FROM datadog.monitoring.monitors
WHERE monitor_id = 12345678;

A downtime for a release window, and a role:

INSERT INTO datadog.service_management.downtimes (data)
SELECT '{"type": "downtime",
         "attributes": {"message": "release window", "scope": "env:prod",
                        "monitor_identifier": {"monitor_tags": ["team:web"]},
                        "schedule": {"start": "2026-09-01T22:00:00Z", "end": "2026-09-01T23:00:00Z"}}}';

INSERT INTO datadog.organization.roles (data)
SELECT '{"type": "roles", "attributes": {"name": "read-only-auditors"}}';

UPDATE datadog.organization.roles
SET data = '{"id": "<role-id>", "type": "roles", "attributes": {"name": "auditors"}}'
WHERE role_id = '<role-id>';

Get started​

Pull the provider from the public registry:

registry pull datadog;

Provider docs are at datadog-provider.stackql.io. Let us know what you build. Star us on GitHub.

We've rebuilt the StackQL Kubernetes provider from the ground up:

• k8s - every built-in control plane API group in a pinned Kubernetes minor release (currently 1.36): core, apps, batch, autoscaling, networking, storage, rbac, policy, apiextensions, admissionregistration, certificates, coordination, discovery, events, flowcontrol, node, scheduling, authentication, authorization and apiregistration (20 services, 152 resources, 573 operations)

The provider is generated from the per-group specs published in the Kubernetes repository, so the same provider works against kind, EKS, GKE, AKS, OpenShift or bare metal. Subresources (status, scale, log, eviction, binding, approval) are first-class resources, list pagination is traversed transparently, and LIMIT and label/field selectors are pushed down to the API server.

Connect with kubectl proxy​

The provider defaults to null_auth, designed for the kubectl proxy workflow - the proxy authenticates with your kubeconfig (including the EKS, GKE and AKS credential plugins), and StackQL connects to the local port with no configuration:

kubectl proxy --port=8001
export KUBE_HOST='localhost:8001'
export KUBE_PROTOCOL='http'
stackql shell

KUBE_HOST and KUBE_PROTOCOL resolve the provider's server variables from the environment, so queries carry no connection clauses at all (an explicit WHERE protocol = ... AND cluster_addr = ... still wins when you want to address another cluster in the same session).

The cluster is a database​

Row columns are the top-level fields of each object (metadata, spec, status, data); nested values are one json_extract away. The pod estate with phase and node placement:

SELECT json_extract(metadata, '$.namespace') AS namespace,
       json_extract(metadata, '$.name') AS name,
       json_extract(status, '$.phase') AS phase,
       json_extract(spec, '$.nodeName') AS node
FROM k8s.core.pods_all_namespaces;

Pods that are not running - a one-line cluster health check:

SELECT json_extract(metadata, '$.namespace') AS namespace,
       json_extract(metadata, '$.name') AS name,
       json_extract(status, '$.phase') AS phase
FROM k8s.core.pods_all_namespaces
WHERE json_extract(status, '$.phase') NOT IN ('Running', 'Succeeded');

Desired versus ready replicas for every deployment in a namespace:

SELECT json_extract(metadata, '$.name') AS name,
       json_extract(spec, '$.replicas') AS want,
       json_extract(status, '$.readyReplicas') AS ready
FROM k8s.apps.deployments
WHERE namespace = 'default';

Node inventory with kubelet version and schedulability:

SELECT json_extract(metadata, '$.name') AS name,
       json_extract(status, '$.nodeInfo.kubeletVersion') AS kubelet,
       json_extract(status, '$.nodeInfo.osImage') AS os,
       json_extract(spec, '$.unschedulable') AS cordoned
FROM k8s.core.nodes;

RBAC audit and warning events​

Column names are snake_case at the SQL surface (role_ref, string_data, api_version), mapped to the API's camelCase on the wire - the same convention as the aws and azure providers. Who is bound to cluster-admin:

SELECT json_extract(metadata, '$.name') AS binding,
       subjects
FROM k8s.rbac.cluster_role_bindings
WHERE json_extract(role_ref, '$.name') = 'cluster-admin';

Recent warning events across the cluster, usually the first place to look when something is off:

SELECT json_extract(metadata, '$.namespace') AS namespace,
       reason,
       message
FROM k8s.core.events_all_namespaces
WHERE type = 'Warning';

Server-side filtering and pagination​

Label and field selectors are ordinary WHERE parameters, pushed to the API server so the filtering happens where the data lives:

SELECT json_extract(metadata, '$.name') AS name
FROM k8s.core.pods_all_namespaces
WHERE label_selector = 'k8s-app=kube-dns';

SELECT ... LIMIT n lands on the wire as the Kubernetes limit parameter, and the continue token chain is followed transparently - a SELECT returns all rows even when the API server caps page sizes.

Provision, mutate and tear down​

Mutations are the usual SQL verbs - INSERT creates an object, UPDATE applies a merge patch, REPLACE is a full update and DELETE removes it. Body columns are the native wire property names:

-- create
INSERT INTO k8s.core.config_maps(namespace, metadata, data)
SELECT 'default', '{"name": "app-config"}', '{"greeting": "hello"}';

-- partial update (merge patch)
UPDATE k8s.core.config_maps
SET data = '{"mood": "optimistic"}'
WHERE namespace = 'default' AND name = 'app-config';

-- remove it
DELETE FROM k8s.core.config_maps
WHERE namespace = 'default' AND name = 'app-config';

Subresources are resources, so scaling a deployment is an UPDATE on its scale subresource:

UPDATE k8s.apps.deployments_scale
SET spec = '{"replicas": 3}'
WHERE namespace = 'default' AND name = 'web';

and point-in-time pod logs are a queryable column:

SELECT log FROM k8s.core.pods_log
WHERE namespace = 'default' AND name = 'web-6d5f9c7b8-x2x9k';

Streaming operations (exec, attach, port-forward, watch) use protocol upgrades and are out of scope for the generated provider.

Who am I, and can I​

The authentication and authorization review kinds are mapped too. The zero-parameter self review is a plain SELECT; the parameterized access reviews are an INSERT whose verdict comes back with RETURNING:

-- who am I
SELECT json_extract(status, '$.userInfo.username') AS username
FROM k8s.authentication.self_subject_reviews;

-- can I delete pods in prod
INSERT INTO k8s.authorization.self_subject_access_reviews(spec)
SELECT '{"resourceAttributes": {"verb": "delete", "resource": "pods", "namespace": "prod"}}'
RETURNING status;

What changed from the original provider​

This is a major update to the previous published k8s provider (v23.03.00121):

• Coverage expands from 5 services to all 20 built-in API groups, one flat service per group (networking.k8s.io is k8s.networking)
• Resource names are plural snake_case (k8s.core.pods, k8s.apps.stateful_sets), consistent with the aws, google and databricks providers
• Request body columns are the native wire property names (metadata, spec, data), not data__ prefixed; snake_case spellings of camelCase wire names are accepted everywhere
• SELECT and DESCRIBE columns present as snake_case aliases of the camelCase wire properties
• Namespaced list-all operations are separate _all_namespaces resources, and subresources are separate resources (deployments_scale, pods_log)

The previous provider version remains in the registry for pinning if you need it.

Direct authentication​

To skip the proxy and hit the API server directly, supply a bearer token via the KUBE_TOKEN environment variable (the same variable the Terraform Kubernetes provider uses), along with the cluster CA bundle:

export KUBE_TOKEN=$(kubectl create token my-serviceaccount)
AUTH='{ "k8s": { "type": "bearer", "credentialsenvvar": "KUBE_TOKEN" }}'
stackql shell --auth="${AUTH}" --tls.CABundle cluster-ca.pem

For managed clusters, a token from the platform credential helper (aws eks get-token, gke-gcloud-auth-plugin, kubelogin) works the same way; those tokens are short lived, so prefer the proxy vector for long sessions.

Get started​

Pull the provider from the public registry:

registry pull k8s;

Provider docs are at k8s-provider.stackql.io. Let us know what you build. Star us on GitHub.

We've released an update to the StackQL AWS provider, regenerated from the latest AWS service definitions. Significant additions include:

• 9 new services
• Over 200 new resources
• Over 500 new operations
• Full support for S3 object level CRUD operations

New Services​

Nine services are new in this release:

Expanded Coverage in Existing Services​

• ec2 - the largest operation count increase in this release: account-level VPC encryption controls (account_vpc_encryption_controls), application status checks, capacity reservation cancellation quotes, and a major IPAM build-out covering internet registry associations, route origin authorizations, route protection findings and discovered routes
• acm - public ACME issuance: acme_accounts, acme_endpoints, acme_domain_validations and external account bindings, plus per-domain certificate validation status
• quicksight - agentic BI: agents, spaces, knowledge_bases, approval policies, DLP settings and OAuth client applications
• bedrock_agentcore_control - datasets, dataset versions and examples, evaluation harness_endpoints and versions, capacity providers and gateway rate limits
• wellarchitected - AI-assisted reviews: agent profiles, goals, contexts and agent_recommendations with per-item detail
• odb (Oracle Database@AWS) - autonomous database coverage: autonomous_databases, backups, clones, peers, wallet details, Exadata VM clusters and Exascale storage vaults
• iotsitewise - industrial data pipelines: pipelines, pipeline executions, enrichment jobs, dataset export jobs, ad hoc queries and search
• glue - business catalog additions: assets, asset_types, glossaries and glossary_terms
• drs - orchestrated disaster recovery: recovery_plans, plan steps and execution tracking
• billing - credits, credit allocation histories, billing preferences and enterprise support charge summaries
• appconfig - feature experiments: experiment_definitions, runs and run events
• healthlake - FHIR data transformation profiles and jobs

Another 180+ services picked up incremental operations and resources, including connect, securityagent, cleanrooms, socialmessaging, devops_agent and artifact.

S3 Object Content Lifecycles​

The headline feature in this release: aws.s3.objects now supports the full content lifecycle for text objects. The object body is projected as a single contents column, so you can read, write, overwrite and delete object contents with standard SQL verbs - no SDK code, no presigned URLs.

The motivating case is reading a Terraform state file straight out of S3:

SELECT contents FROM aws.s3.objects
WHERE region = 'ap-southeast-2'
AND bucket = 'my-bucket'
AND key = 'env/terraform.tfstate';

Writing works through the same resource. INSERT creates an object, and since S3 PutObject is create-or-overwrite, REPLACE updates it in place:

-- create an object
INSERT INTO aws.s3.objects(region, bucket, key, contents)
SELECT 'ap-southeast-2', 'my-bucket', 'app/config.json',
'{"feature_flags": {"dark_mode": true}, "log_level": "info"}';

-- overwrite its contents
REPLACE aws.s3.objects
SET contents = '{"feature_flags": {"dark_mode": false}, "log_level": "warn"}'
WHERE region = 'ap-southeast-2'
AND bucket = 'my-bucket'
AND key = 'app/config.json';

-- delete it
DELETE FROM aws.s3.objects
WHERE region = 'ap-southeast-2'
AND bucket = 'my-bucket'
AND key = 'app/config.json';

Listing is unchanged - a WHERE clause with just bucket and region routes to the list operation, adding key routes to the object read:

SELECT key, size, last_modified FROM aws.s3.objects
WHERE region = 'ap-southeast-2' AND bucket = 'my-bucket';

Content round-trips byte-for-byte, including multi-MB objects and objects uploaded via multipart upload. Text objects only for now - binary content is not supported and base64 support is deferred.

Get Started​

Pull the latest provider from the public registry:

stackql registry pull aws;

Authenticate with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, then explore:

SELECT vpc_id, state, cidr_block, is_default
FROM aws.ec2.vpcs
WHERE region = 'us-east-1';

Provider docs, including required parameters and example queries for every resource, are at aws-provider.stackql.io. Visit us on GitHub and let us know how you're using it.

If you've audited buckets in more than one cloud, you already know what this is like. Three consoles. Three CLIs. Three auth patterns. And a bit of glue code to pull the outputs together every time someone asks a question that spans all three.

The question is usually simple. "Which buckets are public across all our clouds?" You end up running three separate scripts, then reconciling three different output shapes just to give one answer.

This tutorial doesn't make that faster. It replaces it. One query, one view, all three clouds.

What StackQL is, in three sentences​

StackQL is SQL for cloud APIs. You write a normal SQL query, and it makes the API calls to AWS, GCP, Azure, or dozens of other providers to get the answer back as rows and columns.

The important part for this tutorial: it queries the live provider APIs on every run. There's no local database, no cache, no sync job to schedule. What you see is the current state of your cloud, right now.

It exists because someone got tired of writing the same audit logic three times.

What you'll need​

You'll need Docker installed and read-only credentials for whichever clouds you want to audit. All three are optional. If you leave a cloud's credentials blank, it gets skipped.

For AWS, GCP, and Azure, the audit needs a service account or role that can list storage buckets and read their configuration. The audit-action repo has the exact permissions per provider here.

Run the audit​

The tutorial folder has three files: a docker-compose file, an .env.audit.example template, and a README. Grab the folder from docs/tutorials/preview-bucket-01/ in the stackql/stackql repo.

1. Copy the env template

cp ./audit/.env.audit.example ./audit/.env.audit

Then open .env.audit and fill in credentials for whichever clouds you want. The variables are named for what they are: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, GOOGLE_CREDENTIALS (the service account JSON, on one line), and GOOGLE_ORG_ID.

If you're only auditing one or two clouds, leave the others blank.

2. Pull the StackQL Docker image

docker compose -f docker-compose.bucket.audit.yaml pull

The tutorial pins to stackql/stackql:v0.10.601. The audit uses views under stackql_preview.* which are still evolving, so pinning matters for reproducibility.

3. Run the audit

docker compose -f docker-compose.bucket.audit.yaml run --rm stackql

On a modest account this takes about 30 seconds. Output goes straight to your terminal as a table.

What the output tells you​

When the audit finishes, you get a single table printed to your terminal. Something like this (bucket names are placeholders):

Example output from the audit. Bucket configuration across three cloud providers in one normalised table.

The point isn't the specific findings. It's the shape.

Three different resource types (S3 buckets, GCS buckets, and Azure Storage Accounts) normalised into one table with the same columns. Encryption class, public flag, HTTPS enforcement. Side by side, for every cloud you provided credentials for.

A few patterns to notice in the table:

• Public buckets across providers. Filter by public = true and you have the answer to your security team's question, in every cloud, in one place.
• Encryption side by side. The encryption_class column shows which buckets use provider-managed keys versus customer-managed. Different providers have different naming conventions internally; StackQL normalises them to the same two categories.
• HTTPS enforcement in one column. Whether it's S3 bucket policies, GCS uniform access, or Azure secure transfer, same column, same values, no translation needed.

This is what SQL for cloud APIs actually delivers. Not just SQL as a query language, but a uniform data model across providers, so you can ask one question and get one answer.

Performance and scope​

StackQL runs each provider's query in parallel using the credentials you provided. There's no local database being populated and no background sync; every run hits the live APIs and returns what's there right now.

On the test account this took about 30 seconds. The query is scoped to one AWS region and one GCP organization, which keeps the API footprint bounded and reflects the current preview default. Streaming output and broader audits (across regions, deeper across resource types) are in flight from the StackQL team.

What this covers today​

Today, this audit covers storage buckets across AWS, GCP, and Azure. One region at a time for AWS, one organization at a time for GCP. Entitlements, IAM, and other resource types are on the roadmap as separate audits under stackql_preview.* and will get their own tutorials as they land.

Where to go next​

github.com/stackql/stackql is the flagship repo. If this tutorial was worth reading, give it a star. It's genuinely the thing that tells the team to build more tutorials like this one, and it helps other engineers with the same three-console problem find the project.

A few natural next steps from here:

• Run it against your own accounts. The audit works exactly the same way against real credentials. You can start with one cloud and add the others by adding their env vars.
• Move it into CI. The stackql-audit-action repo has ready-to-use GitHub Actions workflows for all-clouds, single-cloud, deep-audit, and OIDC-authenticated variants. Drop one into your repo, add secrets, and you have a scheduled cross-cloud audit running on every push.
• Ask questions. The StackQL community Slack is the fastest way. GitHub issues on the flagship repo also work.

More audits are on the way, including entitlements, deeper checks, and larger estates. If there's a specific one you'd find useful, opening an issue is the fastest way to get it on the roadmap.

We've released a new StackQL provider for ClickHouse Cloud:

• clickhouse - the ClickHouse Cloud control plane (api.clickhouse.cloud): organizations, services (lifecycle, scaling, settings, passwords, private endpoints), API keys, members and invitations, organization roles, backups and backup configuration, usage cost and quotas, activities, ClickPipes, the ClickStack surface (dashboards, alerts, sources, webhooks, saved searches), user-defined functions and Managed Postgres (10 services, 44 resources, 139 operations)

The provider covers the Cloud management API only. The ClickHouse server HTTP interface - SQL against a service endpoint and the system.* tables - is a separate surface with separate authentication and is reserved as a future sibling provider, clickhouse_server. What takes two Terraform providers today (the Cloud infrastructure provider and the DBops provider) will be two namespaces in one StackQL session.

Organization scope from the environment​

Every resource except organizations is scoped to an organization. The organization ID is a server variable that StackQL resolves from CLICKHOUSE_ORG_ID when it is set, so queries carry no organization clause:

SELECT name, state, provider, region
FROM clickhouse.services.services;

A WHERE organization_id = '...' still takes precedence when you need to address another organization in the same session. Columns and WHERE/INSERT keys are snake_case (created_at, ip_access_list, service_id); the provider maps them to the API's camelCase on the wire.

Estate inventory​

State, footprint and scaling configuration for every service in one result set:

SELECT name, state, provider, region,
       num_replicas,
       min_replica_memory_gb, max_replica_memory_gb,
       json_extract(current_scaling, '$.effectiveAutoscalingMode') AS scaling_mode,
       idle_scaling, idle_timeout_minutes, clickhouse_version
FROM clickhouse.services.services
ORDER BY provider, region, name;

Organization quotas report usage against limits, which is the quickest answer to "how much room is left":

SELECT quota_code, name, value AS quota_limit, usage
FROM clickhouse.organizations.quotas;

Usage cost by day and entity​

The usage cost endpoint returns one row per entity per day in ClickHouse Credits, with the compute, storage, backup and data-transfer components in a metrics object:

SELECT date, entity_type, entity_name, total_chc,
       json_extract(metrics, '$.computeCHC') AS compute_chc,
       json_extract(metrics, '$.storageCHC') AS storage_chc,
       json_extract(metrics, '$.backupCHC')  AS backup_chc
FROM clickhouse.organizations.usage_costs
WHERE from_date = '2026-08-01' AND to_date = '2026-08-31'
ORDER BY date, entity_name;

Joined to the services list, the same data answers which stopped or idle services still carried cost in the window:

SELECT s.name, s.state, SUM(c.total_chc) AS chc_in_window
FROM clickhouse.services.services s
JOIN clickhouse.organizations.usage_costs c
  ON c.service_id = s.id
WHERE c.from_date = '2026-08-01' AND c.to_date = '2026-08-31'
  AND s.state IN ('stopped', 'idle')
GROUP BY s.name, s.state
ORDER BY chc_in_window DESC;

Scaffolding​

Provisioning is the usual SQL verbs. A service is an INSERT; the state command (start, stop, awake) is an EXEC; access-list changes are an UPDATE whose PATCH body takes add/remove arrays, passed through as written:

INSERT INTO clickhouse.services.services
  (name, provider, region, min_replica_memory_gb, max_replica_memory_gb, num_replicas, idle_scaling, idle_timeout_minutes)
SELECT 'analytics-dev', 'aws', 'us-east-1', 8, 8, 1, true, 15;

UPDATE clickhouse.services.services
SET ip_access_list = '{"add": [{"source": "203.0.113.0/24", "description": "office"}],
                       "remove": [{"source": "0.0.0.0/0", "description": "Anywhere"}]}'
WHERE service_id = '<service-uuid>';

EXEC clickhouse.services.services.update_state
  @serviceId = '<service-uuid>',
  @command = 'stop';

Organizations that use Custom Roles assign API key roles by ID, so the role lookup and the key creation are one statement, and the generated secret comes back with RETURNING:

INSERT INTO clickhouse.keys.keys (name, assigned_role_ids, state)
SELECT 'finops-reader', '["' || id || '"]', 'enabled'
FROM clickhouse.roles.roles
WHERE name = 'Organization API Reader'
RETURNING json_extract(result, '$.key.id')    AS id,
          json_extract(result, '$.keyId')     AS key_id,
          json_extract(result, '$.keySecret') AS key_secret;

Backup configuration is queryable across the estate, and settable per service:

SELECT s.name, b.backup_period_in_hours, b.backup_retention_period_in_hours, b.backup_start_time
FROM clickhouse.services.services s
JOIN clickhouse.backups.backup_configurations b
  ON b.service_id = s.id;

Observability as code​

ClickStack dashboards, alerts, sources, webhooks and saved searches are resources with full CRUD, so a dashboard definition lives in version control and is applied with an INSERT:

INSERT INTO clickhouse.clickstack.dashboards (service_id, name, tiles, tags)
SELECT '<service-uuid>', 'Service Overview',
       '[{"name": "Error rate", "x": 0, "y": 0, "w": 6, "h": 3,
          "config": {"displayType": "line",
                     "select": [{"aggFn": "count", "where": "SeverityText = ''ERROR''"}]}}]',
       '["production"]';

and audited with a SELECT:

SELECT d.name, json_array_length(d.tiles) AS tiles, d.tags
FROM clickhouse.clickstack.dashboards d
WHERE d.service_id = '<service-uuid>';

Alerts and webhooks follow the same pattern (clickhouse.clickstack.alerts, clickhouse.clickstack.webhooks), and the organization activity log is a table for audit questions:

SELECT created_at, type, actor_type, actor_details
FROM clickhouse.organizations.activities
ORDER BY created_at DESC;

The data platform estate in one query​

Cross-provider joins are ordinary SQL, so ClickHouse Cloud services sit alongside Snowflake warehouses and Databricks clusters in one inventory:

SELECT 'clickhouse' AS platform, name, state, region
FROM clickhouse.services.services
UNION ALL
SELECT 'snowflake', name, state, NULL
FROM snowflake.warehouses.warehouses
UNION ALL
SELECT 'databricks', cluster_name, state, NULL
FROM databricks_workspace.compute.clusters
WHERE deployment_name = '<workspace>';

Authentication​

Create an API key pair in the ClickHouse Cloud console (Settings -> API Keys) with the role the queries need - Organization API Reader / Service API Reader for inventory and cost queries, Service API Admin for provisioning - and export three variables:

export CLICKHOUSE_CLOUD_API_KEY=...      # Key ID
export CLICKHOUSE_CLOUD_API_SECRET=...   # Key Secret
export CLICKHOUSE_ORG_ID=...             # organization ID

The API allows a fixed window of requests per key (documented as 10 per 10 seconds); a 429 is the signal to pace wide scans.

Get started​

Pull the provider from the public registry:

registry pull clickhouse

Provider docs are at clickhouse-provider.stackql.io. Let us know what you build. Star us on GitHub.