4 posts tagged with "sql"

Tutorial 3 introduced stackql-deploy against a single Google Cloud project. One manifest, one provider, one VPC. The framework proved its shape: declarative resources described as SQL-anchored files, an exists-create-statecheck-exports loop that runs idempotently against live cloud APIs, no state file to maintain.

This tutorial takes the same framework and puts it to work across two clouds at once. One manifest declares both a GCP VPC and an AWS VPC. One build command provisions both in the same run. The manifest becomes a single source of truth for infrastructure that spans providers, without introducing a second tool or a second DSL.

Along the way, the AWS side of the walkthrough shows features that the minimal Google example does not need: per-environment values, tag-based resource identification, resource composability through exports, and provider-specific query helpers. The interesting part is not just deploying to two clouds, but managing both through the same manifest and lifecycle.

Tutorial 2 in this series showed the query-before-mutation pattern by hand. Query live state, apply a policy gate, mutate what's out of policy, verify convergence. Every step was a SQL statement typed into a shell. The pattern held, but the shape was low-level. One resource at a time. One session at a time. No structure for repeating it across environments or holding it right over time.

stackql-deploy takes that pattern and turns it into a framework. Define your resources once as SQL-anchored files, declare them in a manifest, and let the framework run the exists check, the create-or-update, the state check, and the exports as a single build. Same underlying pattern, now something a team can operate at scale.

This tutorial walks through the tool end to end. What it is, how a project is shaped, how a single resource is defined, and how the whole thing runs against a real Google Cloud project.

If you have access to a Databricks workspace, you can run StackQL queries directly from the Databricks Web Terminal using your Databricks identity.

How It Works​

Download the latest release of stackql, then run the convenience script included (similar scripts are included for other cloud provider terminals - e.g. AWS Cloud Shell).

curl -L https://bit.ly/stackql-zip -O && unzip stackql-zip
sh stackql-databricks-shell.sh

Example Queries​

Here are the sample queries run in the video, just change the deployment_name for your workspace.

User entitlements

SELECT
  deployment_name,
  id,
  userName,
  displayName,
  entitlement
FROM databricks_workspace.iam.vw_user_entitlements
WHERE deployment_name = 'dbc-74aa95f7-8c7e';

All workspace settings

SELECT * FROM
  databricks_workspace.settings.vw_all_settings
WHERE deployment_name = 'dbc-74aa95f7-8c7e';

Tag policies filtered by key prefix

SELECT
  tag_key as key,
  description
FROM databricks_workspace.tags.tag_policies
WHERE deployment_name = 'dbc-74aa95f7-8c7e'
AND tag_key LIKE 'class%';

Catalog count by type

SELECT
  catalog_type,
  COUNT(*) as num_catalogs
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = 'dbc-74aa95f7-8c7e'
GROUP BY catalog_type;

Provider Coverage​

The databricks_workspace provider covers workspace related services, the databricks_account provider covers account-level operations including provisioning, billing, and account IAM.

The web terminal flow covers workspace-scoped queries using the token of the logged-in user. For account-level queries (provisioning, billing, account IAM), you need a Databricks service principal with account admin rights and OAuth2 credentials:

export DATABRICKS_ACCOUNT_ID="your-account-id"
export DATABRICKS_CLIENT_ID="your-client-id"
export DATABRICKS_CLIENT_SECRET="your-client-secret"

These are the same variables used by the Databricks CLI and Terraform provider, so if you already have those configured the auth story is identical.

Get Started​

Full provider documentation:

• Databricks Workspace Provider for StackQL
• Databricks Account Provider for StackQL

Visti StackQL on GitHub.

Window functions and Common Table Expressions (CTEs) are now generally available in StackQL. These features work with both the embedded SQLite backend and PostgreSQL backend.

Window Functions​

Window functions allow you to perform calculations across sets of rows related to the current row. Supported functions include:

• Ranking: ROW_NUMBER(), RANK(), DENSE_RANK(), NTILE()
• Offset: LAG(), LEAD(), FIRST_VALUE(), LAST_VALUE(), NTH_VALUE()
• Distribution: PERCENT_RANK(), CUME_DIST()
• Aggregates as window functions: SUM(), COUNT(), AVG(), etc. with OVER clause

Example: Ranking Contributors​

SELECT
    login,
    contributions,
    DENSE_RANK() OVER (ORDER BY contributions DESC) as rank
FROM github.repos.contributors
WHERE owner = 'stackql' AND repo = 'stackql';

Example: Running Totals​

SELECT
    login,
    contributions,
    SUM(contributions) OVER (ORDER BY contributions DESC) as running_total,
    ROUND(100.0 * contributions / SUM(contributions) OVER (), 2) as pct_of_total
FROM github.repos.contributors
WHERE owner = 'stackql' AND repo = 'stackql';

Common Table Expressions (CTEs)​

CTEs let you define temporary named result sets using the WITH clause. This simplifies complex queries by breaking them into logical components.

Example: Aggregating Across Multiple Resources​

WITH all_contributors AS (
    SELECT login, contributions
    FROM github.repos.contributors
    WHERE owner = 'stackql' AND repo = 'stackql'
    UNION ALL
    SELECT login, contributions
    FROM github.repos.contributors
    WHERE owner = 'stackql' AND repo = 'stackql-deploy'
)
SELECT
    DENSE_RANK() OVER (ORDER BY SUM(contributions) DESC) as rank,
    login,
    SUM(contributions) as total_contributions
FROM all_contributors
GROUP BY login
ORDER BY total_contributions DESC;

Documentation​

Full documentation is available:

• Window Functions
• WITH (CTEs)
• SELECT

Let us know your thoughts! Visit us and give us a star on GitHub.