Databricks Providers Update - November 2026

We've released an update to the StackQL Databricks providers, regenerated from the latest Databricks platform APIs (SDK v0.158.0):

• databricks_account - account-level operations: 10 services, 55 resources
• databricks_workspace - workspace-level operations: 36 services, 315 resources

Includes 370 resources and over 1,300 operations across both providers, with eight new services and more than 50 new resources, and 49 built-in views. Both providers now work against Databricks on AWS, Azure and Google Cloud; cloud selection and credentials are covered in the addendum at the end of this post.

New Services

Provider	Service	Description
databricks_workspace	agentkit	Agent Kit managed memory stores and session stores: memories, sessions, session items, forks, memory extraction, and store permissions
databricks_workspace	sandbox	Sandboxes: create, start, stop and update isolated environments, run commands synchronously, and manage files and directories inside them
databricks_workspace	aifunctions	The AI Functions API: ai_classify, ai_decide, ai_extract and ai_parse_document as EXEC methods
databricks_workspace	domains	Data domains in Unity Catalog, with owners, hierarchy and draft state
databricks_workspace	knowledgeassistants	Agent Bricks Knowledge Assistants, their knowledge sources (with sync), examples and permissions
databricks_workspace	environments	Workspace base environments for serverless compute (previously under compute.default_base_environments), including the default environment and refresh
databricks_workspace	dataclassification	Data classification configuration per Unity Catalog catalog
databricks_account	networking	Account private networking: endpoints and private network gateways on network connectivity configurations

Expanded Coverage in Existing Services

• postgres (Lakebase) - every Lakebase resource now has the full get, update and delete lifecycle alongside create and list (branches, endpoints, roles, databases, catalogs, CDF configs, data APIs, snapshots), plus snapshot schedules, undelete for branches and reveal_password for roles. Earlier releases could only create and list several of these resources because the API addresses them by resource name on a shared path; the generator now maps each one to its own path
• catalog - AI Gateway skills (create, list, get, update, delete and finalize) and per-user MCP service credentials
• iamv2 (account and workspace) - identity visibility filters, workspace assignments, and external group and service principal lookups
• ml - experiment issues, feature materialization sets, feature engineering operations, backfill and purge
• billing (account) - budget spend statuses
• disasterrecovery (account) - get and delete for failover groups
• supervisoragents - get, update and delete for supervisor agents, tools and examples
• vectorsearch - start and stop lifecycle operations on vector search endpoints
• dashboards - cancel a running AI/BI Genie response
• bundledeployments - get and update for deployments, get for operations and deployed resources

Request Body Columns

Operations whose request body is a single object in the Databricks API (for example creating a network connectivity configuration, a private endpoint rule, a sandbox or a domain) now expose that object's fields directly as INSERT and UPDATE columns, which is what the API accepts:

INSERT INTO databricks_account.settings.network_connectivity (account_id, name, region)
SELECT '<account_id>', 'lakeside-prod-ncc', 'australiaeast';

Earlier releases modelled these bodies as one JSON column named after the object (network_connectivity_config, private_endpoint_rule), a shape the API rejected. Statements written that way need the object's fields instead.

Removed Upstream

The following operations were removed from the Databricks platform APIs and no longer appear in the providers: account billing.usage_policy, account iamv2 access identity rules and attribute control entries, compute.default_base_environments (now environments.workspace_base_environments), database instance failover and autoscaling upgrade, the published dashboard embed endpoint, pipeline restore, vector search index auto-evaluation and reranker fine-tuning, Lakebase tables, forward ETL and replication group previews, and AI Gateway agent services.

Sandboxes

Create a sandbox, run a command in it and read the output, then stop it - the compute lifecycle as INSERT, EXEC and SELECT:

/* create */
INSERT INTO databricks_workspace.sandbox.sandboxes (
  sandbox_id,
  display_name,
  deployment_name
)
SELECT
  'etl-scratch',
  'etl-scratch',
  '<deployment_name>';

/* list */
SELECT name, display_name, status
FROM databricks_workspace.sandbox.sandboxes
WHERE deployment_name = '<deployment_name>';

/* run a command synchronously */
EXEC databricks_workspace.sandbox.sandbox_commands.execute
  @sandbox_id = '<sandbox_id>',
  @deployment_name = '<deployment_name>'
  @@json = '{"cmd": "ls -la /workspace"}';

/* stop and delete */
EXEC databricks_workspace.sandbox.sandboxes.stop
  @sandbox_id = '<sandbox_id>', @deployment_name = '<deployment_name>';

DELETE FROM databricks_workspace.sandbox.sandboxes
WHERE sandbox_id = '<sandbox_id>'
AND deployment_name = '<deployment_name>';

Agent Kit Memory and Sessions

Managed memory stores and session stores for agents, with their contents as queryable rows:

SELECT name, display_name, storage_backend, owner_user_id
FROM databricks_workspace.agentkit.memory_stores
WHERE deployment_name = '<deployment_name>';

SELECT name, display_name, create_time
FROM databricks_workspace.agentkit.session_stores
WHERE deployment_name = '<deployment_name>';

SELECT *
FROM databricks_workspace.agentkit.sessions
WHERE session_store_id = '<session_store_id>'
AND deployment_name = '<deployment_name>';

AI Functions

The AI Functions API is exposed as EXEC methods, so a classification or document parse is one statement:

EXEC databricks_workspace.aifunctions.ai_functions.ai_classify
  @deployment_name = '<deployment_name>'
  @@json = '{"content": "The invoice is 30 days overdue", "labels": ["billing", "support", "sales"]}';

Data Domains and Knowledge Assistants

Unity Catalog data domains with their owners and hierarchy, and Agent Bricks Knowledge Assistants with their knowledge sources:

SELECT domain_id, name, parent_domain_id, business_owner_ids
FROM databricks_workspace.domains.domains
WHERE deployment_name = '<deployment_name>';

SELECT id, display_name, state, endpoint_name
FROM databricks_workspace.knowledgeassistants.knowledge_assistants
WHERE deployment_name = '<deployment_name>';

SELECT id, display_name, source_type, state
FROM databricks_workspace.knowledgeassistants.knowledge_sources
WHERE knowledge_assistant_id = '<knowledge_assistant_id>'
AND deployment_name = '<deployment_name>';

Lakebase Snapshots

Snapshots and snapshot schedules for a Lakebase branch, now with the complete lifecycle:

SELECT snapshot_id, name, status
FROM databricks_workspace.postgres.postgres_snapshots
WHERE project_id = '<project_id>'
AND deployment_name = '<deployment_name>';

SELECT name, schedule
FROM databricks_workspace.postgres.postgres_snapshot_schedules
WHERE project_id = '<project_id>'
AND branch_id = '<branch_id>'
AND deployment_name = '<deployment_name>';

Account Private Networking

Private endpoints and private network gateways at the account level:

SELECT endpoint_id, display_name, region, state, use_case
FROM databricks_account.networking.endpoints
WHERE account_id = '<account_id>';

SELECT display_name, traffic_mode, bandwidth_tier_gigabits_per_second, state
FROM databricks_account.networking.private_network_gateways
WHERE account_id = '<account_id>'
AND ncc_id = '<network_connectivity_config_id>';

Budget Spend

Where each budget stands against its alerts:

SELECT name, alert_spend_statuses
FROM databricks_account.billing.budget_spend_statuses
WHERE account_id = '<account_id>';

Get Started

Pull the latest providers from the public registry:

stackql registry pull databricks_account
stackql registry pull databricks_workspace

Authenticate with the same service principal environment variables used by Terraform and the Databricks CLI (DATABRICKS_ACCOUNT_ID, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET), add DATABRICKS_CLOUD=azure or DATABRICKS_CLOUD=gcp for an account on those clouds (stackql v1.0.888 or later), then explore:

SELECT workspace_id, workspace_name, workspace_status, deployment_name
FROM databricks_account.provisioning.workspaces
WHERE account_id = '<account_id>';

Provider docs are at databricks-account-provider.stackql.io and databricks-workspace-provider.stackql.io. Visit us on GitHub and let us know how you're using it.

Addendum: Azure Databricks and Databricks on Google Cloud

Both providers now work against Databricks accounts and workspaces on AWS, Azure and Google Cloud, with the same statements and the same three credential variables. Cloud selection needs stackql v1.0.888 or later.

Selecting the cloud

One variable picks the cloud; the OAuth token endpoint, the account API host and the workspace host domain follow it:

• DATABRICKS_CLOUD - aws (default), azure or gcp
• DATABRICKS_ACCOUNT_HOST - optional, a bare account console host (no scheme) for accounts that are not on the standard hosts; overrides DATABRICKS_CLOUD
• DATABRICKS_DEPLOYMENT_NAME - optional, defaults the deployment_name predicate for databricks_workspace queries

Cloud	DATABRICKS_CLOUD	Account console and token host	Workspace host	deployment_name
AWS	aws	accounts.cloud.databricks.com	<deployment_name>.cloud.databricks.com	dbc-abcd0123-a1bc
Azure	azure	accounts.azuredatabricks.net	<deployment_name>.azuredatabricks.net	adb-1234567890123456.7
GCP	gcp	accounts.gcp.databricks.com	<deployment_name>.gcp.databricks.com	1234567890123456.7

deployment_name is the host prefix of the workspace URL on every cloud. With nothing set, behaviour is unchanged from earlier releases: AWS hosts and the AWS token endpoint. One cloud per process: the token endpoint follows the environment rather than a SQL predicate, so use a separate stackql process or shell for each cloud.

Credentials

The credential on every cloud is a service principal with a Databricks OAuth secret, created in the account console (User management > Service principals > Add service principal, then Generate secret) and given the Account admin role for account-level operations. Cloud identity tokens (AWS IAM, Microsoft Entra ID, Google service accounts) are not used.

• Azure: sign in to accounts.azuredatabricks.net with the Entra ID identity that administers the Databricks account. A Databricks-managed service principal, or an Entra ID application registered as a service principal, both work when the OAuth secret is generated in the Databricks account console.
• GCP: accounts.gcp.databricks.com. The Google service account that Databricks itself uses is not the principal.
• Workspace access: the service principal must be a member of each workspace it queries (an admin for admin-only resources such as workspace configuration). From SQL, with the account provider:

REPLACE databricks_account.iam.workspace_assignment
SET permissions = '["USER"]'
WHERE account_id = '<account_id>'
AND workspace_id = '<workspace_id>'
AND principal_id = '<service_principal_id>';

Azure Databricks

export DATABRICKS_CLOUD=azure
export DATABRICKS_ACCOUNT_ID=...
export DATABRICKS_CLIENT_ID=...
export DATABRICKS_CLIENT_SECRET=...
stackql shell

/* workspaces in the account, with their Azure subscription and resource group */
SELECT
  workspace_id,
  workspace_name,
  workspace_status,
  location,
  deployment_name,
  azure_workspace_info
FROM databricks_account.provisioning.workspaces
WHERE account_id = '<account_id>';

/* SQL warehouses in a workspace */
SELECT id, name, state, cluster_size, auto_stop_mins
FROM databricks_workspace.sql.warehouses
WHERE deployment_name = 'adb-1234567890123456.7';

Databricks on Google Cloud

$env:DATABRICKS_CLOUD = "gcp"
$env:DATABRICKS_ACCOUNT_ID = "..."
$env:DATABRICKS_CLIENT_ID = "..."
$env:DATABRICKS_CLIENT_SECRET = "..."
stackql.exe shell

/* workspaces in the account, with their Google project */
SELECT
  workspace_id,
  workspace_name,
  workspace_status,
  location,
  deployment_name,
  cloud_resource_container
FROM databricks_account.provisioning.workspaces
WHERE account_id = '<account_id>';

/* catalogs in a workspace */
SELECT full_name, catalog_type, owner
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = '1234567890123456.7';

What differs by cloud

• provisioning.workspaces reports the region in aws_region on AWS and in location on Azure and GCP; Azure rows carry azure_workspace_info, GCP rows cloud_resource_container and network_id.
• The infrastructure objects under account provisioning (credentials, storage, networks, vpc_endpoints, private_access, encryption_keys) and the views over them are the AWS and GCP account APIs. Azure Databricks workspaces are created through Azure Resource Manager, so none of these exist on an Azure account and Databricks answers those calls with an HTML error page rather than an empty list; on GCP, credentials answers 503.
• billing.billable_usage (the usage CSV download) returns data for AWS and GCP accounts; an Azure Databricks account returns no rows because consumption is billed through the Azure subscription. Budgets, budget policies and budget spend statuses work everywhere.
• Workspace APIs are the same on all three clouds. Resources tied to one cloud's primitives (AWS instance profiles, for example) exist only there, and where a workspace's cloud or region does not offer a feature Databricks answers with an error rather than an empty list; the provider surfaces that error as is.

The per-cloud setup is also documented on the provider pages: databricks-account-provider.stackql.io and databricks-workspace-provider.stackql.io.