17 posts tagged with "infrastructure-as-code"

stackql-deploy 2.0 is a full rewrite in Rust. The Python package (stackql-deploy on PyPi) is archived at 1.9.4. CLI interface and stack file format are unchanged - no migration required.

Why Rust​

The move to Rust was primarily about distribution and operational simplicity. Rust also brings stronger guarantees around performance and memory safety. Running everything in-process without Foreign Function Interface (FFI) boundaries simplifies the architecture while maintaining predictable resource usage.

Embedded Postgres Wire Protocol Server​

The most significant functional change in 2.0 is that stackql-deploy now runs the StackQL engine as an embedded in-process server over a local postgres wire protocol connection rather than shelling out to the StackQL binary as an external process.

There is nothing to start, stop, or configure. The server is lifecycle-managed by stackql-deploy itself and binds to localhost only - no port is exposed on the network, no inbound firewall rules needed in CI.

The previous model spawned a new StackQL process per operation. The embedded server keeps a persistent connection for the duration of a deployment run. For stacks with many resources, the reduction in process spawn overhead is noticeable - particularly on Windows where process creation is expensive.

Additional Features Added​

In addition to added the architectural change to use the embedded server, several other workflow improvements were added including:

• Enabling resource scoped variable exports in /*+ exists */ queries: When an exists query returns a named field (e.g. vpc_id) instead of count, the value is captured as a resource-scoped variable (this.vpc_id) and made available to all subsequent queries for that resource (e.g. statecheck, exports). This eliminates the need for redundant lookups to resolve resource identifiers between query stages.
• Support for capturing RETURNING payloads from DML operations: INSERT, UPDATE, and DELETE statements can include a RETURNING clause. Fields from the response can be mapped to resource-scoped variables via return_vals in the manifest, keyed by operation (create, update, delete). This allows identifiers assigned by the provider during creation to be used immediately without a round-trip query.
• Additional template filters: Including the to_aws_tag_filters filter, which converts global_tags to the AWS Resource Groups Tagging API TagFilters format, and type-preserving YAML-to-JSON serialization that maintains string types through the rendering pipeline.
• Improved logging and exception handling: Enhanced visibility simplifying troubleshooting.

Installation​

curl -L https://get-stackql-deploy.io | tar xzf -

Invoke-WebRequest https://get-stackql-deploy.io -OutFile stackql-deploy.zip
Expand-Archive stackql-deploy.zip -DestinationPath .

jobs:
  deploy:
    runs-on: ubuntu-latest
    env:
      GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
    steps:
      - uses: actions/checkout@v4
      - uses: stackql/stackql-deploy-action@v2
        with:
          command: 'build'
          stack_dir: 'examples/my-stack'
          stack_env: 'prod'
          env_vars: 'GOOGLE_PROJECT=my-project'

      - name: Deploy Stack
        id: stackql-deploy
        uses: stackql/stackql-deploy-action@v2
        with:
          command: 'build'
          stack_dir: 'examples/my-stack'
          stack_env: 'prod'
          output_file: 'deployment-outputs.json'
          env_vars: 'GOOGLE_PROJECT=my-project'

      - name: Use outputs
        run: |
          echo '${{ steps.stackql-deploy.outputs.deployment_outputs }}' | jq .

cargo install stackql-deploy

Usage​

The CLI interface is unchanged from the Python version:

# deploy a stack
stackql-deploy build my-stack prod \
  --e GOOGLE_PROJECT=${GOOGLE_PROJECT}

# test a stack
stackql-deploy test my-stack prod \
  --e GOOGLE_PROJECT=${GOOGLE_PROJECT}

# tear down a stack
stackql-deploy teardown my-stack prod \
  --e GOOGLE_PROJECT=${GOOGLE_PROJECT}

# dry run
stackql-deploy build my-stack prod \
  --e GOOGLE_PROJECT=${GOOGLE_PROJECT} \
  --dry-run

Stack files and stackql_manifest.yml structure are unaffected - no migration work needed.

Python Package Deprecation​

stackql-deploy 1.9.4 on PyPi is the final Python release. The Python source repository is archived. If you have pip install stackql-deploy in any scripts or CI pipelines, replace it with one of the install methods above. The 1.9.4 package remains on PyPi and installable, but will not receive updates.

Links​

• GitHub: github.com/stackql/stackql-deploy
• crates.io: crates.io/crates/stackql-deploy
• StackQL docs: stackql.io/docs

Updated StackQL providers for Databricks are now available: databricks_account and databricks_workspace, giving you SQL access to the full Databricks control plane across account-level and workspace-level operations.

Provider Structure​

The following updated providers are available:

Coverage​

There are over 30 services, 300+ resources, and 983 operations spanning IAM, compute, catalog, billing, jobs, ML, serving, sharing, vector search, and more.

Example Queries​

List workspaces in an account​

SELECT
  workspace_id,
  workspace_name,
  workspace_status,
  aws_region,
  compute_mode,
  deployment_name,
  datetime(creation_time/1000, 'unixepoch') as creation_date_time
FROM databricks_account.provisioning.workspaces
WHERE account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce';

Query account users and roles​

SELECT
  id as user_id,
  displayName as display_name,
  userName as user_name,
  active,
  IIF(JSON_EXTRACT(roles,'$[0].value') = 'account_admin', 'true', 'false') as is_account_admin
FROM databricks_account.iam.account_users
WHERE account_id = 'ebfcc5a9-9d49-4c93-b651-b3ee6cf1c9ce';

List catalogs in a workspace​

SELECT
  full_name,
  catalog_type,
  comment,
  datetime(created_at/1000, 'unixepoch') as created_at,
  created_by,
  datetime(updated_at/1000, 'unixepoch') as updated_at,
  updated_by,
  enable_predictive_optimization
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = 'dbc-36ff48e3-4a69';

Download billable usage to CSV​

This one is worth calling out. You can pull billable usage data for a given period and write it straight to a CSV file:

./stackql exec \
  -o text \
  --hideheaders \
  -f billable_usage.csv \
  "SELECT contents
  FROM databricks_account.billing.billable_usage
  WHERE start_month = '2025-12'
  AND end_month = '2026-01'
  AND account_id = 'your-account-id'"

Authentication​

Both providers authenticate using OAuth2 with a Databricks service principal. Set the following environment variables:

export DATABRICKS_ACCOUNT_ID="your-account-id"
export DATABRICKS_CLIENT_ID="your-client-id"
export DATABRICKS_CLIENT_SECRET="your-client-secret"

These are the same variables used by Terraform, the Databricks SDKs, and the Databricks CLI.

Get Started​

Pull the providers:

registry pull databricks_account;
registry pull databricks_workspace;

Start querying via the shell or exec:

SELECT * FROM databricks_account.iam.account_groups WHERE account_id = 'your-account-id';

Full documentation is available at databricks-account-provider.stackql.io and databricks-workspace-provider.stackql.io. Let us know what you think on GitHub.

We've released a new dedicated StackQL AWS Cloud Control provider, providing full CRUDL operations across AWS services via the Cloud Control API including purpose-built resource definitions leveraging Cloud Control's consistent schema.

Resource Naming Convention​

Resources follow a clear pattern to differentiate operations:

This separation means listing thousands of resources won't trigger rate limits from individual GET calls:

-- Fast enumeration (list operation only)
SELECT bucket_name 
FROM awscc.s3.buckets_list_only 
WHERE region = 'us-east-1';

-- Full resource details (get operation)
SELECT * 
FROM awscc.s3.buckets 
WHERE region = 'us-east-1' 
AND data__Identifier = 'my-bucket';

Provider Coverage​

The awscc provider includes:

• 237 services and 2371 resources covering the breadth of AWS
• Full CRUDL support for all Cloud Control compatible resources
• Consistent schema derived from AWS CloudFormation resource specifications

Example Operations​

Create an S3 Bucket​

INSERT INTO awscc.s3.buckets (
  BucketName,
  region
)
SELECT 
  'my-new-bucket',
  'us-east-1';

Query EC2 Instances​

SELECT 
  instance_id,
  instance_type,
  tags
FROM awscc.ec2.instances
WHERE region = 'ap-southeast-2'
AND data__Identifier = 'i-1234567890abcdef0';

Delete a Resource​

DELETE FROM awscc.lambda.functions
WHERE data__Identifier = 'my-function'
AND region = 'us-east-1';

Enhanced Documentation​

The provider documentation at awscc.stackql.io now features:

• Interactive schema explorer with expandable nested property trees
• Complete field documentation including complex object structures
• Ready-to-use SQL examples for SELECT, INSERT, and DELETE operations
• IAM permissions reference for each resource operation

Get Started​

Pull the new provider:

stackql registry pull awscc

Query your AWS resources:

stackql shell
>> SELECT region, bucket_name FROM awscc.s3.buckets_list_only WHERE region = 'us-east-1';

Let us know your thoughts! Visit us and give us a star on GitHub.

pystackql now includes a markdownkv output format optimized for LLM processing of control plane and data plane data from cloud providers.

Background​

Recent research from ImprovingAgents.com tested 11 data formats to determine which ones LLMs parse most accurately. Using 1,000 synthetic employee records and 1,000 randomized queries, they measured how well different formats preserved data integrity through LLM processing.

The results:

Markdown-KV showed a 37% improvement over CSV and 16 percentage points over JSON. The tradeoff: it uses approximately 2.7x more tokens than CSV.

What is Markdown-KV?​

Markdown-KV uses hierarchical markdown headers with code blocks for key-value pairs:

# Query Results

## Record 1

id: i-1234567890abcdef0
name: prod-web-01
region: us-east-1
instance_type: t3.large
state: running


## Record 2

id: i-0987654321fedcba0
name: staging-web-01
region: us-west-2
instance_type: t3.medium
state: stopped

The format combines clear hierarchy, explicit key-value pairs, and readability for both humans and LLMs.

Usage​

from pystackql import StackQL

stackql = StackQL()

# Query with Markdown-KV output
result = stackql.execute(
    """
    SELECT instanceId, instanceType, state, availabilityZone 
    FROM aws.ec2.instances 
    WHERE region = 'us-east-1'
    """,
    output='markdownkv'
)

# Use with LLMs
response = llm_client.complete(
    f"Identify instances that should be stopped:\n\n{result}"
)

Works in server mode too:

stackql = StackQL(server_mode=True)

result = stackql.execute(
    "SELECT name, region, encryption FROM google.storage.buckets WHERE project = 'my-project'",
    output='markdownkv'
)

When to Use It​

Markdown-KV is useful when:

• Feeding infrastructure data to LLMs for analysis, security reviews, or recommendations
• Building RAG pipelines that need to accurately retrieve and reason about infrastructure
• Accuracy matters more than token efficiency (infrastructure decisions typically do)
• Query results are focused datasets (most StackQL queries are)

The token cost is a real tradeoff, but infrastructure queries typically return targeted result sets, not massive datasets. When you're asking an LLM to analyze your production environment, accuracy matters.

Getting Started​

Update pystackql:

pip install --upgrade pystackql

Add output='markdownkv' to your execute calls or in the StackQL object instantiation:

result = stackql.execute(query, output='markdownkv')

Resources​

• ImprovingAgents.com research on LLM data formats
• pystackql v3.8.2 changelog
• pystackql documentation
• StackQL documentation

The Markdown-KV output format is available in pystackql v3.8.2 and later.

⭐ Star us on GitHub and join our community!

StackQL now supports the Model Context Protocol (MCP). This integration enables AI agents and assistants to query and manage cloud infrastructure across multiple providers using natural language.

What is the Model Context Protocol?​

The Model Context Protocol is an open standard that enables AI applications to securely connect to external data sources and tools. By running StackQL as an MCP server, AI agents like Claude, ChatGPT, and other LLM-based assistants can interact with your cloud infrastructure using StackQL's powerful SQL-based query capabilities.

Why MCP + StackQL?​

Combining MCP with StackQL creates a powerful interface for AI-assisted infrastructure management:

• Natural Language Infrastructure Queries: Ask questions about your cloud resources in plain English and get structured data back
• Multi-Cloud Support: Access resources across AWS, Google Cloud, Azure, and 100+ other providers through a single interface
• Secure and Standardized: MCP provides a secure, standardized way for AI agents to interact with your infrastructure
• SQL-Powered Analytics: Leverage StackQL's full SQL capabilities including joins, aggregations, and complex queries through AI agents

Deployment Options​

StackQL's MCP server supports three flexible deployment modes to suit different architectural requirements:

1. Standalone MCP Server​

Perfect for development and AI agent integration:

stackql mcp \
  --mcp.server.type=http \
  --mcp.config '{"server": {"transport": "http", "address": "127.0.0.1:9912"}}'

2. Dual-Protocol Server (In-Memory)​

Run both MCP and PostgreSQL wire protocol simultaneously with high-performance in-memory communication:

stackql srv \
  --mcp.server.type=http \
  --mcp.config '{"server": {"transport": "http", "address": "127.0.0.1:9912"}}' \
  --pgsrv.port 5665

This mode is ideal when you need both AI agent access and traditional database client connectivity.

3. Reverse Proxy with TLS​

For production environments requiring distributed deployments and encrypted connections:

stackql srv \
  --mcp.server.type=reverse_proxy \
  --mcp.config '{"server": {"tls_cert_file": "/path/to/cert.pem", "tls_key_file": "/path/to/key.pem", "transport": "http", "address": "127.0.0.1:9004"}, "backend": {"dsn": "postgres://stackql:stackql@127.0.0.1:5446?default_query_exec_mode=simple_protocol"}}' \
  --pgsrv.port 5446

Available MCP Tools​

When running as an MCP server, StackQL exposes several tools that AI agents can invoke:

Integration with Claude Desktop​

To integrate StackQL with Claude Desktop, add this configuration to your MCP settings file (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "stackql": {
      "command": "stackql",
      "args": [
        "mcp",
        "--mcp.server.type=http",
        "--mcp.config",
        "{\"server\": {\"transport\": \"http\", \"address\": \"127.0.0.1:9912\"}}"
      ]
    }
  }
}

Example Use Cases​

Once configured, you can ask your AI assistant questions like:

• "Show me all my EC2 instances across all AWS regions"
• "List all Google Cloud Storage buckets with public access"
• "Find all Azure virtual machines that haven't been updated in 30 days"
• "Compare compute costs across AWS, Azure, and GCP"
• "Show me IAM policies that grant admin access in my Google Cloud projects"

The AI agent will use StackQL's MCP server to execute the appropriate queries and return structured results.

Example Query Flow​

Here's how an AI agent interacts with StackQL via MCP:

# AI agent lists available providers
Tool: list_providers
Response: ["google", "aws", "azure", "github", ...]

# AI agent explores a provider's services
Tool: list_services
Args: {"provider": "google"}
Response: ["compute", "storage", "cloudresourcemanager", ...]

# AI agent executes a query
Tool: query_v2
Args: {"sql": "SELECT name, status FROM google.compute.instances WHERE project = 'my-project' AND zone = 'us-east1-a'"}
Response: [{"name": "instance-1", "status": "RUNNING"}, ...]

Getting Started​

1. Download StackQL version 0.9.250 or later from stackql.io/install

2. Set up provider authentication:

export GOOGLE_CREDENTIALS=$(cat /path/to/credentials.json)
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key

3. Start the MCP server:

stackql mcp \
  --mcp.server.type=http \
  --mcp.config '{"server": {"transport": "http", "address": "127.0.0.1:9912"}}'

4. Configure your AI assistant to use the StackQL MCP server (see MCP documentation for details)

Documentation​

For comprehensive documentation on configuring and using the MCP server, including:

• Detailed configuration options
• TLS/mTLS setup
• Architecture considerations
• Testing and troubleshooting

Visit the MCP command documentation.

What's Next?​

We're actively developing additional MCP capabilities and welcome your feedback. Future enhancements may include:

• Enhanced resource provisioning and lifecycle management through MCP
• Built-in prompt templates for common infrastructure queries
• Extended tool catalog for specialized operations
• Support for additional MCP transport protocols

Try It Out!​

The MCP server feature is available now in StackQL 0.9.250. We'd love to hear about your experiences integrating StackQL with AI agents. Share your use cases, provide feedback, or contribute to the project on GitHub.

⭐ Star us on GitHub and join our community!