Skip to main content

12 posts tagged with "infrastructure-as-code"

View All Tags

· One min read

The stackql-deploy docs site is now available, offering a comprehensive guide to using stackql-deploy for your cloud resource deployments and tests. The site includes detailed documentation, examples, and best practices to help you get started quickly and effectively.

tip

stackql-deploy is a declarative, stateless (and state file-less) infrastructure-as-code and test framework, driven by stackql queries. stackql-deploy is capable of provisioning, updating, de-provisioning and testing cloud and SaaS stacks across all cloud and SaaS providers.

stackql-deploy-github-actions-screenshot

Let us know what you think! ⭐ us on GitHub.

· One min read

stackql-deploy is now available in the GitHub Actions Marketplace.

tip

stackql-deploy is a declarative, stateless (and state file-less) infrastructure-as-code and test framework, driven by stackql queries. stackql-deploy is capable of provisioning, updating, de-provisioning and testing cloud and SaaS stacks across all cloud and SaaS providers.

Given this example stackql-deploy stack definition in a GitHub repo, you would simply add the following to your GitHub Actions workflow:

...
jobs:
stackql-actions-test:
name: StackQL Actions Test
runs-on: ubuntu-latest
env:
GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Deploy a Stack
uses: stackql/setup-deploy@v1.0.1
with:
command: build
stack-dir: examples/k8s-the-hard-way
stack-env: dev
env-vars: GOOGLE_PROJECT=stackql-k8s-the-hard-way-demo

Example output is shown here:

stackql-deploy-github-actions-screenshot

Let us know what you think! ⭐ us on GitHub.

· 5 min read

stackql-deploy is a multi-cloud resource provisioning framework using stackql. It is inspired by dbt (data build tool), which manages data transformation workflows in analytics engineering by treating SQL scripts as models that can be built, tested, and materialized incrementally. With StackQL, you can create a similar framework for cloud and SaaS provisioning. The goal is to treat infrastructure-as-code (IaC) queries as models that can be deployed, managed, and interconnected.

This ELT/model-based framework for IaC allows you to provision, test, update, and tear down multi-cloud stacks, similar to how dbt manages data transformation projects, with the benefits of version control, peer review, and automation. This approach enables you to deploy complex, dependent infrastructure components in a reliable and repeatable manner.

Features

StackQL simplifies the interaction with cloud resources by using SQL-like syntax, making it easier to define and execute complex cloud management operations. Resources are provisioned with INSERT statements, and tests are structured around SELECT statements.

Features include:

  • Dynamic state determination (eliminating the need for state files)
  • Pre-flight and post-deploy assurance tests for resources
  • Simple flow control with rollback capabilities
  • Single code base for multiple target environments
  • SQL-based definitions for resources and tests

Installing stackql-deploy

To get started with stackql-deploy, run the following:

pip install stackql-deploy

stackql-deploy will automatically download the latest release of stackql using the pystackql Python package. You can then use the info command to get runtime information:

$ stackql-deploy info
stackql-deploy version: 1.1.0
pystackql version : 3.6.1
stackql version : v0.5.612
stackql binary path : /home/javen/.local/stackql
platform : Linux x86_64 (Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35), Python 3.10.12

Project structure

A stackql-deploy project is a directory with declarative SQL definitions to provision, de-provision, or test resources in a stack. The key components and their definitions are listed here:

  • stackql_manifest.yml : The manifest file for your project, defining resources and properties in your stack.
  • stackql_resources directory : Contains StackQL queries to provision and de-provision resources in your stack.
  • stackql_tests directory : Contains StackQL queries to test the desired state for resources in your stack.

Getting started

Use the init command to create a starter project directory:

stackql-deploy init activity_monitor

You will now have a directory named activity_monitor with stackql_resources and stackql_tests directories and a sample stackql_manifest.yml file, which will help you to get started.

Usage

The general syntax for stackql-deploy is described here:

stackql-deploy [OPTIONS] COMMAND [ARGS]...

Commands include:

  • build: Create or update resources based on the defined stack.
  • teardown: Remove or decommission resources that were previously deployed.
  • test: Execute test queries to verify the current state of resources against the expected state.
  • info: Display the version information of the stackql-deploy tool and current configuration settings.
  • init: Initialize a new project structure for StackQL deployments.

Optional global options (for all commands) include:

  • --custom-registry TEXT: Specify a custom registry URL for StackQL. This URL will be used by all commands for registry interactions.
  • --download-dir TEXT: Define a download directory for StackQL where all files will be stored.
  • --help: Show the help message and exit.

Options for build, test, and teardown include:

  • --on-failure [rollback|ignore|error]: Define the action to be taken if the operation fails. Options include rollback, ignore, or treat as an error.
  • --dry-run: Perform a simulation of the operation without making any actual changes.
  • -e <TEXT TEXT>...: Specify additional environment variables in key-value pairs.
  • --env-file TEXT: Path to a file containing environment variables.
  • --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]: Set the logging level to control the verbosity of logs during execution.

Example

Using the activity_monitor stack we created previously using the init command, we can start defining a stack and defining the associated queries; here is the manifest file:

version: 1
name: activity_monitor
description: oss activity monitor stack
providers:
- azure
globals:
- name: subscription_id
description: azure subscription id
value: "{{ vars.AZURE_SUBSCRIPTION_ID }}"
- name: location
value: eastus
- name: resource_group_name_base
value: "activity-monitor"
resources:
- name: monitor_resource_group
description: azure resource group for activity monitor
props:
- name: resource_group_name
description: azure resource group name
value: "{{ globals.resource_group_name_base }}-{{ globals.stack_env }}"
# more resources would go here...

globals.stack_env is a variable referencing the user-specified environment label.

Our stackql_resources directory must contain a .iql file (StackQL query file) with the same name as each resource defined in the resources key in the manifest file. Here is an example for stackql_resources/monitor_resource_group.iql:

/*+ createorupdate */
INSERT INTO azure.resources.resource_groups(
resourceGroupName,
subscriptionId,
data__location
)
SELECT
'{{ resource_group_name }}',
'{{ subscription_id }}',
'{{ location }}'

/*+ delete */
DELETE FROM azure.resources.resource_groups
WHERE resourceGroupName = '{{ resource_group_name }}' AND subscriptionId = '{{ subscription_id }}'

Similarly, our stackql_tests directory must contain a .iql file (StackQL query file) with the same name as each resource defined in the stack. Here is an example for stackql_tests/monitor_resource_group.iql:

/*+ preflight */
SELECT COUNT(*) as count FROM azure.resources.resource_groups
WHERE subscriptionId = '{{ subscription_id }}'
AND resourceGroupName = '{{ resource_group_name }}'

/*+ postdeploy, retries=2, retry_delay=2 */
SELECT COUNT(*) as count FROM azure.resources.resource_groups
WHERE subscriptionId = '{{ subscription_id }}'
AND resourceGroupName = '{{ resource_group_name }}'
AND location = '{{ location }}'
AND JSON_EXTRACT(properties, '$.provisioningState') = 'Succeeded'

Now we can build, test, and teardown our example stack using these commands (starting with a dry-run, which will render the target queries without executing them):

# stackql-deploy build|test|teardown {stack_name} {stack_env} [{options}]
stackql-deploy build example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000 --dry-run
stackql-deploy build example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
stackql-deploy test example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
stackql-deploy teardown example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000

Give us your feedback! ⭐ us here!

· 4 min read

Understanding roles is integral to applying the principal of least privilege to GCP environments.

A quick primer on roles in GCP

A Role in GCP is a collection of permissions to services and APIs on the platform. Roles are "bound" to principals or members (users, groups and service accounts).

These bindings are referred to as "policies" which are scoped at a particular level - organisation, folder, project, resource.

There are three types of roles - Primitive Roles, Predefined Roles and Custom Roles.

Primitive (or Basic) Roles

These are legacy roles set at a GCP project level which include Owner, Editor, and Viewer. These are generally considered to be excessive in terms of permissions and their use should be minimised if not avoided altogether.

Predefined Roles

These are roles with fine grained access to discrete services in GCP. Google has put these together for your convenience. In most cases predefined roles are the preferred mechanism to assign permissions to members.

Custom Roles

Custom roles can be created with a curated collection of permissions if required, reasons for doing so include:

  • if the permissions in predefined roles are excessive for your security posture
  • if you want to combine permissions across different services, and cannot find a suitable predefined role although it is preferred to assign multiple predefined roles to a given member

Anatomy of an IAM Policy

An IAM Policy is a collection of bindings of one or more members (user, group or service account) to a role (primitive, predefined or custom). Policies are normally expressed as JSON objects as shown here:

{
"bindings": [
{
"members": [
"group:project-admins@my-cloud-identity-domain.com"
],
"role": "roles/owner"
},
{
"members": [
"serviceAccount:provisioner@my-project.iam.gserviceaccount.com",
"user:javen@avensolutions.com"
],
"role": "roles/resourcemanager.folderViewer"
}
]
}

Groups are Google Groups created in Cloud Identity or Google Workspace (formerly known as G-Suite)

Application of policies is an atomic operation, which will overwrite any existing policy attached to an entity (org, folder, project, resource).

Querying Roles with StackQL

Predefined and primitive roles are defined in the roles resource in StackQL (google.iam.roles) - which returns the following fields (as returned by DESCRIBE google.iam.roles):

NameDescription
nameName of the role in the format roles/[{service}.]{role}
for predefined or basic roles, or qualified for custom roles,
e.g. organizations/{org_id}/roles/[{service}.]{role}
descriptionAn optional, human-readable description for the role
includedPermissionsAn array of permissions this role grants (only displayed with
VIEW = 'full')
etagOutput only, used internally for consistency
titleAn optional, human-readable title for the role (visible in the
Console)
deletedA read only boolean field showing the current deleted state
of the role
stageThe current launch stage of the role, e.g. ALPHA

Get the name for a role

Often, you may know the "friendly" title for a role like "Logs Bucket Writer", but you need the actual role name to use in an Iam policy - which is roles/logging.bucketWriter. A simple query to find this using StackQL is shown here:

SELECT name
FROM google.iam.roles
WHERE title = 'Logs Bucket Writer';
/* RETURNS:
|----------------------------|
| name |
|----------------------------|
| roles/logging.bucketWriter |
|----------------------------|
*/

Conversely, if you have the name but want the friendly title you could use:

SELECT title
FROM google.iam.roles
WHERE name = 'roles/logging.bucketWriter';
/* RETURNS:
|--------------------|
| title |
|--------------------|
| Logs Bucket Writer |
|--------------------|
*/

Wildcards can also be used with the LIKE operator, for example to get the name and title for each predefined role in the logging service you could run:

SELECT name, title
FROM google.iam.roles
WHERE name LIKE 'roles/logging.%';

Get the permissions for a role

To return the includedPermissions you need to add the following WHERE clause:

WHERE view = 'FULL'

An example query to list the permissions for a given role is shown here:

SELECT includedPermissions
FROM google.iam.roles
WHERE view = 'FULL' AND
name = 'roles/cloudfunctions.viewer';
/* RETURNS
["cloudbuild.builds.get","cloudbuild.builds.list",...]
*/

A more common challenge is that you know a particular permission such as cloudfunctions.functions.get and you want to know which roles contain this permission you could run the following query:

SELECT name, title
FROM google.iam.roles
WHERE view = 'FULL'
AND includedPermissions LIKE '%cloudfunctions.functions.get%';

Creating custom roles and more...

In forthcoming articles, we will demonstrate how you can create custom roles using StackQL INSERT operations, as well as how you can construct a simple IAM framework to manage and provision access to resources in GCP, stay tuned!