Skip to main content

51 posts tagged with "stackql"

View All Tags

· 5 min read

AWS creates default VPCs in each region for convenience. However, these default VPCs often contain noncompliant network ACLs and security group rules that do not align with best practices for AWS Config and Security Hub. Deleting these default VPCs is beneficial, especially for regions not used by your organization or architectures that do not utilize a VPC.  

This guide demonstrates how to use StackQL to enumerate and delete all default VPCs and their associated resources in all AWS regions.  

What you need

All you need to do is to install the pystackql package using:

pip install pystackql

Deleting resources will require a privileged AWS IAM user. You can do this from a terminal with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables set (and optionally AWS_SESSION_TOKEN set if you are using sts assume-role).  

You can also use the StackQL Cloud Shell Scripts, from within AWS Cloud Shell, to run authenticated StackQL queries using:

sh stackql-aws-cloud-shell.sh

How it works

Default VPCs in AWS regions have a CIDR block of 172.31.0.0/16, before deleting anything, the program ensures it is not in use by using:

SELECT
id
FROM aws.ec2.network_interfaces
WHERE region = '{region}'
AND vpc_id = '{vpc_id}'

If any resources are using the VPC (such as EC2, RDS, ELB/ALB, VPC attached Lambda functions, etc), it will skip these VPCs.  Once determined that the VPC is not in use, all of the resources associated with the VPC must be deleted before the VPC itself can be deleted; this includes:

  • External Routes (aws.ec2.routes)
  • Internet Gateways (aws.ec2.internet_gateways)
  • NACLs (aws.ec2.network_acls)
  • Subnets (aws.ec2.subnets)
  • and then finally, the VPC (aws.ec2.vpcs)

The default route table and default security group are deleted automatically when deleting the VPC

This is done by discovering the resources using StackQL SELECT queries and deleting them using StackQL DELETE queries, like:

DELETE FROM aws.ec2.network_acls
WHERE data__Identifier = '{nacl_id}'
AND region = '{region}'

Complete code

The following Python program uses StackQL to list and delete default VPCs and their associated resources (subnets, route tables, internet gateways, security groups, and network ACLs) across all AWS regions.

Get the complete code here
from pystackql import StackQL
stackql = StackQL()
stackql.executeStmt("REGISTRY PULL aws")

def ensure_one_or_zero(resource_list, resource_name, region):
if len(resource_list) > 1:
raise RuntimeError(f"ah snap! multiple default {resource_name} resources found in {region}")
elif len(resource_list) == 0:
print(f"/* no default {resource_name} found in {region} */\n")
return False
return True

regions = [
'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2',
'eu-central-1', 'eu-central-2', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'eu-north-1', 'eu-south-1',
'ap-east-1', 'ap-south-1', 'ap-south-2', 'ap-northeast-1', 'ap-northeast-2', 'ap-northeast-3',
'ap-southeast-1', 'ap-southeast-2', 'ap-southeast-3', 'ap-southeast-4', 'af-south-1',
'ca-central-1', 'me-south-1', 'me-central-1', 'sa-east-1'
]

for region in regions:
vpc_ids = stackql.execute(
f"""
SELECT
vpc_id
FROM aws.ec2.vpcs
WHERE region = '{region}'
AND cidr_block = '172.31.0.0/16'
AND JSON_ARRAY_LENGTH(tags) = 0
"""
)
if not ensure_one_or_zero(vpc_ids, 'VPC', region): continue
vpc_id = vpc_ids[0]['vpc_id']

# check if vpc is in use
network_interfaces = stackql.execute(
f"""
SELECT
id
FROM aws.ec2.network_interfaces
WHERE region = '{region}'
AND vpc_id = '{vpc_id}'
"""
)
if len(network_interfaces) > 0:
print(f"/* skipping deletion of default VPC ({vpc_id}) in {region} because it is in use */\n")
continue

print(f"/* deleting resources for default VPC ({vpc_id}) in {region} */\n")

# get route table
route_table_ids = stackql.execute(
f"""
SELECT
route_table_id
FROM aws.ec2.route_tables
WHERE region = '{region}'
AND vpc_id = '{vpc_id}'
"""
)
ensure_one_or_zero(route_table_ids, 'route table', region)
route_table_id = route_table_ids[0]['route_table_id']

# get inet gateway id
inet_gateway_ids = stackql.execute(
f"""
SELECT gateway_id
FROM aws.ec2.routes
WHERE data__Identifier = '{route_table_id}|0.0.0.0/0'
AND region = '{region}'
"""
)
ensure_one_or_zero(inet_gateway_ids, 'internet gateway', region)
inet_gateway_id = inet_gateway_ids[0]['gateway_id']

# delete routes
print(f"/* deleting default VPC routes in route table ({route_table_id}) in {region} */")
print(f"""
DELETE FROM aws.ec2.routes
WHERE data__Identifier = '{route_table_id}|0.0.0.0/0'
AND region = '{region}';
""")

# detatch inet gateway
print(f"/* detaching default VPC internet gateway ({inet_gateway_id}) in {region} */")
print(f"""
DELETE FROM aws.ec2.vpc_gateway_attachments
WHERE data__Identifier = 'IGW|{vpc_id}'
AND region = '{region}';
""")

# delete inet gateway
print(f"/* deleting default VPC internet gateway ({inet_gateway_id}) in {region} */")
print(f"""
DELETE FROM aws.ec2.internet_gateways
WHERE data__Identifier = '{inet_gateway_id}'
AND region = '{region}';
""")

# delete nacl
nacl_ids = stackql.execute(
f"""
SELECT
id
FROM aws.ec2.network_acls
WHERE vpc_id = '{vpc_id}'
AND region = '{region}'
"""
)
ensure_one_or_zero(nacl_ids, 'network acl', region)
nacl_id = nacl_ids[0]['id']
print(f"/* deleting default VPC NACL ({nacl_id}) in {region} */")
print(f"""
DELETE FROM aws.ec2.network_acls
WHERE data__Identifier = '{nacl_id}'
AND region = '{region}';
""")

# delete subnets
subnet_ids = stackql.execute(
f"""
SELECT
subnet_id
FROM aws.ec2.subnets
WHERE vpc_id = '{vpc_id}'
AND region = '{region}'
"""
)
for subnet_id in subnet_ids:
print(f"/* deleting default VPC subnet ({subnet_id['subnet_id']}) in {region} */")
print(f"""
DELETE FROM aws.ec2.subnets
WHERE data__Identifier = '{subnet_id['subnet_id']}'
AND region = '{region}';
""")

# delete vpc
print(f"/* deleting default VPC ({vpc_id}) in {region} */")
print(f"""
DELETE FROM aws.ec2.vpcs
WHERE data__Identifier = '{vpc_id}'
AND region = '{region}';
""")

run the program using:

python3 delete-all-default-vpcs.py > delete_default_vpcs.iql

to generate a StackQL script you can run as a batch using stackql exec or as individual statements in the stackql shell.

Conclusion

Deleting default VPCs can help improve your AWS security posture by removing potentially noncompliant network configurations. This program leverages StackQL to automate the process, ensuring consistency across all regions.

Let us know what you think! ⭐ us on GitHub.

· One min read

New versions of the azure providers for stackql are available now in the stackql-provider-registry.

Summary stats for the main azure provider:

Total Services196
Total Resources4,020
Total Methods11,160

New versions of the azure_extras, azure_isv and azure_stack providers are available as well.

New services available include:

  • Microsoft Entra Verified ID
  • Database Watcher for Azure SQL
  • Azure Compute Fleet
  • Azure Edge Zones
  • Azure Standby Pools
  • Informatica Intelligent Data Management Cloud
  • MongoDB Atlas on Azure
  • Oracle Database Service for Azure
  • Split Feature Data Platform

Let us know what you think! ⭐ us on GitHub.

· 2 min read

Analyzing firewall rules is crucial for maintaining security in your cloud infrastructure. Using StackQL, you can efficiently query and analyze Google Cloud firewall configurations to ensure that your security policies are correctly implemented and that there are no unexpected open ports or protocols that might pose a security risk. Below is a simple query that retrieves important details about the ingress firewall rules for a specific network in a Google Cloud project.

SELECT 
name,
source_range,
ip_protocol,
allowed_ports,
direction
FROM (
SELECT
name,
source_ranges.value as source_range,
JSON_EXTRACT(allowed.value, '$.IPProtocol') as ip_protocol,
JSON_EXTRACT(allowed.value, '$.ports') as allowed_ports,
direction
FROM google.compute.firewalls, json_each(sourceRanges) as source_ranges, json_each(allowed) as allowed
WHERE project = 'stackql-k8s-the-hard-way-demo'
AND network = 'https://www.googleapis.com/compute/v1/projects/stackql-k8s-the-hard-way-demo/global/networks/kubernetes-the-hard-way-dev-vpc'
) t
WHERE
source_range = '0.0.0.0/0'
and direction = 'INGRESS';

This query provides a comprehensive list of all ingress firewall rules that apply to any IP address (0.0.0.0/0) within the specified Google Cloud project and network. The results include the firewall rule name, the source IP range, the protocol, the allowed ports, and the direction of the traffic, an example is shown below:

|-----------------------------------------------|--------------|-------------|---------------|-----------|                                                                                         
| name | source_range | ip_protocol | allowed_ports | direction |
|-----------------------------------------------|--------------|-------------|---------------|-----------|
| default-allow-icmp | 0.0.0.0/0 | icmp | null | INGRESS |
|-----------------------------------------------|--------------|-------------|---------------|-----------|
| default-allow-rdp | 0.0.0.0/0 | tcp | ["3389"] | INGRESS |
|-----------------------------------------------|--------------|-------------|---------------|-----------|
| default-allow-ssh | 0.0.0.0/0 | tcp | ["22"] | INGRESS |
|-----------------------------------------------|--------------|-------------|---------------|-----------|
| kubernetes-the-hard-way-dev-allow-external-fw | 0.0.0.0/0 | tcp | ["22"] | INGRESS |
|-----------------------------------------------|--------------|-------------|---------------|-----------|
| kubernetes-the-hard-way-dev-allow-external-fw | 0.0.0.0/0 | tcp | ["6443"] | INGRESS |
|-----------------------------------------------|--------------|-------------|---------------|-----------|
| kubernetes-the-hard-way-dev-allow-external-fw | 0.0.0.0/0 | icmp | null | INGRESS |
|-----------------------------------------------|--------------|-------------|---------------|-----------|

You can use this query to help quickly identify potential security vulnerabilities. Regularly auditing these rules ensures that your cloud environment remains secure and that only the necessary ports and protocols are open to the internet.

Give us your feedback! ⭐ us here!

· One min read

We have released the latest StackQL provider for Google, which includes:

  • 14 new services (including alloydb, apphub, biglake, bigquerydatapolicy, looker and more)
  • 231 new resources
  • 1,185 new methods

More information is available here. Run the following to install or update the Google provider:

-- run from stackql shell
REGSITRY PULL google;

or

# from the command line
stackql registry pull google

Give us your feedback! ⭐ us here!

· 5 min read

stackql-deploy is a multi-cloud resource provisioning framework using stackql. It is inspired by dbt (data build tool), which manages data transformation workflows in analytics engineering by treating SQL scripts as models that can be built, tested, and materialized incrementally. With StackQL, you can create a similar framework for cloud and SaaS provisioning. The goal is to treat infrastructure-as-code (IaC) queries as models that can be deployed, managed, and interconnected.

This ELT/model-based framework for IaC allows you to provision, test, update, and tear down multi-cloud stacks, similar to how dbt manages data transformation projects, with the benefits of version control, peer review, and automation. This approach enables you to deploy complex, dependent infrastructure components in a reliable and repeatable manner.

Features

StackQL simplifies the interaction with cloud resources by using SQL-like syntax, making it easier to define and execute complex cloud management operations. Resources are provisioned with INSERT statements, and tests are structured around SELECT statements.

Features include:

  • Dynamic state determination (eliminating the need for state files)
  • Pre-flight and post-deploy assurance tests for resources
  • Simple flow control with rollback capabilities
  • Single code base for multiple target environments
  • SQL-based definitions for resources and tests

Installing stackql-deploy

To get started with stackql-deploy, run the following:

pip install stackql-deploy

stackql-deploy will automatically download the latest release of stackql using the pystackql Python package. You can then use the info command to get runtime information:

$ stackql-deploy info
stackql-deploy version: 1.1.0
pystackql version : 3.6.1
stackql version : v0.5.612
stackql binary path : /home/javen/.local/stackql
platform : Linux x86_64 (Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35), Python 3.10.12

Project structure

A stackql-deploy project is a directory with declarative SQL definitions to provision, de-provision, or test resources in a stack. The key components and their definitions are listed here:

  • stackql_manifest.yml : The manifest file for your project, defining resources and properties in your stack.
  • stackql_resources directory : Contains StackQL queries to provision and de-provision resources in your stack.
  • stackql_tests directory : Contains StackQL queries to test the desired state for resources in your stack.

Getting started

Use the init command to create a starter project directory:

stackql-deploy init activity_monitor

You will now have a directory named activity_monitor with stackql_resources and stackql_tests directories and a sample stackql_manifest.yml file, which will help you to get started.

Usage

The general syntax for stackql-deploy is described here:

stackql-deploy [OPTIONS] COMMAND [ARGS]...

Commands include:

  • build: Create or update resources based on the defined stack.
  • teardown: Remove or decommission resources that were previously deployed.
  • test: Execute test queries to verify the current state of resources against the expected state.
  • info: Display the version information of the stackql-deploy tool and current configuration settings.
  • init: Initialize a new project structure for StackQL deployments.

Optional global options (for all commands) include:

  • --custom-registry TEXT: Specify a custom registry URL for StackQL. This URL will be used by all commands for registry interactions.
  • --download-dir TEXT: Define a download directory for StackQL where all files will be stored.
  • --help: Show the help message and exit.

Options for build, test, and teardown include:

  • --on-failure [rollback|ignore|error]: Define the action to be taken if the operation fails. Options include rollback, ignore, or treat as an error.
  • --dry-run: Perform a simulation of the operation without making any actual changes.
  • -e <TEXT TEXT>...: Specify additional environment variables in key-value pairs.
  • --env-file TEXT: Path to a file containing environment variables.
  • --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]: Set the logging level to control the verbosity of logs during execution.

Example

Using the activity_monitor stack we created previously using the init command, we can start defining a stack and defining the associated queries; here is the manifest file:

version: 1
name: activity_monitor
description: oss activity monitor stack
providers:
- azure
globals:
- name: subscription_id
description: azure subscription id
value: "{{ vars.AZURE_SUBSCRIPTION_ID }}"
- name: location
value: eastus
- name: resource_group_name_base
value: "activity-monitor"
resources:
- name: monitor_resource_group
description: azure resource group for activity monitor
props:
- name: resource_group_name
description: azure resource group name
value: "{{ globals.resource_group_name_base }}-{{ globals.stack_env }}"
# more resources would go here...

globals.stack_env is a variable referencing the user-specified environment label.

Our stackql_resources directory must contain a .iql file (StackQL query file) with the same name as each resource defined in the resources key in the manifest file. Here is an example for stackql_resources/monitor_resource_group.iql:

/*+ createorupdate */
INSERT INTO azure.resources.resource_groups(
resourceGroupName,
subscriptionId,
data__location
)
SELECT
'{{ resource_group_name }}',
'{{ subscription_id }}',
'{{ location }}'

/*+ delete */
DELETE FROM azure.resources.resource_groups
WHERE resourceGroupName = '{{ resource_group_name }}' AND subscriptionId = '{{ subscription_id }}'

Similarly, our stackql_tests directory must contain a .iql file (StackQL query file) with the same name as each resource defined in the stack. Here is an example for stackql_tests/monitor_resource_group.iql:

/*+ preflight */
SELECT COUNT(*) as count FROM azure.resources.resource_groups
WHERE subscriptionId = '{{ subscription_id }}'
AND resourceGroupName = '{{ resource_group_name }}'

/*+ postdeploy, retries=2, retry_delay=2 */
SELECT COUNT(*) as count FROM azure.resources.resource_groups
WHERE subscriptionId = '{{ subscription_id }}'
AND resourceGroupName = '{{ resource_group_name }}'
AND location = '{{ location }}'
AND JSON_EXTRACT(properties, '$.provisioningState') = 'Succeeded'

Now we can build, test, and teardown our example stack using these commands (starting with a dry-run, which will render the target queries without executing them):

# stackql-deploy build|test|teardown {stack_name} {stack_env} [{options}]
stackql-deploy build example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000 --dry-run
stackql-deploy build example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
stackql-deploy test example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000
stackql-deploy teardown example_stack prd -e AZURE_SUBSCRIPTION_ID 00000000-0000-0000-0000-000000000000

Give us your feedback! ⭐ us here!