Skip to main content

13 posts tagged with "multicloud"

View All Tags

· 2 min read

Proud to announce the release of the first version of our middleware server and playground for StackQL.

StackQL allows you to query and interact with cloud services and APIs using SQL grammar and an ORM which is a direct reflection of a provider API, no database is required or implemented

StackQL Middleware Server​

Our middleware solution allows you to use StackQL as a query language to interact with APIs, much like GraphQL - however, the query DSL is SQL, providing a friendlier, more data-centric experience for developers. As shown in the example below, developers can POST queries to a /stackql endpoint; the queries are parsed and executed by a StackQL runner via the middleware server.

As the StackQL middleware API server and runner are stateless, the solution is horizontally scalable. With cache and authorization to be implemented, the solution provides a flexible, robust, scalable, performant back end for applications.

Furthermore, using SQL semantics, developers can JOIN data between API providers and perform projections, filtering, aggregation, windowing operations, and more on simple or complex data types.

StackQL Playground​

The StackQL Playground is a TypeScript app that connects to a StackQL Middleware Server, which provides access to backend APIs using SQL. Features of the playground include...

Explore API Providers​

The StackQL Playground allows you to explore and query providers, services, resources, fields, and methods.

Submit Queries​

Using the Playground you can submit queries to the /stackql endpoint of the StackQL Middleware Server.

View, Sort, and Filter Results​

Query results can be sorted or filtered in the grid result set in the StackQL Playground app, JSON results can also be viewed or copied from the JSON results tab.

Save Results as CSV or JSON​

Users can save results to local CSV or JSON files as well.

Generate Types​

Furthermore, after modeling a query you can export the TypeScript types using the Get Types button.

Huge thanks to Yuncheng Yang for the work he put in on this!

You can find the complete code here to launch an environment using docker-compose, which includes the StackQL Middleware Server, a StackQL runner (runs the queries on the back end), and the StackQL Playground app.

Let us know what you think.

· One min read

A new version of the Azure provider for StackQL plus additional built-in functions are available now.

Version 0.3.0 of the Azure provider for StackQL is available now. This update includes support for extended resource properties, along with support for Hybrid Azure Kubernetes Services. The Azure provider allows you to query across your Azure estate for cloud security posture, asset inventory, analysis and reporting, finops, sysops, and more - all using a natural query language (SQL) and a natural object mapping.

In addition, we have announced the release of several new built-in functions, including SPLIT_PART() - to split a string by a delimiter and extract a single element, additional unicode functions, and expanded regular expression support, including REGEXP_REPLACE() and more.

An example StackQL query using the split_part() function with the azure v0.3.0 provider is shown here: 

SELECT name,  
split_part(id, '/', 3) as subscription,
split_part(id, '/', 5) as resource_group,
json_extract(properties, '$.hardwareProfile.vmSize') as vm_size
FROM azure.compute.virtual_machines WHERE resourceGroupName = 'stackql-ops-cicd-dev-01' AND subscriptionId = '242c6a2d-16f9-4912-90f6-59b1cf85509d';

You can find more information on the latest Azure provider here.

· 5 min read

Pleased to announce the initial release of the AWS provider for StackQL.

StackQL allows you to query, provision, and manage cloud and SaaS resources using a simple, SQL-based framework.

The initial release of the AWS provider covers EC2, S3, and the Cloud Control API - with support for other services to be released soon. The documentation for the StackQL AWS provider is available here.

Follow the steps below to get started querying AWS in the StackQL interactive command shell:

Authenticate and Connect​

Connect to an authenticated shell using the syntax shown below: 

# AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID should be set as environment variables
AUTH="{ \"aws\": { \"type\": \"aws_signing_v4\", \"credentialsenvvar\": \"AWS_SECRET_ACCESS_KEY\", \"keyID\": \"${AWS_ACCESS_KEY_ID}\" }}"
stackql shell --auth="${AUTH}"

Download the AWS provider​

Download the AWS provider from the StackQL Provider Registry: 

REGISTRY PULL aws v0.1.3;

Explore the AWS provider​

Explore the AWS provider using StackQL metacommands (such as SHOW and DESCRIBE), for example...

Show available services​

Show the services available in the StackQL AWS provider: 

SHOW SERVICES IN aws;

Show available resources​

Show the resources available in the AWS EC2 service (filtered by a fuzzy match on instances): 

SHOW RESOURCES IN aws.ec2 LIKE '%instances%';

Show 'selectable' fields​

Show the 'selectable' fields available in a resource: 

DESCRIBE EXTENDED aws.ec2.instances;

Show operations available​

Show the available operations on a resource: 

SHOW EXTENDED METHODS IN aws.ec2.instances;

Run some queries​

Now that you've identified the available resources and fields let's run some queries!

Instances by region (across multiple regions)​

SELECT 'N. Virginia' as region, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'us-east-1'
UNION
SELECT 'N. California' as region, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'us-west-1'
UNION
SELECT 'Sydney' as region, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'ap-southeast-2';

Instances grouped by instanceType​

SELECT instanceType, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'ap-southeast-2'
GROUP BY instanceType;

Instances grouped by instanceState​

SELECT instanceState, COUNT(*) as num_instances
FROM aws.ec2.instances
WHERE region = 'ap-southeast-2'
GROUP BY instanceState;

Enjoy!

· One min read

Proud to announce the release of the Microsoft Azure provider for StackQL.

StackQL allows you to query and interact with your cloud and SaaS assets using a simple SQL framework

The StackQL provider for Azure provides key visibility across the Azure estate for CSPM, asset inventory and analysis, finops and more, as well as our IaC and ops (lifecycle management) functionality.

Created using the Autorest project using Azure specification docs from the azure-rest-api-specs repository, the StackQL azure provider exposes 230 services, 2,450 resources (of which 1,985 or 81% are available using SELECT statements) and 10,140 methods in total.

Core services are available in the azure provider, all other services are available using the azure_extras provider.

We will be adding integrated interactive authentication, for now this is cli/sdk based, all of the documentation is here.

Give it a test run and let us know what you think!

· 2 min read

Excited to announce the release of the Kubernetes provider for StackQL.

StackQL allows you to query and interact with your cloud and SaaS assets using a simple SQL framework

The k8s provider can be used to query and interact with events, namespaces, nodes, persistent volumes, pvcs, pods, services, service accounts and more.

Complete provider documentation for all of the Kubernetes resources, properties and methods is available here.

Here are the steps to get started with the Kubernetes provider:

Setup​

If you are using a proxy (kubectl proxy), follow these instructions:

Using a proxy​

  1. Open an interactive shell (authentication will be handled using the proxy and your .kube/config): 
AUTH='{ "k8s": { "type": "null_auth" } }'
./stackql shell --auth="${AUTH}"
  1. Pull the latest k8s provider for StackQL:
REGISTRY PULL k8s v0.1.1;
  1. Query away adding the following expressions to WHERE clauses in your k8s queries:
  • protocol = 'http'
  • cluster_addr = 'localhost:8080' (or whatever port your proxy is listening on)
select name, namespace, uid, creationTimestamp 
from k8s.core_v1.pod 
where protocol = 'http' 
and cluster_addr = 'localhost:8080'  
order by name asc limit 3;

Direct cluster access​

  1. Generate an access token for your cluster, see Access Clusters Using the Kubernetes API.

  2. Generate a certificate bundle for your cluster using the following code (for MacOS or Linux): 

kubectl get secret -o jsonpath="{.items[?(@.type==\"kubernetes.io/service-account-token\")].data['ca\.crt']}" | base64 -i --decode > k8s_cert_bundle.pem
note

Alternatively, you could add the --tls.allowInsecure=true argument to the stackql command, it is not recommended however.

  1. Export the token to a variable and supply this as the provider authentication for StackQL: 
export K8S_TOKEN='eyJhbGciOi...'
AUTH='{ "k8s": { "type": "api_key", "valuePrefix": "Bearer ", "credentialsenvvar": "K8S_TOKEN" } }'
./stackql shell --auth="${AUTH}" --tls.CABundle k8s_cert_bundle.pem
  1. Pull the latest k8s provider for StackQL:
REGISTRY PULL k8s v0.1.1;
  1. Run some queries (provide the cluster_addr as a WHERE clause parameter): 
select name, namespace, uid, creationTimestamp 
from k8s.core_v1.service_account 
where cluster_addr = '35.244.65.136' 
and namespace = 'kube-system' 
order by name asc;

Welcome your feedback by getting in touch or raising issues at stackql/stackql-provider-registry, give us some ⭐️ love while you are there!