awless.io, the mighty CLI for AWS
Some very clearly prefer the CLI (source: Imgur)
Disclaimer : this is a translation of the article, the original can be found here .
While cloud computing has gained popularity, the Amazon Web Services (AWS) platform, which provides the work of many Internet giants today and sets the standards for how many cloud services should be organized, has become the undisputed leader in the cloud computing market. To understand how to take advantage of the ever-growing number of services and the opportunities they provide, a DevOps engineer must keep up with the pace of cloud technology that Amazon sets, and this becomes a real test.
DevOps often wonder how to better manage the rapidly growing AWS infrastructure. They are considering whether it is better to use the GUI ( AWS console ) or the CLI ( AWS CLI ) for this. But can it be used both?
We present to your attention the third option created to facilitate the work of DevOps: awless.io - the new open source CLI for AWS, written in Go. The project was approved by Jeff Barr, the AWS rock star, and in just a few months gathered over 2.5K stars on GitHub. In this article, I will explain why Wallix, the company I work for, created awless.io, and why we believe this tool can help DevOps.
What, by the way, are these tweets talking about:
Love and hate GUI
AWS provides many services
Sysadmins tend to hate GUIs. The Linux Foundation considers them the fate of beginners. Is AWS Console an exception? Console provides the ability to manage all available services and allows you to use the setup wizards to solve a variety of tasks. We appreciate the opportunities provided by Console, especially when performing one-time tasks, as process visualization in this case is very useful. But solving standard, repetitive tasks using the GUI takes too much time and can lead experienced users out of themselves.
We’re in the sixth step in the AWS Console and we still haven’t created an instance
Using the GUI, it’s difficult to automate actions, leave notes for yourself, or interact with a colleague to perform an operation. Remember the last time you explained to a colleague (or, for example, mom) on the phone how to do something using the web interface, while digging through it? This is really annoying.
Difficulties aws CLI
The CLI, on the other hand, is great for routine tasks. After pre-configuring with the CLI, you can do anything without manually entering login information and without touching the mouse.
For example, this command is enough to run the t1.micro instance:
find \
-name "sws"
> aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t1.micro \
--key-name keypair-xxxxxxxx --security-groups sg-xxxxxxxxNote that remembering the Amazon Machine Image Identifier (AMI) or key pair name may not be easy, but this problem can be solved using the alias shell command . For example, to begin with, we find the desired security group:
> aws ec2 describe-security-groups --group-name "awless-secgroup"and grab the group ID from the output, which by default should look something like this:
{
"SecurityGroups": [
{
"IpPermissionsEgress": [
{
"PrefixListIds": [],
"FromPort": 80,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 80,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
},
{
"PrefixListIds": [],
"FromPort": 443,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 443,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"Description": "awless-secgroup",
"IpPermissions": [
{
"PrefixListIds": [],
"FromPort": 22,
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"ToPort": 22,
"IpProtocol": "tcp",
"UserIdGroupPairs": [],
"Ipv6Ranges": []
}
],
"GroupName": "awless-secgroup",
"VpcId": "vpc-00b68c65",
"OwnerId": "519101999238",
"GroupId": "sg-687a5d0e"
}
]
}Although this approach seems more “advanced” compared to using the AWS Console, it is not always convenient and efficient. When using aws CLI, you have to constantly switch from entering commands to performing other actions, such as copying the output of previous commands or editing JSON snippets. Existing open source alternatives to aws CLI only slightly change the syntax of operations, add highlighting, auto-completion and other whistles.
Tools such as AWS CloudFormation are convenient for creating stacks, but writing the appropriate templates is a separate task, and the templates themselves cannot always replace the manual execution of several commands. Once we were tormented, trying to find resources created using CloudFormation templates using the aws CLI. Guessing the right place in kilobytes of JSON where Id needs to be written is not an easy task. In the end, we gave up and used the GUI.
Phenomenon awless.io
awless.io is an alternative CLI for AWS, in which ease of use is highlighted in front of exhaustive functionality. With this tool, we strive to simplify 90 percent of the possible tasks by changing existing teams.
awless.io is not just an AWS CLI frontend: it is a new console interface written in Go using the official AWS Go SDK. This is a fully open source project released under the Apache license. Awless.io currently supports the following AWS services: IAM, EC2 (including ELBv2), S3, Route53, RDS, SQS, SNS, Cloudfront, Cloudwatch, CloudFormation and Lambda. In the near future it is planned to expand this list.
Originally, awless.io was developed by my Wallix Innovation Department to address a number of internal challenges. De facto Wallix sponsored the development, but essentially awless.io is a project created by the development community, and your participation in it is strongly encouraged.
Benefits of awless.io
When creating awless.io, we tried to follow the principles laid down by popular tools such as git; most teams have the form:
awless [verb] [noun] [parameter=value ...]For example, all these are possible commands:
# Показать существующих пользователей
awless list users
# Создать новый инстанс
awless create instance
# Добавить пользователя в группу
awless attach user name=my-user-name group=my-group
# Запустить шаблон
awless run https://example.com/create_vpc.awsThe core principles of awless.io include:
- the use of "smart" default settings that will allow you to use the tool even with minimal knowledge of cloud technology
- information output in a format that is both readable and convenient for parsing
- local saving of a copy of the infrastructure model in the form of an RDF graph (Resource Description Framework), which allows you to access it and perform calculations not remotely
awless.io focuses on using the same verbs (such as list , create , delete ) in the syntax of different operations, which results in a small (and easy to remember) set of commands. The syntax is unified for different services. For example, you can write:
awless list [instances|users|buckets|records]instead:
aws ec2 describe-instances
aws iam list-users
aws s3api list-buckets
aws route53 list-resource-record-setsGetting started with awless.io
awless.io available on GitHub . We also distribute it as compiled packages and through Homebrew for macOS:
brew tap wallix/awless; brew install awlessBuilding from source is very simple, provided that Go is installed on the system:
go get -u github.com/wallix/awlessAt the first start, awless.io will ask for credentials or, if aws-cli is installed and configured on the system, it will automatically pull them out of its configuration. In addition, it is worth installing auto-completion for bash or zsh .
Exploring AWS Infrastructure
awless.io provides DevOps with a wide range of options for navigating the deployed AWS infrastructure. The two main commands for this are list and show . For each type of resource, awless.io first displays the most relevant information.
Listing Objects
awless.io allows you to list all kinds of AWS objects, such as users, security groups, or instances:
> awless list users
| ID ▲ | NAME | LASTUSED | CREATED |
|-----------------------|-----------|----------|-----------|
| AIDAI37GETCRRVNLDKFYQ | hbinsztok | 8 days | 10 months |
| AIDAI3FX74KAQQKIOM2XY | awless-ts | 13 weeks | 4 months |
[…]
If the width of the terminal does not allow to display the data completely, then the columns are cut off automatically, and a warning is displayed that such information was hidden:
> awless list securitygroups
| ID ▲ | VPC | INBOUND |
|-------------|--------------|-------------------------------------|
| sg-1d578b64 | vpc-00b68c65 | [0.0.0.0/0](tcp:1433) |
| | | [0.0.0.0/0;::/0](tcp:443) |
| | | [::/0;0.0.0.0/0](tcp:80) |
| | | [::/0;0.0.0.0/0](tcp:2242) |
| | | [::/0;0.0.0.0/0](tcp:22) |
| | | [0.0.0.0/0](tcp:3389) |
| sg-30abc756 | vpc-00b68c65 | [0.0.0.0/0](tcp:443) |
| | | [0.0.0.0/0](tcp:80) |
[...]
Columns truncated to fit terminal: 'Outbound', 'Name', 'Description'
The output is easily and naturally sorted and filtered:
> awless list instance --sort uptime --filter zone=eu-west-1b
| ID | ZONE | NAME | STATE |
|---------------------|------------|---------------------|---------|
| i-03a1f22b36ddb0739 | eu-west-1b | blog-test | running |
| i-05adde5a930da5ed3 | eu-west-1b | authoringtest | stopped |
| i-06b8f07c1e4afb49a | eu-west-1b | AwlessWithScheduler | running |
[...]
Columns truncated to fit terminal: 'Type', 'Public IP', 'Private IP', 'Uptime', 'KeyPair'
The output format can be customized. By default, data is output to a human-readable table (the result is output to markdown, which can be passed as input to a third-party tool, for example, to pandoc ). Data can also be output in csv, json or tsv formats:
> awless list subnets --format csv
ID,Name,CIDR,Zone,Default,Vpc,Public,State
subnet-0c41ad68,,172.31.0.0/20,eu-west-1a,true,vpc-00b68c65,true,available
subnet-2486b17c,demo-env-subnet,10.0.0.0/24,eu-west-1c,false,vpc-67b25c00,true,available
[...]
One of the distinguishing features of awless.io is the ability to work with the infrastructure, presented as an RDF graph, and perform complex queries locally. Using a standardized format allows you to use existing tools, such as Protégé , to perform queries on a graph. A graph saved locally allows you to some extent work with the infrastructure even without access to the Internet:
> ping amazon.com
ping: cannot resolve amazon.com: Unknown host
> awless list instances --local
| ID ▲ | ZONE | NAME | STATE |
|---------------------|------------|---------------------|---------|
| i-01b17116fed0a09bd | eu-west-1c | windows-demo-env | running |
| i-05c0a275adb5484fa | eu-west-1c | git.inno.wallix.com | running |
| i-06b8f07c1e4afb49a | eu-west-1b | AwlessWithScheduler | running |
[...]
Retrieving Resource Information
The show command allows you to get information about any resource:
> awless show i-03a1f22b36ddb0739
| PROPERTY ▲ | VALUE |
|-------------------|---------------------|
| Architecture | x86_64 |
| Hypervisor | xen |
| ID | i-03a1f22b36ddb0739 |
| Image | ami-70edb016 |
| KeyPair | keypair-blog |
| Name | blog-test |
| NetworkInterfaces | [eni-10e4352d] |
| Private IP | 10.0.20.245 |
| Public IP | 34.252.193.183 |
| RootDevice | /dev/xvda |
| RootDeviceType | ebs |
| SecurityGroups | [sg-a2a282c4] |
| State | running |
| Subnet | subnet-472a1731 |
| Type | t2.micro |
| Uptime | 46 hours |
| Vpc | vpc-96c7eef2 |
| Zone | eu-west-1b |
eu-west-1[region]
↳ @awless-test-vpc[vpc]
↳ @awless-public-subnet[subnet]
↳ @blog-test[instance]
Depending on: keypair-blog[keypair], @default[securitygroup], vol-0f82cd93824f456e5[volume]
Siblings: @AwlessWithScheduler[instance]
This command displays the characteristics and dependencies of the cloud resource. For example, in the case of an instance, the command displays its Virtual Private Cloud (VPC), subnet, objects associated with it, and information about the mounted volume, key pair, and instance security groups. The list of properties will vary depending on the type of resource:
> awless show AIDAI3FX74KAQQKIOM2XY
| PROPERTY ▲ | VALUE |
|------------------|-----------------------------------------------|
| Arn | arn:aws:iam::519101999238:user/awless-test-usr|
| Created | 4 months |
| ID | AIDAI3FX74KAQQKIOM2XY |
| Name | awless-test-user |
| PasswordLastUsed | 2 weeks |
| Path | / |
Depending on: @PepsWallixS3Assume[policy], @PepsWallixS3[policy]
Single Line and Patterns
awless.io is a great tool for viewing and analyzing cloud resources. But it also provides a powerful template engine that supports operations with the infrastructure (creating, editing and deleting resources).
Single line
The awless.io templates can be applied by entering single-line commands, or using scripts. To create an instance, key pair or resource of any other type, two simple commands are enough:
> awless create instance
> awless create keypair
awless.io does not require entering all the necessary information at once. Instead, it asks for data as the operation progresses, providing auto-completion at the time of input.
Let's first create a key pair that we use to connect via SSH to the instances that we will create immediately after:
> awless create keypair
Please specify (Ctrl+C to quit, Tab for completion):
keypair.name? keypair-blog
create keypair name=keypair-blog
Confirm? (y/n): y
[info] Generating locally a RSA 4096 bits keypair…
[info] 4096 RSA keypair generated locally and stored in ‘/Users/henri/.awless/keys/keypair-blog.pem’
OK keypair = keypair-blog
[info] Revert this template with `awless revert 01BHAE1C0CPMNYQ0KFPRQQ7XYA`
We only entered the awless create keypair command , the name of the keypair-blog key pair, and confirmed its creation.
awless.io uses both command line input and default options that you can configure (for example, AMI and instance type); awless.io asks for user confirmation before making changes to the existing infrastructure.
Now we easily create an instance using our brand new key pair:
> awless create instance keypair=@keypair-blog
Please specify (Ctrl+C to quit, Tab for completion):
instance.name? instance-blog
instance.subnet?
@awless-private-subnet @awless-public-subnet @demo-env-subnet
instance.subnet? @demo-env-subnet
create instance count=1 image=ami-70edb016 keypair=keypair-blog name=instance-blog subnet=subnet-2486b17c type=t2.micro
Confirm? (y/n): y
[info] create tag ‘Name=instance-blog’ on ‘i-094ceffee40892f66’ done
[info] create instance ‘i-094ceffee40892f66’ done
OK instance = i-094ceffee40892f66
[info] Revert this template with `awless revert 01BHAGCXY80FPK5FR73GTWWSD2`
To connect to the instance directly, you can use the ssh command :
> awless ssh instance-blog
awless could not validate the authenticity of ‘34.252.193.183:22’ (unknown host)
ecdsa-sha2-nistp256 public key fingerprint is SHA256:Ay8ODMNQiEt1U6BoTsPyIajoaHZ047v7uNjTV1SSQmk.
Do you want to continue connecting and persist this key to ‘/Users/henri/.ssh/known_hosts’ (yes/no)? yes
[info] Login as ‘ec2-user’ on ‘34.252.193.183’, using keypair ‘/Users/henri/.awless/keys/keypair-blog.pem’ with ssh client ‘/usr/bin/ssh’
__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2016.09-release-notes/
18 package(s) needed for security, out of 52 available
Run “sudo yum update” to apply all updates.
Amazon Linux version 2017.03 is available.
[ec2-user@ip-10–0–20–245 ~]$
There is no need to pass keys as arguments to the command, as they are stored locally, there is no need to specify a username, as awless.io guesses it yourself. The devil is in the details, and awless.io does everything possible to make using the CLI as convenient as possible.
In addition to simplified short commands, there are other new and useful features at your service. If you carefully read the previous output of the command, you could pay attention to the fact that revert Id is associated with each successful action. Indeed, changes as a result of the execution of most awless.io templates (single-line and full scripts) can be canceled very simply.
Like git, awless.io keeps a log of completed operations:
> awless log
[...]
ID: 01BHAE1C0CPMNYQ0KFPRQQ7XYA, Date: May 29 17:43:50, Author: user/hbinsztok, Region: eu-west-1
OK create keypair name=keypair-blog[keypair-blog]
ID: 01BHAGCXY80FPK5FR73GTWWSD2, Date: May 29 18:25:06, Author: user/hbinsztok, Region: eu-west-1
OK create instance count=1 image=ami-70edb016 keypair=keypair-blog name=instance-blog subnet=subnet-2486b17c type=t2.micro [i-094ceffee40892f66]
And like commits, operations can be easily rolled back:
> awless revert 01BHAGCXY80FPK5FR73GTWWSD2
delete instance id=i-094ceffee40892f66
Confirm? (y/n): y
OK delete instance
Patterns
Previously, we used templates containing only one command (single-line). In general, templates are a generalization and are a sequence of awless.io commands.
# Создать новую группу безопасности для данного инстанса
securitygroup = create securitygroup vpc={instance.vpc} description={securitygroup.description} name=ssh-from-internet
# Открыть доступ через порт 22 к инстансам в данной группе безопасности
update securitygroup id=$securitygroup inbound=authorize protocol=tcp cidr=0.0.0.0/0 portrange=22
# Создать новую клучевую пару
keypair = create keypair name={keypair.name}
# Создать инстанс в группе безопасности, к которому можно подключиться, используя созданную ключевую пару
create instance subnet={instance.subnet} image={instance.image} type={instance.type} keypair=$keypair name={instance.name} count=1 securitygroup=$securitygroup
We have added the following features for building a chain of commands:
- the result of the operation can be stored in a variable, the corresponding syntax: value = command . In the future, the stored value can be accessed like this: $ value ;
- use of template parameters, i.e. missing command arguments that awless.io will request at runtime. They are written in braces, as {instance.image} in the example above.
Templates are an extremely efficient and useful feature of awless.io. Using them, for example, you can easily:
- Deploy a highly reliable load-balanced infrastructure for your Wordpress site using a short template .
- create a self-scalable group of instances and track the load on the CPU of each individual, for the dynamic creation of new instances if necessary.
To execute the template as an argument, you can pass the file name in the official awless-templates repository (with the prefix repo:) , URL or the name of the doc file template:
> awless run repo:dynamic_autoscaling_watching_CPU
Try it yourself! Especially after that, you can use the awless revert command , which we discussed above, to remove all resources created as a result of executing templates (instances, security groups, target groups, target load balancers, etc.).
Please note that so far we have only implemented a draft version of the template engine, and prior to the release of awless.io 1.0, changes and new functions can be added to it. After release 1.0, all new versions will be accompanied by utilities to simplify migrations.
Team Overview
You can get help on awless.io functions like this:
> awless help
> awless [verb] -h
In these tables contains the main keywords used in commands and templates.
And this is just the beginning
Today, after more than 20 alpha releases released since the project launched in January 2017, we present awless.io 0.1.0. This project is still under development. We plan to improve our product in many ways, in particular, add support for even more AWS services and features. Please, if your favorite service (still) is not supported, open new ones or add existing issues on GitHub. Here is the development plan and functionality that we plan to add over the next year:
- 0.2.0 - Work with several regions. Using a local graph to suggest the selection of resources in accordance with their regions; easy switching between regions;
- 0.3.0 - Improving the scripting language. We want to make a formal description of semantics and add type checking for templates in order to diagnose errors before executing the request on the AWS side (this also applies to dry-runs);
- 0.3.0 - Improving the scripting language. We want to make a formal description of semantics and add type checking for templates in order to diagnose errors before executing the request on the AWS side (this also applies to dry-runs);
- 0.4.0 - Adding the ability to perform complex queries to the global infrastructure using a local graph.
We will be happy to assist you in the implementation of the project, including if you tell your friends about it. And in order to stay up to date with all the innovations, follow the news in the official twitter @awlessCLI .