Disclaimer

Over the past month, I listened to interviews with developers on all three products and heard the statement “consider [Ansible / Salt / StackStorm] glue”. And now I, as an amateur homemade amateur, will be pleased to say that in my garage there is by no means the only pot of glue. I have 6 different types of glue for different applications, different glued materials and environmental conditions. All these 3 products are in the same camp, and each can be successfully used to achieve completely different goals. Recently, there has been a big overlap in functionality, consisting in the fact that they all penetrate the field of network automation. The opinions below belong to me , not to my employer (who sells billions of dollars worth of network infrastructure and deployment products).

I used all three products, in the development of two of them (Salt and StackStorm) I made a significant contribution and partially contributed to the development of Ansible. Frankly, the product I’m least familiar with is Ansible, but I talked to my colleagues and gathered information to fill in the blanks.

If you are going to scroll through the text and find out which product I announced as the winner, you will be disappointed. Think about your requirements and try more than one product.

Use under adult supervision

Ask yourself a few questions:

• What environments need to be supported? What is the set of servers and network devices?
• Who are my users? Hardcore system administrators, information security specialists, developers?
• How much am I willing to devote to user development?

With or without an agent?

Pay attention to it, this is really important. In this article, I am going to focus on the issues of device automation and their orchestration. These devices can be routers, switches, firewalls, emitting electromagnetic waves of the next generation of circulator, it does not matter. What is really important - an agent will not be installed in their operating system. Ansible had a compatible solution - using SSH as a transport, so it fits well with the world of endpoint configurations for which SSH is the least common denominator. SaltStack was created as a bus for the high-speed and secure minion (agent) that controls the communication, but it also has an agentless mode of use. StackStorm was the last to enter the playing field and by its architecture does not give preference to any of the works. It supports agent-based tools through packages for Chef, Puppet, Salt, as well as its own SSH remote controls and built-in support for calling Ansible playbooks of scripts.

API

Another key difference is the API:

• Ansible provides a CLI interface; Ansible Tower (enterprise version) has an API;
• StackStorm has a CLI, but also has a REST API for all components and services in the free version;
• Salt has both a CLI and a REST API (it is not enabled by default), it is also possible to use the "Enterprise API" as part of the Enterprise product, which includes features such as RBAC.

Ansible

Ansible - the brainchild of Michael DeHaan. The product was developed to automate the uniform tasks of server administration in large environments. Michael was in the newly formed RedHat technology group, where he founded various projects (such as Cobbler ), and after leaving RedHat he founded Ansible (although now Ansible is owned by RedHat). An excerpt from Michael's Ansible Essentials blog explaining the purpose of the project:

“We wanted to create another very democratic project to solve new open source problems in Red Hat, which could involve a wide range of developers. We remembered about busrpc. This project existed to fill the gaps between Cobbler and Puppet. Cobbler can prepare the system, and Puppet can add configuration files. But, since Puppet is too descriptive, it cannot be used to perform operations such as rebooting servers or completing all tasks “on demand” in the interim .

These on-demand tasks evolved into Ansible playbooks, and then the Ansible module ecosystem appeared and began to develop.

Design

Ansible is simple, which is its main quality (it will immediately become clear if you look at the other 2 qualities). There are no daemons or databases in it, and the installation requirements are minimal. Ansible just installs on a Linux machine and that’s it. You can define target servers in a static file, group them into meaningful sections, or use some dynamic host discovery module (such as Amazon EC2 or OpenStack) to search for virtual machines based on an API call. After the inventory has been carried out, you can select host-specific or group-specific variables for use in playbooks. They, again, are stored in static text files.

Then Ansible will connect to the selected host or group and start the playbook. A scriptbook (playbook) is a series of Ansible modules for execution on remote hosts written in YAML.

Connecting to a remote host is a bit like a well-planned military exercise : logged in, done your job, and logged out.

Ansible works according to the following algorithm: connecting to the server via SSH (or WS-Man / WinRM for Windows), copying Python code, executing and deleting itself.

Architecture

The Ansible architecture is straightforward: there is an application running on the local computer, and tasks performed on the remote host that communicate with each other via SSH and through files transferred via SCP / SFTP. Ansible does not have a server-client architecture, unlike the other two products, so task parallelization occurs on the local machine, but does not scale to multiple servers (if you do not use Tower).

When managing remote machines, Ansible does not leave its code on them, so the question of how to update Ansible when upgrading to a new version is not really worth it.

Extensibility

Modules in Ansible are really easy to develop, as, indeed, in the other two products. Read the style guide if you later decide to try to stomp your solution into the open source product repository instead of refactoring it again.

#!/usr/bin/python
import datetime
import json
date = str(datetime.datetime.now())
print json.dumps({
    "time" : date
})

ansible/hacking/test-module -m ./timetest.py

You should see something like the following:

{"time": "2012-03-14 22:13:48.539183"}

In the modules, you can define your own code for generating “ facts ” about the remote host at the “ collection ” stage , which can be used by your own or third-party modules. This can be implemented in the form of viewing both installed files and configuration to determine how to configure the service.

Corporate Support

Ansible Tower is an Enterprise version that turns the Ansible command line into a service with a web interface, scheduler and notification system.

Task Manager

It also has a user interface for playbooks, through which you can automate the deployment of cloud infrastructure, and then automatically add virtual machines to the list.
It is worth noting that the task scheduler, cloud deployments and server are functions of the free versions of both Salt and StackStorm.

Network support

The Ansible Network Support section is the most comprehensive of all three products and covers all major network equipment and platform vendors. In Ansible it is possible:

• Automate the configuration of everything related to network equipment, from specifying systems to connect to configuring basic services, all using modules and playbooks for a specific network platform.
• Test and verify the current state of the network, or to run or postpone the fact-finding process about the existing configuration.
• Continuously check for changes to network settings.

Ansible supports Arista, Cisco (all programmable platforms), F5, Juniper and other manufacturers. At Ansible alone, support is primarily provided and provided by vendors, not the community. At the moment, these are the best APIs, the greatest functionality and work with the latest platforms (since newer versions are supported).

Strengths

• Very quick and easy to get started.
• Lots of examples, documentation, and modules from the community.
• Ansible Tower delivers features for large enterprise deployments.
• Network modules supported by vendors.

Weaknesses

• In the absence of proper control, operators can save playbooks and SSH keys to their laptops. Not entirely a flaw in Ansible, but constant monitoring is required.
• There is no history of automation events, control over the target host is maintained only for the duration of the playbook execution, no more. Cannot be used for long-term tasks.

Stackstorm

I used StackStorm from version v0.11 (early beta preview) up to the latest version v2.2. It is a sophisticated, widespread application platform, like Salt. Although the story about it takes time, but still often the listener creates the wrong impression of the system. I see this as both strength and weakness. Weakness, because the complexity of the system can force to abandon the deployment altogether or use a simpler but incorrect solution where StackStorm would be a good alternative (often in such cases you even write your own solution from scratch). The strength is that as soon as it becomes clear how to use strengths, the system becomes truly flexible.


StackStorm Interface

Design

The core of StackStorm is an executable engine with plug-in rules, designed as a deeply customizable IFTTT service (if-this-then-that, “if that, then that”). You can configure StackStorm to respond to events by triggering a simple “action” (command) or complex workflow. Workflows are available in two versions: as an ActionChain — a chain of actions (a proprietary DSL workflow) or as a workflow on OpenStack Mistral, the engine of which is based on YAML.

StackStorm also includes a service for “chatops,” with which you can initiate workflows from events or messages from the chat platform (for example, Slack).

The list of the main components of StackStorm:

• Sensors are Python plugins for inbound or outbound integration that receive or observe events. When an event that occurs in the external system is processed by the sensor, the StackStorm trigger will fire.
• Triggers - representations of external events in StackStorm. There are general triggers (for example, timers and webhooks) and integration triggers (for example, an alarm in Sensu or a task update in JIRA). A new type of trigger can be identified by writing a sensor plugin.
• Actions are the outgoing StackStorm integration. There are common actions (ssh, REST calls), integrations (OpenStack, Docker, Puppet) or custom actions. Actions are either Python plugins or other scripts that integrate with StackStorm by adding a few lines of metadata. Actions can be called directly by the user through the CLI or API, or used and called as part of rules and workflows.
• The rules bind triggers to actions (or workflows) by applying matching criteria and linking the information received from the trigger to the input of actions.
• Workflows combine actions into “uber actions,” defining the order, transition conditions, and data transfer between them. Most automation solutions have more than one step and therefore require more than one action. Workflows, like atomic actions, are available in the actions library, can be called manually or run according to the rules .
• Packages are deployment units for content. They simplify the management and sharing of StackStorm-enabled content by grouping integration tools (triggers and actions) and automation (rules and workflows). A growing number of packages are available in the StackStorm community. The user can create their packages, post them on Github or send them to StackStorm Exchange .
• An audit trail with complete information about the launch of the context and the results of the actions that were launched manually or automatically.

Design

StackStorm consists of a number of services that use the message queue (rabbit) and data storage (mongo) to save their state and exchange data with each other. StackStorm also has a web-based interface (yes, even in the free version!) That allows you to configure rules, perform actions "on demand" and check the audit log.

Architecture

Unlike Ansible and Salt, StackStorm was not designed to configure workstations or communications. StackStorm includes packages for Salt, Chef, Puppet, and Ansible, so if you need to use Chef for the agent and configuration management system, use StackStorm to call API-based services such as Sensu or Kubernetes. This is an easily achievable and obvious solution. In Salt, there is still a concept of execution on the final machine or on the master (optional). If you need to call the Kubernetes API, then the question of which computer called the API becomes controversial. The same goes for the configuration of network devices.

MongoDB can be scaled using well-documented templates, the same goes for RabbitMQ. Services themselves were developed with the HTTP / RESTful API and can use load balancing to scale. Roles can be deployed on a single server or distributed across multiple servers depending on your needs.

I really like StackStorm extensibility, this is definitely a key advantage over the other two products. StackStorm extension points are called packages . They are self-contained, can be stored in Git, and manage their dependencies through virtual package-level environments in Python. When installing the package, either the Git URL or the HTTP URL, (optional) credentials are specified, and StackStorm already downloads, configures and installs the package.

If StackStorm were a programming language, it would be strongly typed . For actions, the types of all input data are indicated; for triggers, fields and types are indicated. This makes it easy to predict what will be returned by a third-party extension, and is a unique feature of StackStorm.

Unlike Salt and Ansible, StackStorm does not contain any extensions on delivery , all of them must be installed individually. This facilitates deployment and simplifies dependencies.
When developing an integration mechanism for StackStorm, you can create sensors, actions, and workflows in one definition. Salt and Ansible modules are standalone. Thus, if an extension (for example, Salt) includes Beacons, “Execution Modules” and “Status Modules”, then they have nothing in common, except for the name and author. This can create problems when managing pip dependencies.

The solution to this problem in StackStorm is that each package has its own requirements.txt file, as well as a YAML file that describes the purpose, requirements and version of the package. You can update the package linearly while maintaining the existing configuration. Packages also contain a template configuration, unlike Ansible and Salt, in which the configuration format of the modules is stored only in the documentation, making them more susceptible to user errors. In addition, you often have to look at the code of the module if the developer did not bother to document the configuration parameters properly.

Another unique feature is that ChatOps aliases (chat commands) are built into packages. Therefore, when installing, for example, the NAPALM package, all chat commands for NAPALM will also be installed.

Corporate Support

StackStorm is an open source Apache-2 licensed product hosted on GitHub. StackStorm is owned by Brocade (which recently sold part of its assets, and part of StackStorm is now owned by Extreme Networks).

When licensing StackStorm, a product called “Brocade Workflow Composer” is purchased, and the license holder receives additional functionality, as well as Enterprise-level support. The system that I worked with in the production environment was licensed, and their support team left me with the impression of being responsive and knowledgeable. You also get RBAC, in which you can specify for each level of action who has access to which actions.

Brocade workflow composer

Network support

You are lucky if you use Brocade VLX or SDX, as they are well supported , which is to be expected.

In April 2017, they combined support for the NAPALM library and the cross-platform Python abstraction package for Cisco, Juniper, Arista, and other vendors. Now you can use the NAPALM functionality in terms of routing, interfaces, BGP peering and some other great features. Matt Oswalt (co-author of O'Reilly's book on network automation) has written a good blog about progress in this direction.

NAPALM Demo for StackStorm

Strengths

• The free web interface is easy to use by default and does not require in-depth knowledge of Python or DevOps.
• ChatOps integration is built-in, works out of the box (using Slack, the API key is simply deployed) and supports many other chat platforms.
• OpenStack Mistral is really powerful, and if you master it, it will become possible to immediately cover several DevOps tools, it is easy to create branches and loops.
• Brocade Workflow Composer is a great way to get non-developers to use the system.

Weaknesses

• Does not have as many expansion packs available as Salt and Ansible. First, check to see if packages are available for your systems and APIs and what features are included.
• The OpenStack Mistral system is still rather poorly documented; there are a lot of trial and error in the syntax of YAQL queries.

Salt

First of all, Salt is a product, SaltStack is a company. So when I talk about Salt, I’m talking about Salt Open, an open source version.

Salt has a massive list of components, at first (and when I say “at first,” I mean “first year”), it can be really amazing. Salt has many functions, therefore, usually comparing Salt with Ansible, Salt fans say “yes, but it does a lot more for us”. As with StackStorm, this works for and against Salt. If I just told you to bring grains (crystals) from the mine, you would think that I have in mind Tolkien’s novel, but as soon as you know what Salt mine (salt mine) is in Salt terminology, everything will become obvious.

Design

Salt was born as a distributed system for remotely executing commands and query data on remote nodes or minions . Remote execution is possible either on separate nodes or on groups according to arbitrary selection criteria - “ targeting ”.

Salt has been expanded to a configuration management system capable of supporting remote hosts in specified states (for example, ensuring that certain packages are installed on them and certain services are running). There are many components to Salt , and I'm just sure I missed something!

• A master server that runs basic services for communicating with Salt minions. It also contains a keystore for encrypting between it and the minions.
• Minions are agents who use the micro version of Salt for local execution and communication with the master.

• Engines - they Salt Engines (salt) engines - are external processes running for a long time, that work with Salt.

  
  
• States or formulas - files containing YAML and template data for setting minions. The template engine is also very flexible . It is not limited to Jinja support, but also supports chetah, genshi, mako (very important for those with Puppet experience), wempy, or even pure Python.

Minions (proxies or regular) can be addressed using grains (grains, crystals), pillars (pillars, columns) or identifiers. There are other plugins for targeting , as well as the ability to create your own, based on something like a SQL query or KVP repository.

• Grains - Salt contains an interface for retrieving information about a downstream system. It is called the "grains" interface because it is a "salt" consisting of "grains" of information. Grains are built for the operating system, domain name, IP address, kernel, OS type, memory, and many other system properties. The grains interface is available for Salt modules and components, so the necessary minion commands automatically become available on the respective systems.

  
  
• Pillars is a Salt interface designed to provide global values ​​distributed among minions. Pillar is a free-form data resource that can be JSON, YAML or other required and can be stored in files or externally . This is a unique Salt feature that allows you to integrate it with other systems in which a common data warehouse would be useful (for example, ITSM or the resource registry).

To extract data, you can also use data from minions and store them in a Salt mine (salt mine). In the future, this data can be used in other tasks, such as configuration of states based on templates. Unlike Ansible (which only supports YAML), it can be implemented in different formats.

Architecture

Salt architecture is based on the principle of wheel hubs and spokes (fan structure). Some very large deployments use several wizards, but this happens quite rarely. Wizards can easily scale to many thousands of nodes, in part because of the lightweight ZeroMQ message bus. Other deployment models:

1. Installation without a wizard.
2. Hierarchical masters, interconnected with the help of syndic.

The wizard contains status files that are usually placed on a shared storage volume. They are configured in a tree view to be able to use targeting to define server groups for configuration and environments / applications for deployment.

The event-based Salt system uses beacons. Like the StackStorm sensor and trigger system, Salt beacons trigger events on the message bus, which can then be processed in the reactor (on the master). The rule mechanism in the reactor is pretty crude compared to StackStorm, because usually a state or execute command is activated behind the beacon that triggers the event. However, beacons work on minions, so if events are detected on servers, everything is extremely clear. Since StackStorm and Ansible have no agents, this will be a unique Salt feature.

Thorium (thorium) - a complex reactor for Salt, was included in the last release as an experiment and, perhaps , will be supported in future releases. Thorium adds support for more complex rules and event collection.

Extensibility

Everything in Salt is extensible, right down to modules that display execution results in the CLI. This is a big plus for Salt, as you can easily implement your own changes without having to maintain a parallel fork in the main project. Each function in Salt is also pluggable.

The most common scenarios for expansion may be the development of a state module (to describe how a piece of software or service should be configured) or a execution module (code to interact with an API or system). Both types of modules can be written using a relatively small template, both are well documented and can be tested using a good built-in test environment. You can perform nodal testing of modules using PyTest, even without being on the wizard or without a working wizard at all. Integration testing requires a Linux system, although with little hacker skills modules can also be run on OSX (Windows is out of the question, as is the case with StackStorm with Ansible).

You can either maintain your standalone package, or contribute directly to the Salt project on GitHub. The biggest drawback when entering code into the main project is that users may need to wait a whole cycle of updates to easily install your modules. This takes about 3-5 months at their current pace, so although the Salt system “ comes with batteries ”, it has disadvantages.

Salt also has a package manager, SPM , which is primarily aimed at grouping state management formulas. It can be used to package modules in order to circumvent the slow release cycle, which I will mention in its weaknesses (although this is not well documented).

Salt has been developing very rapidly over the past few years and has undergone some major changes. Because of this, there may be incompatibility between community-developed modules. I also believe (although this is not unique to Salt) that the modules provided by the community are poorly tested.

Corporate Support

Salt Open is an open source version, but Salt Enterprise can also be licensed, which comes with some nice features, such as:

• Web-based interface for targeting, code execution, stateful control and integration with LDAP;
• Integration with ServiceNow, which allows you to deploy new servers and apply states from the ServiceNow ITSM integration;
• RBAC with LDAP integration (of course);
• "Enterprise API", which translates the functions of the Enterprise version into the REST API.

Network support

Because Salt relies on a message bus, and ZeroMQ has a number of dependencies that typically require a complete network configuration at the OS level, managing network devices is not obvious when using Salt. The latest Salt release has significantly improved proxy minion support . Proxy minions - virtual minions - this is a process that can work anywhere to remotely control devices via SSH, HTTP or using another transport mechanism. It uses the same functionality as a regular minion, but it also has some features . To avoid confusion with the proxy in Puppet (the proxy there is the central machine through which all requests go), I’ll say that this is just a process associated withthe device you are accessing. Thus, this is a separate process at each minion. It is usually lightweight and consumes about 40 MB of RAM. You can create proxy minions by developing Python modules that can run on a minion. The Salt team has demonstrated this in past years at SaltConf.

Currently supported network platforms:

• JunOS (Juniper);
• NXOS (Cisco);
• Cisco NSO (Cisco NETCONF Orchestra);
• NAPALM
  
  Salt recently joined support for NAPALM thanks to the very smart engineers at Cloudflare. NSO and NAPALM have similarities, but NSO carries Cisco licensing costs, and you will need to decide which path to follow at an early stage.

NAPALM Support in Salt Demo

Strengths

• Salt can work with or without an agent ( salt-ssh).
• Ultra-high performance for large deployments with ZeroMQ.
• Agent-based architecture allows you to deploy beacons on hosts running both Windows and Linux, and even detect events locally.
• Widely uses some very large deployments (like LinkedIn).
• Salt can easily be combined with an existing set of databases or APIs due to its strong extensibility.

Weaknesses

• Extensions built into the kernel are too rare for rapidly changing environments.
• Modules cannot clearly declare their dependencies, i.e., they need to manage a single virtual environment and pip dependencies.

Conclusion

Event based or not?

This is the biggest difference between the three products. Salt and StackStorm have this functionality. StackStorm has services that can be written (sensors), and strongly typed events that can be triggered, as well as a complex rule mechanism. Salt has beacons - services that can be run both on agents and on the central master to detect events on local machines. This is a unique opportunity. The open source version of Ansible does not allow (and does not try) to respond to events.

Community support

I saw that network manufacturers specifically designed modules for Ansible , while in the case of other platforms (with the exception of Brocade for StackStorm), the modules were introduced by the community. Undoubtedly, Ansible has the broadest support for network platforms. Although with the introduction of NAPALM and NSO in both StackStorm and Salt, this changes the situation, since both systems support Arista, JunOS (Juniper), Cisco APIC-EM, NXOS, etc.

Time to start

The strength of Ansible is the minimum number of settings (they, in principle, are not). Popularity in network tasks may be due to the simplicity and familiarity of network administrators with the CLI for managing remote devices without the need to deploy any additional servers to run the software. If you have many small isolated sites (for example, commercial branches), you should consider whether to split the architecture. My employer manages data networks in some large chain supermarkets, and I would not dare to have a master in the center, while shops in rural areas may have unreliable communications.

Configuration Data Warehouses

Salt is unique in that its key stores are pluggable. If you want to store passwords or keys from Hashicorp Vault, it's like two fingers. If you want to store bits of data in an SQL database, everything, again, works out of the box. Think about what other systems and platforms might need to access or enter the data you are targeting.

Security

When comparing Ansible and Salt, Salt has its own keystore for communicating with agents, while Ansible uses SSH for transport. Ansible's poorly managed environment is typically a collection of private keys stored on administrators' laptops (please do not do this). Salt offers a unique feature for data protection. Templates, states or grains can be stored in an external secure data warehouse. StackStorm stores data in MongoDB, which your information security team should definitely check before releasing the system into a production environment.

Training

If you do not want to be the only specialist supporting this platform, you will need to train some colleagues. Salt and Ansible have published detailed books, StackStorm - no. Salt and (RedHat) Ansible offer training solutions almost exclusively in the United States, StackStorm does not (yet). Salt and Ansible have PluralSight courses, but they are really basic.

Licensing

Both Salt and StackStorm are licensed with Apache-2, Ansible is GPLv3. If you are not very familiar with OSS licensing, I recommend the TLDR Legal website. Salt was used as an example by SuSE to create a system management product (in part because of the flexibility of their OSS license).

Skills

It's anecdotal (I follow this very carefully), but Ansible has been getting the attention of network administrators and DevOps engineers around the world. You will surely find that hiring an Ansible engineer is a lot easier than Salt or StackStorm. But DevOps engineers are as rare as the teeth of chickens, so you'll pay a lot of dollars regardless of platform.

Which glue should I choose?

Please try at least two platforms and make an informed decision. I already wrote about this earlier, but when working with DevOps tools, people can discover the miracle of automation, and having studied the tool, then just religiously stick to it.

You should not, as a child who first tasted chocolate, simply believe that a brief moment of joy is the merit of Cadberry.

Acknowledgment : Thanks for the feedback and contributions from Salt, StackStorm, Ansible, and the community .

Anthony Shaw

Dimension Data's director of innovation and technical development, a father, Christian, and Python fan, solves problems with opensource, ideas, and people .

References

1. Ansible vs Salt (SaltStack) vs StackStorm .