October 25, 2015 at 23:26

SparkleFormation - CloudFormation template generator with rainbows and unicorns

    Sparkleformation

    If you are seriously using AWS (Amazon Web Services), you probably know about the ability to describe the infrastructure using JSON templates. In AWS, this service is called CloudFormation . In essence, this solution allows you to describe the desired state of any resources available in AWS (instances, opsworks, ELB layers, security groups, etc.). A set of resources is called a stack. After loading the CloudFormation template, the system itself will either create the necessary resources on the stack, if they are not already present, or try to update the existing ones to the desired state.

    This works well if you have a small amount of resources, but as soon as the infrastructure grows, problems appear:
    • In JSON, it’s not possible to use loops and for similar resources you have to repeat the same parameters in case of a change too (not DRY)
    • Double escaping needed to write configuration for cloud-init
    • There are no comments in JSON and it has poor human readability

    In order to avoid such problems, engineers from Heavy Water wrote on ruby ​​DSL and CLI to generate and work with these templates called SparkleFormation ( github ).

    DRY


    When I came to my current project, we had a CloudFormation template containing about 1,500 lines of resource descriptions and about 0 lines of comments. After using SparkleFormation, the template began to occupy 300 lines, many of which are comments. How did we achieve this? First, let's see how CloudFormation works, a typical resource description looks like this:
    Creating ELB
        "AppsElb": {
      "Type": "AWS::ElasticLoadBalancing::LoadBalancer",
      "Properties": {
        "Scheme": "internal",
        "Subnets": [
          {"Ref": "Subnet1"},
          {"Ref": "Subnet2"}
        ],
        "SecurityGroups": [
          {"Ref": "SG"}
        ],
        "HealthCheck": {
          "HealthyThreshold": "2",
          "Interval": "5",
          "Target": "TCP:80",
          "Timeout": "2",
          "UnhealthyThreshold": "2"
        },
        "Listeners": [
          {
            "InstancePort": "80",
            "LoadBalancerPort": "80",
            "Protocol": "TCP",
            "InstanceProtocol": "TCP"
          },
          {
            "InstancePort": "22",
            "LoadBalancerPort": "2222",
            "Protocol": "TCP",
            "InstanceProtocol": "TCP"
          },
          {
            "InstancePort": "8500",
            "LoadBalancerPort": "8500",
            "Protocol": "TCP",
            "InstanceProtocol": "TCP"
          }
        ]
      }
    }


    Since SparkleFormation allows you to use regular ruby ​​code inside DSL, you can rewrite it like this:
    Creating an ELB in SparkleFormation
      resources(:AppsElb) do
    type 'AWS::ElasticLoadBalancing::LoadBalancer'
    properties do
      scheme 'internal'
      subnets [PARAMS[:Subnet1], PARAMS[:Subnet2]]
      security_groups [ref!(:SG)]
      # port mapping 80->80, 22 -> 2222, etc.
      listeners = { :'80' => 80, :'2222' => 22, :'8500' => 8500 }.map do |k, v|
        { 'LoadBalancerPort' => k.to_s,
          'InstancePort' => v,
          'Protocol' => 'TCP',
          'InstanceProtocol' => 'TCP' }
      end
      listeners listeners
      health_check do
        target 'TCP:80'
        healthy_threshold '2'
        unhealthy_threshold '2'
        interval '5'
        timeout '2'
      end
    end
  end


    As you can see, we no longer repeat in the description of each port and adding a new one will take only one line. Moreover, if we need to create a lot of almost the same type of resources, but differing by 1-2 parameters, SparkleFormation provides such an entity as dynamics, where you can describe an abstract resource to which the parameters are passed:
    Documentation example
    # dynamics/node.rb
SparkleFormation.dynamic(:node) do |_name, _config={}|
  unless(_config[:ssh_key])
    parameters.set!("#{_name}_ssh_key".to_sym) do
      type 'String'
    end
  end
  dynamic!(:ec2_instance, _name).properties do
    key_name _config[:ssh_key] ? _config[:ssh_key] : ref!("#{_name}_ssh_key".to_sym)
  end
end

    And then we can call this abstract resource in the template:
    SparkleFormation.new(:node_stack) do
  dynamic!(:node, :fubar)
  dynamic!(:node, :foobar, :ssh_key => 'default')
end


    Thus, we can reuse the resources we need and, if necessary, change everything in one place.

    Cloud init


    We often take the opportunity to transfer an instance in the form of a yaml file when loading cloud-init and use it to install packages, configure CoreOS, individual services and other settings. The problem is that yaml should pass the instance to user-data in the CloudFormation template and it looked something like this:
    Crazy escaping
            "UserData": {
          "Fn::Base64": {
            "Fn::Join": [
              "",
              [
                "#cloud-config\n",
                "\n",
                "coreos:\n",
                "  etcd:\n",
                "    discovery: ", {"Ref": "AppDiscoveryURL"}, "\n",
                "    addr: $private_ipv4:4001\n",
                "    peer-addr: $private_ipv4:7001\n",
                "  etcd2:\n", 
...


    As you can see, this is absolutely unreadable, ugly and poorly maintained, not to mention that you can forget about syntax highlighting. Due to the fact that you can use ruby ​​code inside DSL, the whole yaml can be moved to a separate file and simply called:
    user_data Base64.encode64(IO.read('files/cloud-init.yml'))

    As you can see, this is much nicer than editing it inside JSON. Instead of IO.read, you can use an HTTP call for any parameters if you need it.

    CLI


    In our project, we use our own wrapper for managing templates, but this same command provides the CLI (Command Line Interface) for managing templates, called sfn . Using it, you can download, delete and update CloudFormation stacks using the sfn create, sfn destroy and sfn update commands. There is also implemented integration with knife.

    In general, after 4 months of using SparkleFormation, I am satisfied with it and I hope that I will no longer return to plain JSON to describe the infrastructure. The plans are to try the entire workflow, including sfn, offered by the Heavy Water team.
    Tags:

    Also popular now: