Using Ansible for Marathon/Chronos deployments

Here at Motus, Ansible has long been our tool of choice for application deployments and bulk ad-hoc sysadmin commands (we use Puppet for configuration management). As we have moved from “traditional” application deployments – where we ship binaries to multiple servers, perform local deployment commands, and verify – to deployments on Marathon and Chronos – where we deploy using HTTP commands – we have had to tweak our Ansible usage somewhat. This post covers some of the things that we’ve done to allow Ansible to work well in this environment.

Environments

We rely heavily on the inventory functionality in Ansible to control multiple environments. Each inventory has a list of hosts and a set of group variables specific to that environment:

├── prodcopy
│   ├── group_vars
│   │   └── boston
│   └── hosts
├── production
│   ├── group_vars
│   │   └── east
│   └── hosts
└── staging-aws-east
    ├── group_vars
    │   └── aws-east
    └── hosts

Inside the hosts file is the standard Ansible list of hosts, but in order to make web service calls work well, we add 127.0.0.1 to our list of servers in each region:

[boston]
server1.motusdev.com
server2.motusdev.com
...
127.0.0.1

The reason for that is because…

Web Service Playbooks

Traditional Ansible playbooks apply to a particular set of hosts (the hosts parameter at the top). This tells Ansible the hosts on which to execute the commands in the playbook. However, when doing deploys to a cluster (like Marathon or Chronos), we only need to execute one HTTP POST call – so we use what Ansible calls local playbooks. That causes the commands in the playbook to be run locally.

However, in order for Ansible to properly pick up environment information from the inventory, however, the local connection needs to be part of the inventory – so that’s why you need to add 127.0.0.1 to the host file of every environment.

Initially, we were using Ansible variables to create the JSON body of the web service requests:

---
- hosts: 127.0.0.1
  connection: local
  vars:
    app_name: "/app/foo"
  tasks:
    - set_fact:
        request_body: {
          "id": "/app/foo",
          "cpus": 0.1,
          "mem": 256.0,
          "uris": ["file:///root/.dockercfg"],
          "container": {
              "type": "DOCKER",
              "docker": {
                "image": "motus/app:{{tag}}",
                "network": "BRIDGE",
                "portMappings": [
                    { "containerPort": 80, "hostPort": 0, "protocol": "tcp" }
                ]
              }
          },
          "env": {
              "SOME_ENV_VARIABLE": "{{some_env_specific_value}}",
          }
      }

    - name: Kick off marathon app
      uri: url=http://{{mesos_master}}:8080/v2/apps/{{app_name}}
           method=PUT
           body='{{ request_body | to_json }}'
           HEADER_Content-Type="application/json"
           status_code=200,201,204

This playbook uses the set_fact module to create an Ansible dictionary containing all of the data needed for JSON and then uses the to_json filter to turn it into a string for use in the web service call. The variables in the set_fact call come from the inventory variable definition, allowing us to change values based on the environment.

This worked fine when all the environment-specific variables were strings, but when we wanted to start changing the numeric values (such as cpus or mem in the example above), we ran into this bug. No matter what I tried, I couldn’t get the values to be output as numbers in the final JSON.

Web Service Playbooks, Take 2

To fix the bug listed above, and to make our playbooks more DRY, we decided to use the template module to create the JSON and the file lookup to grab the contents of the file and stick into a variable. This works like this:

We create a template file containing the JSON structure and the variable substitutions:

{
    "id": "{{app_name}}",
    "cpus": {{myapp_cpu_share}},
    "mem": {{myapp_memory}},
    "instances": {{myapp_instance_count}},
    "uris": ["file:///root/.dockercfg"],
    "container": {
        "type": "DOCKER",
        "docker": {
          "image": "motus/myapp:{{tag}}",
          "network": "BRIDGE"
        }
    }
}

and we stick it in a templates directory. Then, in the playbook, we do this:

    - name: Create json body
      template: src=templates/{{app_name}}.json.j2 dest=/var/tmp/marathon_app.json

    - name: Kick off marathon app
      uri: url=http://{{mesos_master}}:8080/v2/apps/{{app_name}}
           method=PUT
           body='{{ lookup("file", "/var/tmp/marathon_app.json") }}'
           HEADER_Content-Type="application/json"
           status_code=200,201,204

This evaluate the template, still filling in environment-specific variables, but for whatever reason, the templates are able to create numeric values. Then we use the lookup mechanism to grab the file contents.

Hopefully this overview helps clarify how to use Ansible as a web service client and still have the power of environments and playbooks.