Asteris

Introducing Converge

// Steven Borrelli // Converge

We’re delighted to introduce our newest project, Converge. Converge is a configuration management tool that makes it easy to configure servers, desktops, and Raspberry Pi devices.

Converge manages apt and rpm packages, files, directories, docker images and containers, and the Linux LVM subsystem. The CLI, modules, and API server are all bundled in a single 11Mb binary file, making it easy to deploy and run.

Converge is powered by a sophisticated graph execution engine that automatically creates dependencies and runs tasks in parallel. Internally, the desired state of a system is represented as a directed graph.

As can be seen in the example graph for Docker, by using a directed graph for dependencies we can make sure that Converge will not attempt to start Docker until the Docker package is installed and enabled.

Converge tasks visualized

When Converge runs, it compares the system state to the desired state and calculates the differences. The graph engine then walks the graph and attempts to remove all the differences.

Why We Built Converge

For the past couple of years we’ve been building Consul, Kubernetes, and Mesos clusters for our clients. In particular, we’ve spent a lot of time writing server configuration tasks in Ansible for Mantl.

As we helped develop Mantl and worked on other automated deployments, we started to imagine what our ideal configuration management tool would look like.

As developers we know the pain of starting a new project or joining a new team and spending a bunch of time configuring your laptop. Existing tools are powerful, but their complexity means they are used mostly by professional system administrators.

We started development late Summer 2016, and we already have example deployments of docker swarm, elasticsearch, and kubernetes clusters - all which can be brought up with a single command.

Installing Converge

Converge is designed to be simple to deploy and run. A single 11Mb binary contains the cli, modules, and server. Converge is cross-compiled and supports x86 and ARM-based Linux distributions along with macOS, so you can use the same tool across your devices.

The Converge installation script automatically determines your operating system and processor type. converge will be installed into your /usr/local/bin/ directory:

curl get.converge.sh | sudo bash -

Once installed, run converge help to see the command line options.

Your First Converge Module

Once the binary is installed, it’s easy to start configuring your systems. Let’s have Converge create a simple file. Type the following in an editor, and save it as something like hello.hcl:

file.content "hello" {
  destination = "hello.txt"
  content     = "Hello! Converge has detected {{platform.OS}}"
}

You don’t have to worry about text formatting. Running converge fmt hello.hcl will automatically format the file.

converge validate hello.hcl will check for any syntax errors.

Now let’s run this module.

converge plan --local hello.hcl will tell you if there are any change to be made. plan creates a set of differences between your server and the desired state, but doesn’t make any changes.

When you run converge apply, Converge will modify the system to get to your desired state.

 $ converge apply --local test.hcl 
2016-11-22T16:47:35-06:00 |INFO| serving	addr=:47740 component=rpc
2016-11-22T16:47:35-06:00 |WARN| skipping module verification	component=client
2016-11-22T16:47:35-06:00 |INFO| got status	component=client file=test.hcl id=root/file.content.hello run=STARTED stage=APPLY
2016-11-22T16:47:35-06:00 |INFO| got status	component=client file=test.hcl id=root run=STARTED stage=APPLY

root/file.content.hello:
 Messages:
  OK
 Has Changes: yes
 Changes: No changes

Summary: 0 errors, 1 changes

converge apply --local hello.hcl will create the file and ensure it was created. If you run converge apply --local hello.hcl a second time, it will show that no changes need to be made.

And that’s it! We’re working hard to make Converge easy to use. There are many examples at our GitHub repo to get you started.

Once you’ve got the basics down, you can explore some of the engine’s powerful features, including conditionals, task groups, and wait conditions.

Tasks

One of our favorite features of Converge is that it can create execution graphs for any shell-based task. You can even use different interpreters like python or zsh. This makes it easy to use your existing scripts with Converge.

Here’s a example of copying some data to s3 after checking the state of a docker swarm cluster. This also shows the use of param, a variable that can be passed into the configuration via the -p option on the command line or via a Terraform variable:

param "swarm-manager-ip" {}

param "swarm-token-bucket" {
  default = "my-bucket"
}

task "swarm-init" {
  check = "docker info 2>/dev/null | grep \"Swarm: active\""
  apply = "docker swarm init --advertise-addr {{param `swarm-manager-ip`}}"
}

task "swarm-persist-worker-token" {
  check   = "aws s3 ls s3://{{param `swarm-token-bucket`}}/worker"
  apply   = "docker swarm join-token worker -q | aws s3 cp - s3://{{param `swarm-token-bucket`}}/worker"
  depends = ["task.swarm-init"]
}

If the check process returns a non-zero value, the engine considers it a difference and will execute the apply.

The engine won’t run swarm-persist-worker-token until all parent tasks are completed - in this case, the swarm-init task.

To graph a module, simply run converge graph --local tasks.hcl | dot -T png -o tasks.png. For this example, the graph would look like:

Converge tasks

Features

Let’s examine some key features of Converge.

The Graph Engine

At the heart of Converge is a powerful graph engine that automatically creates dependencies and executes tasks in parallel.

For example, if you have a task that depends on the output of another task, the engine will automatically create a dependency and wait for the parent task to complete.

For users, creating explicit dependencies is as easy as adding a depends = ["parent.task"] value to your tasks. See the dependency documentation for more information.

task "names" {
  check = "test -d names"
  apply = "mkdir names"
}

file.content "hello" {
  destination = "names/hello.txt"
  content     = "Hello, World!"

  depends = ["task.names"] # added in the resource that needs the dependency
}

The Converge engine first checks your system and calculates the differences that need to be applied. Once changes are applied, another check is run to ensure that the system has reached the desired state.

API First and GRPC Support

Converge was built from the start to use APIs for all communication.

When Converge is run in server mode, it exposes system configuration via an API. This allows you to treat server configuration as another microservice in your environment.

When the cli client is used, it starts a local server for all communication. This means there is virtually no difference between local and remote configuration.

Converge uses grpc, which will allow us to provide bi-directional streaming communication over http/2 in a future version.

Converge supports authentication via a token and encryption via TLS. Refer to the server documentation.

Terraform Integration

A primary goal of Converge was to allow us to deploy clusters running software like Kubernetes and Docker swarm on them with a single command.

We designed Converge from the start to work well with tools like Hashicorp’s Terraform and cloud-init.

With our Terraform provisioner, it’s easy to use Converge to manage system configuration with just a few lines of code in your Terraform file. Terraform variables can be directly passed into Converge so that values like Ip addresses and external disk ids can be used in system configuration.

In this example, Terraform launches the Converge provisioner and passes in values for the ssh account, the IP of one of the servers, and the value of a Terraform-created S3 bucket.

provisioner "converge" {
  params = {
    user-name = "${var.ssh_user_name}"
    swarm-manager-ip = "${var.swarm-manager-ip}"
    swarm-token-bucket = "${var.bucket-prefix}-${var.name}"
  }
  hcl = [
    "converge/main.hcl",
    "converge/worker.hcl"
  ]
  download_binary = true
  prevent_sudo = false
}

Vagrant support is currently in development.

Module Verification

Converge supports module verification. This means you can put PGP/GPG signed modules on a static web site or s3 and have the client verify the signature before execution. This can be used for secure configuration for unattended systems like AWS autoscaling groups and IoT systems.

$ converge plan --verify-modules https://example.com/modules/basic.hcl

Converge will automatically look for an .asc file. This will result in Converge retrieving the following URIs.

https://example.com/modules/basic.hcl
https://example.com/modules/basic.hcl.asc

These are only a few features of Converge. We’re constantly adding features and built-in modules, with regular point releases every few weeks.

The documentation is the best place to get an up-to-date list of features and built-in modules.

Summary

Converge manages system configuration and is designed to be easy to use and run. It is suitable for deploying distributed systems like Kubernetes or configuring devices as small as a Raspberry Pi. The engine at the core of Converge models the path to the desired system state as a directed graph, and can execute complex workflows.

We’re excited about the future of Converge, and would love to have you join our community. The core team is present on the Converge Slack channel or you can open a GitHub issue.

Pre-compiled binaries can be downloaded from our releases page, or via the script located at get.converge.sh.