Asteris

Using Calico to Connect Mesos and Kubernetes

// Rokas Aleksiƫnas // Networking

Until now, there has been no general solution to bridge network between Kubernetes and Mesos. But, with Mantl 1.1 we’ve changed that. For our network stack we use Calico from MetaSwitch, whom we partnered with, to assign an IP per container for Kubernetes and Mesos/Marathon tasks. This enables communication between these classes of tasks on demand.

Here’s how we do it: on a running Mantl 1.1 cluster, save the following as app.json:

{
    "id":"/calico-apps",
    "apps": [
        {
            "id": "hello-world-1",
            "cmd": "ip addr && sleep 30000",
            "cpus": 0.1,
            "mem": 64.0,
            "ipAddress": {
                "groups": ["calico-k8s-network"]
            }
        }
    ]
}

Next, launch it using the Marathon API:

curl -X PUT http://control.your-mantl-domain/marathon/v2/groups/calico-apps \
     -H "Content-Type: application/json" \
     -d @app.json

This will create a Mesos task, which will be assigned an IP from a predefined Calico IP pool. The task will also be put in a Mesos NetGroup so that it may communicate only with other tasks that are in the same NetGroup. In this case, we want the task to be able to reach a Kubernetes container, so it is put in calico-k8s-network - a default network group for Kubernetes.

Now, let’s launch a Kubernetes task. We’ll use it to verify that networking has been indeed set up correctly by pinging the mesos task. Create a manifest file in one of the control nodes:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  namespace: default
spec:
  containers:
  - image: ubuntu
    command:
      - sleep 
      - "30000"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always

The new manifest file is picked up by running kubelet if it is saved in the /etc/kubernetes/manifests/ directory, and the pod is launched.

Alternatively, one can create the pod by running:

kubectl --server=http://localhost:8085 create -f test.yaml

See if it has been spun up:

kubectl --server=http://localhost:8085 get pods

Lets look up the IP that was assigned for Marathon’s task. Open up the Marathon UI and inspect the task, taking note of the assigned IP. Execute a ping on the container running in Kubernetes:

kubectl --server=http://localhost:8085 exec ubuntu -- ping <MARATHON_TASK_IP>

The output should be similar to:

[[email protected] ~]# kubectl exec ubuntu -- ping 192.168.0.65
PING 192.168.0.65 (192.168.0.65) 56(84) bytes of data.
64 bytes from 192.168.0.65: icmp_seq=1 ttl=62 time=0.862 ms
64 bytes from 192.168.0.65: icmp_seq=2 ttl=62 time=0.448 ms
64 bytes from 192.168.0.65: icmp_seq=3 ttl=62 time=0.315 ms

Voila! The Kubernetes container is talking with the mesos task for the first time.

If you want to experience this yourself - visit mantl.io or clone straight from github repository at https://github.com/CiscoCloud/mantl.

Checkout a branch which has this feature enabled and let it fly:

git clone -b feature/mesos-kube-calico-combo https://github.com/CiscoCloud/mantl

Want More Sleep?

Is your deployment keeping you up a night? Is it secure? Reliable? We'll help you keep the lights on with timely tutorials and tips to give you peace of mind that your cluster is running how it's supposed to.