I’m going to try out CoreOS to create a cluster of machines to host an app consisting of docker containers.
CoreOS is a linux distribution designed to run clusters of containers efficiently and securely. Our application components run in Docker containers, organized as services. The distribution also includes etcd, a key/value store for distributed configuration management and service discovery; and fleet to manage services across clusters. The machines can be automatically updated with any security fixes and patches.
First, let’s create a CloudFormation template we will use to create our stack. I got a minimal template for CoreOS here, adding parameters for VPC subnets and related availability zones. I also updated the AMI for us-east-1, where I’m deploying this cluster, to the most recent stable version of CoreOS.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 | |
The most interesting part is probably the UserData value for the CoreOSServerLaunchConfig resource, which configures the etcd service and ensures the etcd and fleet services are started at boot time. We’ll get back to those later.
aws cli cloudformation create-stack can accept JSON-formatted command specification. It has a template generator command switch –generate-cli-skeleton, which will generate an empty json file template with parameter placeholders. It contains some mutually exclusive keys so you’ll need to remove some keys.
I ran the command, saved the output as a file, and modified it to my needs. I populated the DiscveryURL parameter with a new unique key from from https://discovery.etcd.io/new.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | |
create AWS cloudformation stack
Create a new stack consisting of three coreOS instances using a command like the following:
1 2 3 | |
We can use the usual AWS tools to verify our new stack and instances are running.
We can also verify our cluster members are registered with the free cluster registery we’re using here. It will return the cluster member nodes that have registered.
For example: curl https://discovery.etcd.io/87a8b544cf5hhf7436d601273c1d22d8 might return something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | |
Sweet. I presume we may prefer our own cluster discovery service at some point. It’s just an etcd service.
interact with stack instances
start ssh-agent and add private key
This uses ssh-agent to provide your key when necessary. The ssh -A comand forwards the agent to the hosts so you can ssh between hosts (fleetctl needs it). Use the key you specified when creating your instances.
1 2 3 4 | |
login to instances, at least two, in separate terminal windows
1 2 3 | |
The username is core. You obviously need to allow SSH traffic into your new instances from you management workstation using the AWS security group rules.
check status of etcd service
Be sure to allow the instances to chat with each other on TCP ports 4001 and 7001. At the start of the output below I hadn’t yet edited my security group to allow that communication. Once I made the change, one host became the leader and they added peers.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | |
test etcd distributed key service
see https://coreos.com/docs/quickstart/ as these steps are basically copied from that.
Set a key on one host
1 2 | |
Query the key from another host
1 2 | |
list machines and service units controlled by fleet
1 2 3 4 5 6 7 8 9 10 11 | |
Good, we have some machines but no fleet controlled units yet. Let’s create a service.
Docker container services
Create a docker container nano-service
Create file ~/hello.service
1 2 3 4 5 6 7 8 9 10 11 | |
load and start the unit
1 2 3 4 5 | |
You can see from the output the service was loaded and installed on a host other than the one we’re logged into. View cluster services with fleetctl stats <name.service>
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | |
Notice that two of the ExecStatPre tasks exited with failure. Oh my, what has happened? Those are the docker rm and kill commands which failed because the container named hello hadn’t been created yet. Those commands are executed to ensure we create a new container upon service start and it’s ok if they fail with nothing to do.
Some container log lines are displayed also.
inspect docker images and containers
We can look on the host where our service was installed to verify it has docker image(s) and running container named hello.
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Or, looks like we can use fleetclt to ssh to other machines. We can use the machine id reported by fleetctl list-machines or list-units.
1 2 3 4 5 6 7 | |
super cool!
stop hello.service
Stop the service.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | |
restart hello.service
The hello.service unit is still loaded; we’ll restart it. This time the ExecStartPre tasks removing the previous container should succeed.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | |
That’s the simple quickstart tour of coreOS, etcd, and mantaing services with fleet on an Amazon AWS cluster. I think the next step is to define some more useful, multi-tiered, services. We’ll need to define the relationships betwen the services, configurations, etc. Let’s use etcd!
a bonus service
Here’s the beginning of a private docker registry service.
1 2 3 4 5 6 7 8 9 10 11 12 | |