After following the recent articles about building a Docker swarm on ARM (https://goo.gl/2FjP8f) and (https://goo.gl/ZTXcp), I found myself wanting to spin up services, for which no ARM image was available, or no ARM image with a recent version was available. The alternatives are trinary: 1) do without the thing I want, 2) settle for an older version of the thing I want, or 3) figure out how to build the thing I want. I am a bit of a tinkerer, and I like to have at least a working knowledge of my tools, so I went with the third option. For the most part, this is a very straight-forward process, but every once in a while, you end up having to tweak some things. Do not be intimidated by that, because it is worth it, and not that difficult of a journey. To help with this, let us set up a graphite stack. For this stack I will need several elements, plus some support in the form of an internally hosted image registry.
A registry
We will be using an already created image for this at http://dockr.ly/2kmNgod. The registry provides a place to store your custom images and deploy from. It is a great staging ground before you push your final product up to https://hub.docker.com or whatever other registry you choose.
The registry front end is just a handy little service to have going if you have a registry (http://dockr.ly/2D5DRt3). We will not go into the advanced features like deleting images, which requires additional setup on the registry side, but we will be able to browse our images and get information about them.
I'll be using go-carbon for the cache (https://goo.gl/hgjGZo). The choice was a simple one, since go-carbon will use more than one core in a single instance. It is really easy to setup, and if you have any schema definitions it may work just fine, and it also supports pickle format.
I will be using graphite-api for the render API endpoint https://goo.gl/P43pHC. There are other choices like carbonserverand carbonzipper. I believe go-carbon has support for carbonserver now, but I have not yet tried it. I think going the stock way is not so bad in this instance. It is only occasionally queried, so it does not need to be as high performance as the cache.
I will be using grafana for the display UI, since it is pretty standard (https://grafana.com). We could use the graphite-web package, but the graphing capabilities, while being the same, are much less accessible than grafana's. There are other options as well, and you should check them out before making any final decisions about what is right for you.
Deploy the infrastructure
Since the point of the swarm is really high availability and load balancing of those services (both hardware and network balancing), the services listed above will be launched as separate services. Please note that there are multiple options for each of these, but covering them is beyond the scope of this article. Having said that, let us get our infrastructure needs met, and deploy our registry:
$ docker service create --name=docker-registry --publish=5000:5000/tcp cblomart/rpi-registryAfter this command completes, we have a registry. However, we have a problem that none of our Docker instances will use it because it is not secure. There are two approaches: the first is to configure your docker instances to use an insecure registry (specifically whitelisted), and the second is to get a certificate (self signed, or publicly verifiable). This is an internal registry. To get things moving along, I have chosen the first option. To do so on all of the docker nodes, add the following to /etc/docker/daemon.json, which assumes you have a default setup, and that this is the entire file:
{
"insecure-registries": ["10.0.0.15:5000"]
}
Docker Registry UI
Now let us move to get the registry frontend in place, which will let us build our first image. Note that I used the hostname swarm. This is accurate in my setup, since I have a CNAME record in my DNS server, although your setup may be different:
$ git clone https://github.com/parabuzzle/craneoperator.git $ cd craneoperator $ docker build -t docker-registry-ui . $ docker tag docker-registry-ui swarm:5000/docker-registry-ui-arm $ docker push swarm:5000/docker-registry-ui-armNow we can launch our service. This one, like many others, makes use of environment variables to influence its operation. You will, of course, need to edit these to taste:
$ docker service create --name=docker-registry-ui --publish=8081:80/tcp -e REGISTRY_HOST=swarm -e REGISTRY_PROTOCOL=http -e SSL_VERIFY=false swarm:5000/docker-registry-ui-arm
Carbon cache and render API
Go-carbon is a Go application, and it requires Go 1.8+. So that we do not introduce any unnecessary issues by relying on a recent version of Go being available in my favorite package manager, we will just drop the latest version into a custom location. At the time of this writing, Go 1.9.2 is the latest version, so we will use that. I am also assuming you are running Linux, on an ARM machine, such as an ODROID-XU4, which makes a wonderful workstation. I drive each of my 3 monitors with an XU4, and use x2x to allow keyboard and mouse sharing, but that is an article for another time.
$ cd ~
$ mkdir -p ~/.golang/path
$ wget https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-armv6l.tar.gz
$ tar -zxf go1.9.2.linux-armv6l.tar.gz
$ mv go .golang/root
$ export GOROOT=${HOME}/.golang/root
$ export GOPATH=${HOME}/.golang/path
$ export PATH=${GOROOT}/bin:${GOPATH}/bin:${PATH}
Now we are ready to start working on go-carbon. This one is nice, since it has a Dockerfile already made, and all we have to do is build the binary and set up our config files. Fetching the source, and building the binary can be done in one fell swoop:
$ go get -v github.com/lomik/go-carbonNow that we have that out of the way, let us go set about building our Docker image:
$ cd ${GOPATH}/src/github.com/lomik/go-carbon
$ cp ${GOPATH}/bin/go-carbon .
$ mkdir config-examples
We will go ahead and stop here, since we are going to need to tweak the config file. There are a fair number of options, but you likely will not need any. I will leave it up to you to deal with any customizations and just assume that the defaults are good enough for now. The only edit we will make is to point to the correct schemas file and our data directory we can do this with a simple sed command:
$ ./go-carbon -config-print-default | sed -E 's,(schemas-file =).*,\1 "/data/graphite/go-carbon-schemas.conf",g' | sed -E 's,(data-dir =).*,\1 "/data/graphite/whisper",' > ./conf-examples/go-carbon.confNext, we can get our schemas file in order at /conf-examples/go-carbon-schemas.conf:
[default] pattern = .* retentions = 10s:1h, 30s:3h, 60s:6h, 1h:1d, 6h:1w, 12h:1m, 24h:1yThis gives us plenty of space for our metrics to aggregate and stick around for historical reasons. At this point, we are ready to start building our Docker image. I assume for the purposes of this article that this is the same machine you built go-carbon on and that it has Docker installed.
$ docker build -t go-carbon . && \ $ docker tag go-carbon swarm:5000/go-carbon-arm && \ $ docker push swarm:5000/go-carbon-armNow we can publish our service to our swarm. We have already done this a few times, so it should be familiar:
$ docker service create --name=carbon-cache --publish=2003:2003/tcp --publish=2003:2003/udp swarm:5000/go-carbon-armHowever, now we hit another snag. We need graphite-api installed into the image, because it needs access to the whisper files. You could create a shared filesystem and mount it for both the cache and the api images, but for the purposes of instruction, I think we will just modify our go-carbon image to support both. The first thing to note is the docker image is called “busybox”. Since I need Python, I chose to move this to 'debian:stretch'. The first thing is to get our graphite-api.yaml ready, which resides at ./conf-examples/graphite-api.yaml:
search_index: /data/graphite/index
finders:
- graphite_api.finders.whisper.WhisperFinder
functions:
- graphite_api.functions.SeriesFunctions
- graphite_api.functions.PieFunctions
whisper:
directories:
- /data/graphite/whisper
Since we are starting more than one service, we should use a custom entrypoint script. Let us go ahead and write that now.
#!/bin/sh gunicorn -b 0.0.0.0:8000 -w 2 graphite_api.app:app & sleep 2 /go-carbon -config /data/graphite/go-carbon.confAfter moving over to debian:stretch and installing our packages and config, our Dockerfile in our go-carbon directory should now look something like this:
FROM debian:stretch RUN mkdir -p /data/graphite/whisper/ RUN apt update && apt upgrade -y && apt dist-upgrade -y && apt autoremove -y RUN apt install -y gunicorn graphite-api ADD go-carbon / ADD entrypoint.sh / ADD conf-examples/* /data/graphite/ RUN chmod +x /entrypoint.sh RUN rm /etc/graphite-api.y* ; ln -s /data/graphite/graphite-api.yaml /etc/graphite-api.yaml CMD ["/entrypoint.sh"] EXPOSE 2003 2004 7002 7007 2003/udp 8000 VOLUME /data/graphite/Let us go ahead and rebuild, then push our new image:
$ docker build -t go-carbon . && \ $ docker tag go-carbon swarm:5000/carbon-cache-arm && \ $ docker push swarm:5000/carbon-cache-armWe will then remove our old service and recreate them. I am aware of the upgrade processes for running services, but I felt that a fair amount could be written on that subject alone, so i would leave it as-is.
$ docker service rm carbon-cach $ docker service create --name=carbon-cache --publish=2003:2003/tcp --publish=2003:2003/udp --publish=8000:8000/tcp swarm:5000/go-carbon-arm
Graphite setup
At this point, we are ready for the final piece of the project, which is another image that we will need to rebuild. However, thanks to Docker's "fat manifests" or manifest lists, referencing "debian" will get you the appropriate image for your architecture. Almost all official builds are done this way, so you no longer need to go hunting down an image for the ARM architecture.
Hopefully, in my next article, we will explore setting up your own "fat manifests" in your private registry. This is very useful if you, like me, have mixed architectures like amd64 into your swarm and would like any of your services to be able to deploy to any of your nodes. So let us get started on the Grafanas image. I chose the image at https://goo.gl/pfpVef, since it has all of the plugins you could want already loaded, and that saves a bunch of work for everyone. You can, of course, grab the official image and go through the same process:
$ git clone https://github.com/monitoringartist/grafana-xxl.git $ cd grafana-xxl $ cp Dockerfile Dockerfile.armWe need to change line 17 from:
$ curl https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_${GRAFANA_VERSION}_amd64.deb > /tmp/grafana.deb && \
To the following:
$ curl https://github.com/fg2it/grafana-on-raspberry/releases/download/v${GRAFANA_VERSION}/grafana_${GRAFANA_VERSION}_armhf.deb > /tmp/grafana.deb && \
Then, line 20 needs to change from:
$ curl -L https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64 > /usr/sbin/gosu && \To this:
$ curl -L https://github.com/tianon/gosu/releases/download/1.10/gosu-armhf > /usr/sbin/gosu && \Once that is done, we are ready to rebuild, push, and deploy our service. We will use those increasingly handy and familiar commands again:
$ docker build -t grafana-xxl-arm -f Dockerfile.arm . $ docker tag grafana-xxl-arm swarm:5000/grafana-xxl-arm $ docker push swarm:5000/grafana-xxl-arm $ docker service create --name=grafana --publish=3000:3000/tcp swarm:5000/grafana-xxl-armThe final step is to login to your grafana instance at http://swarm:3000/ with the default username and password of “admin” and “admin”. Once you have logged in, simply add http://swarm:8000/ as your default grafana data source, and you are ready to use Docker.
Be the first to comment