Setting up buildbot in FreeBSD jails

We choose the released version of FreeBSD (11.1-RELEASE at the moment). There is no particular reason for it, and as a matter of fact buildbot as a Python-based server is very cross-platform; therefore the underlying OS platform and version should not make a large difference.

It will make a difference for what you do with buildbot, however. For instance, poudriere is the de-facto standard for building packages from source on FreeBSD. Builds run in jails which may be any FreeBSD base system version older or equal to the host’s version (reason will be explained below). In other words, if the host is FreeBSD 11.1, build jails created by poudriere could e.g. use 9.1, 10.3, 11.0, 11.1, but potentially not version 12 or newer because of incompatibilities with the host’s kernel (jails do not run their own kernel as full virtual machines do). To not prolong this article over the intended scope, the details of which nice things could be done or automated with buildbot are not covered.

Package names on the FreeBSD platform are independent of the OS version, since external software (as in: not part of base system) is maintained in FreeBSD ports. So, if your chosen FreeBSD version (here: 11) is still officially supported, the packages mentioned in this post should work. In the unlikely event of package name changes before you read this article, you should be able to find the actual package names like pkg search buildbot.

Other operating systems like the various Linux distributions will use different package names but might also offer buildbot pre-packaged. If not, the buildbot installation manual offers steps to install it manually. In such case, the downside is that you will have to maintain and update the buildbot modules outside the stability and (semi-)automatic updates of your OS packages.

Vagrant is a popular tool to quickly set up virtual machines from pre-built images. We are using it here for simplicity. Any form of test environment or virtual machine would suffice. If you choose to follow along using Vagrant, please install it and ensure you have a compatible hypervisor installed as well in order to run a virtual machine (for instance VirtualBox).

Official and nightly FreeBSD images for Vagrant are available. With the following commands, we create a new directory for the playground virtual machine (called "VM" from here on) and then use Vagrant to download the FreeBSD 11.1-RELEASE image. Ensure you have enough disk space: the image presented here has around 1.4 GB, and you additionally need to allocate space for the VM.

After vagrant init, the image is available to create new VMs and a Vagrantfile was created in the current directory. We must edit the file, because the metadata (contained in what Vagrant calls a "box" = disk image + metadata) is missing two pieces of information: base MAC address and shell (see bug report). Vagrant’s default shell is bash -l, but FreeBSD does not ship bash in its base system; hence we use sh. Also, we will disable synced folders as we will not need them here and they do not work out of the box (literally!). Without the commented sample configurations, the file should look as follows:

Now let’s provision the virtual machine:

If you see messages like Warning: Connection reset. Retrying…​ for a while, keep hanging on — the official FreeBSD image defaults to connect to the Internet on first startup in order to fetch and install the latest updates. This can take a few minutes and several VM reboots.

Once the VM has fully booted, we can drop into a terminal via SSH. Vagrant handles the connection details for us:

Remember we set /bin/sh as shell in the Vagrantfile? Confusingly, Vagrant 2.0.3 needs this setting to work (else fails while bringing up the virtual machine), but now totally ignores the setting and we find ourselves in csh, the default configured for the connecting user account 🙄. You can recognize it from its default vagrant@freebsd:~ % shell prompt (sh uses $ without extra information), or type ps -p $$ to show details about the shell itself (where $$ resolves to the shell process ID in all popular shells). If you are more familiar with a different shell, you could for example install and use bash like so: sudo pkg install bash && chsh && sudo chsh. If you decide to stick to the default terminal csh, ensure you do not copy-and-paste example shell command lines starting with #, as those are not interpreted as comments in interactive csh shells.

FreeBSD has been supporting the concept of jails since the start of its 4.x release series in the year 2000. This is way before its modern competitors LXC/Docker/rkt and — like most other mechanisms — OS-specific. Some people say that jails are more mature. Since I have not worked with any Linux container mechanisms after OpenVZ many years back, I cannot give any experience or comparison here, and in any case it would probably be apples vs. pears; I like pears when they lay around a little and got soft.

Jails work like a full FreeBSD environment, but access to the outer system’s resources is restricted. For example, a jail may only listen on a network interface and IP address that was assigned to it. Filesystem access and other permissions like mounting of filesystems is (configurably) limited, as well (similar to a chroot environment). The performance difference of running software in a jail vs. directly on the jailhost is usually not noticeable (somewhat related study: packet routing performance analysis by Olivier Cochard-Labbé at EuroBSDcon 2017).

No other operating systems like Linux or Windows can be run in a jail, because the kernel is shared among jailhost (this is what I will call the outer operating system in this article) and all jails. For the same reason, running e.g. FreeBSD 12 in a jail — while the host is still on FreeBSD 11 — might not work because software built for the newer OS version may expect a different kernel interface and crash if run with the older kernel.

Buildbot is a very versatile software. While I mentioned its main use as CI (Continuous Integration) and probably even CD (Continuous Delivery/Deployment) platform, it could theoretically do any automated task that runs on a computer. It’s just so that the "batteries included" are mostly related to building software. If you need something else, you can easily write build steps and other things in your Python-based master configuration file.

The main components to understand are the buildbot master and buildbot worker:

Jails are a cheap way to semantically (and security-wise) separate applications or groups of them. If we later want to move the buildbot worker component or clone it, it is easiest to have the worker — and nothing else — in a jail.

We begin by installing ezjail, a very popular and stable wrapper around FreeBSD’s jail functionality. It makes creation and administration of jails much easier.

Now it’s time to actually create the jails. Since the master offers a web UI and the worker talks to the master, both need IP addresses assigned. For simplicity, we choose local-only addresses here (network 10.0.0.0/24).

Jail networking has several gotchas, one of them being how loopback addresses are handled: namely, when accessing the IP addresses 127.0.0.1 and ::1 inside the jail, the connection does not end up on the jailhost’s loopback interface (else jails could access its parent’s services — a security hole), but the kernel rewrites those connections to the first IPv4/IPv6 address assigned to the jail. If the first assigned IP address is public and a service in the jail listens on 127.0.0.1:1234, port 1234 will suddenly be publically accessible! Therefore, the recommended practice is to have a separate network interface for jails (you could even have one per jail, but in this tutorial we want the jails to communicate with each other directly). This works by "cloning" lo0 into the new interface lo1.

The jails have successfully started, but to do something useful — like installing packages inside — we want Internet access from within the jails (at least if you decide to use the official source pkg.freebsd.org). For that purpose, we set up a NAT networking rule using one of FreeBSD’s built-in firewalls (or rather: package filters), pf.

(mind that $ must be escaped in shells and will land in /etc/pf.conf unescaped)

At this point, your SSH connection will stall (and drop after some time) because the firewall does not have a state of your existing connection. To drop out from the hanging terminal, press Enter, ~, . one after another. To understand how this keyboard shortcut closes the SSH session, please read up about escape characters in the ssh manpage. Now, please reconnect to the VM with vagrant ssh.

Apart from the master, we want to install the web user interface (called "UI" hereinafter) and Git since that is used in buildbot’s sample configuration for fetching a source project (the smaller package git-lite should be enough for fetching of most typical schemes like ssh and https).

We create a regular, unprivileged user to run the buildbot master:

The sample configuration polls a "Hello world" project every few minutes and builds it on changes. Nothing very interesting here, but it explains the principles quite well.

Time to do configure something useful, right? Not so fast! Without a worker, no build could run. For now, we copied the sample configuration to get started. In the next steps, we permanently run the master and set up a worker to actually run the builds.

The built-in mechanism for running buildbot is simply buildbot start. Since this starts the master only once, we opt for a permanent solution to start on boot. The package maintainers have thought of this and provide an rc script (such scripts manage service start, stop and other subcommands like restart/reload). It can be executed at boot (or more exactly in this tutorial: when the jail is started) to bring up the service. For that to happen, we only have to enable the service permanently and specify its working directory and user:

If you are interested how the rc script starts and stops the service, check its code at /usr/local/etc/rc.d/buildbot.

If you are still in the buildbot master jail’s shell, drop out with exit, or alternatively create a new session to the jailhost with vagrant ssh.

Like for the master, we first install required packages and then create an unprivileged user. Watch out to not mistype buildbot-master for buildbot-worker — below, we will only execute commands related to the worker. Git is used in the example builder to fetch the source code for the build. Not to be confused with the GitPoller on the master which is a "change source" i.e. regularly checks if changes exist in a repository; therefore we need Git on both master and worker for our example usage.

We are lucky: buildbot workers do not need any configuration other than the connection details because the master handles all logic. Workers are "dumb" and only perform builds locally, reporting progress and results back to the master over the connection we specified (worker connects to master at IP 10.0.0.2 using default port 9989). Most extensibility of buildbot is in the master (and its master.cfg file). However, flexibility for your actual build purposes is in the workers as well, since you have the freedom to choose a different operating system, configuration and installed software for each worker. Since we work with FreeBSD jails in this tutorial, we are "restricted" to the jailhost’s FreeBSD kernel, but can freely choose any base system and extra packages for the worker as long as the OS release version is not newer than the host (as mentioned in the introduction).

Similar to the buildbot master rc script, you will probably want to run the worker permanently:

Master and worker have been set up, and if you watch log files, activity will be visible:

Here, "processing changes" means that if a change was detected from the previous build, a new build will be triggered. The change source is explicitly connected to trigger a build in the sample configuration — no builds are triggered implicitly only because there is a Git change source; the configuration does only and exactly what you code into it 💪.

There is of course no reason to look into log files to see which build is running. Buildbot features a web-based UI to give an overview, see results, force-trigger builds and more. In the sample master configuration, the www component is already set up to serve HTTP on port 8010. In a real environment, you would not serve unencrypted HTTP or open up the non-standard port 8010 to the outside (mind how listening on port 80 needs superuser privileges). Also, our server contains more than just the buildbot UI: depending on your actual use case for CI/CD, you may also want to serve the build logs and artifacts (such as built software). Hence, we serve the UI with nginx (any other server with HTTP and Web Sockets support would work just as well), and you can later configure yourself which data you are serving to outside users, allowing everyone to see everything and even to trigger builds. By the way, the buildbot UI by default does not perform user authorization. HTTPS is not covered in this tutorial — we will use plain HTTP for test purposes. Nevertheless, the nginx configuration presented below works if you enable SSL/TLS.

(mind again that $ is escaped in the shell but not in the output file)

Remember the line config.vm.network "forwarded_port", guest: 80, host: 8999 in our Vagrantfile? Vagrant’s networking is a little different in that access to a VM’s TCP ports is not directly possible, but typically achieved by a port forward which Vagrant establishes for you. You should therefore see a welcoming nginx example page at http://localhost:8999/ (open in your computer’s browser).

Let us replace the page with an index of what’s on the server — the buildbot master is already active, while as mentioned, other items like serving build artifacts or logs might become important to you later (not in scope of this tutorial).

Reload the browser page. The buildbot UI should come up. There will be a warning about the configured buildbotURL because we use Vagrant’s port forwarding; in production, you should have direct access to https://your-ci.your-company.example.com and configure the value accordingly.

Feel free to browse around the UI. You will find the example builder runtests, our single worker on host worker0 and some other information already available. Since the example builder has a "force" scheduler configured, you can even trigger a first build now! Click "Builds > Builders > runtests > force > Start Build" and see how the build runs. It will fail when trying to run trial, the example project’s test runner because we have not installed this software on the worker (at time of writing, it was not available as separate FreeBSD package).

We are now ready to do something useful with our buildbot instance. Buildbot configuration and essentials are not covered in here — please read the official documentation to get started. The configuration at /usr/jails/master/var/buildbot-master/master.cfg is right at your fingertips and ready for editing. Here is an edit-and-reload workflow that you may need as "trial and error" strategy until you have successfully learned all the basics:

The rc script’s reload command actually calls something like buildbot reconfigure /var/buildbot-master under the hood, telling our master process to reload the configuration.

We worked in a test virtual machine for this setup, but for production grade, you may still want to adapt a few things:

The tutorial narrated about basics of FreeBSD jails and buildbot, followed by the setup of a test virtual machine featuring a buildbot master and single attached worker. With this in place, you can go on to implement your CI/CD intentions with buildbot’s explicit and programmable configuration. Good luck!