Andreas Sommer – I'm a software engineer – Blog http://andidog.de/blog/ Writing about research and programming. en-us Sun, 22 Apr 2018 00:00:00 -0000 http://andidog.de/static/img/logo.png Andreas Sommer – I'm a software engineer – Blog http://andidog.de/blog 144 144 Sun, 22 Apr 2018 00:00:00 -0000 Setting up buildbot in FreeBSD jails http://andidog.de/blog/2018-04-22-buildbot-setup-freebsd-jails <div id="preamble"> <div class="sectionbody"> <div class="paragraph"> <p>In this article, I would like to present a tutorial to set up <a href="https://buildbot.net/">buildbot</a>, a continuous integration (CI) software (like Jenkins, drone, etc.), making use of FreeBSD&#8217;s containerization mechanism "jails". We will cover terminology, rationale for using both buildbot and jails together, and installation steps. At the end, you will have a working buildbot instance using its sample build configuration, ready to play around with your own CI plans (or even CD, it&#8217;s very flexible!). Some hints for production-grade installations are given, but the tutorial steps are meant for a test environment (namely a virtual machine). Buildbot&#8217;s configuration and detailed concepts are not in scope here.</p> </div> <h2 id="_table_of_contents" class="discrete dog-blog-breakpoint">Table of contents</h2> <div id="toc" class="toc"> <div id="toctitle" class="title"></div> <ul class="sectlevel1"> <li><a href="#_choosing_host_operating_system_and_version_for_buildbot">Choosing host operating system and version for buildbot</a></li> <li><a href="#_create_a_freebsd_playground">Create a FreeBSD playground</a></li> <li><a href="#_introduction_to_jails">Introduction to jails</a></li> <li><a href="#_overview_of_buildbot">Overview of buildbot</a></li> <li><a href="#_set_up_jails">Set up jails</a></li> <li><a href="#_install_buildbot_master">Install buildbot master</a></li> <li><a href="#_run_buildbot_master">Run buildbot master</a></li> <li><a href="#_install_buildbot_worker">Install buildbot worker</a></li> <li><a href="#_run_buildbot_worker">Run buildbot worker</a></li> <li><a href="#_set_up_web_server_nginx_to_access_buildbot_ui">Set up web server nginx to access buildbot UI</a></li> <li><a href="#_run_your_first_build">Run your first build</a></li> <li><a href="#_production_hints">Production hints</a></li> <li><a href="#_finished">Finished!</a></li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_choosing_host_operating_system_and_version_for_buildbot">Choosing host operating system and version for buildbot</h2> <div class="sectionbody"> <div class="paragraph"> <p>We choose the released version of FreeBSD (<code>11.1-RELEASE</code> at the moment). There is no particular reason for it, and as a matter of fact buildbot as a Python-based server is very cross-platform; therefore the underlying OS platform and version should not make a large difference.</p> </div> <div class="paragraph"> <p>It will make a difference for what you do with buildbot, however. For instance, <a href="https://www.freebsd.org/doc/handbook/ports-poudriere.html">poudriere</a> is the de-facto standard for building packages from source on FreeBSD. Builds run in jails which may be any FreeBSD base system version older or equal to the host&#8217;s version (reason will be explained below). In other words, if the host is FreeBSD 11.1, build jails created by poudriere could e.g. use 9.1, 10.3, 11.0, 11.1, but potentially not version 12 or newer because of incompatibilities with the host&#8217;s kernel (jails do not run their own kernel as full virtual machines do). To not prolong this article over the intended scope, the details of which nice things could be done or automated with buildbot are not covered.</p> </div> <div class="paragraph"> <p>Package names on the FreeBSD platform are independent of the OS version, since external software (as in: not part of base system) is maintained in <a href="https://www.freebsd.org/ports/">FreeBSD ports</a>. So, if your chosen FreeBSD version (here: 11) is still <a href="https://www.freebsd.org/security/#sup">officially supported</a>, the packages mentioned in this post should work. In the unlikely event of package name changes before you read this article, you should be able to find the actual package names like <code>pkg search buildbot</code>.</p> </div> <div class="paragraph"> <p>Other operating systems like the various Linux distributions will use different package names but might also offer buildbot pre-packaged. If not, the <a href="http://docs.buildbot.net/latest/manual/installation/installation.html">buildbot installation manual</a> offers steps to install it manually. In such case, the downside is that you will have to maintain and update the buildbot modules outside the stability and (semi-)automatic updates of your OS packages.</p> </div> </div> </div> <div class="sect1"> <h2 id="_create_a_freebsd_playground">Create a FreeBSD playground</h2> <div class="sectionbody"> <div class="paragraph"> <p><a href="https://www.vagrantup.com/">Vagrant</a> is a popular tool to quickly set up virtual machines from pre-built images. We are using it here for simplicity. Any form of test environment or virtual machine would suffice. If you choose to follow along using Vagrant, please install it and ensure you have a compatible hypervisor installed as well in order to run a virtual machine (for instance <a href="https://www.virtualbox.org/">VirtualBox</a>).</p> </div> <div class="paragraph"> <p>Official and nightly <a href="https://app.vagrantup.com/freebsd/">FreeBSD images for Vagrant</a> are available. With the following commands, we create a new directory for the playground virtual machine (called "VM" from here on) and then use Vagrant to download the FreeBSD 11.1-RELEASE image. Ensure you have enough disk space: the image presented here has around 1.4 GB, and you additionally need to allocate space for the VM.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">mkdir -p ~/vagrant/freebsd-11.1-buildbot cd ~/vagrant/freebsd-11.1-buildbot vagrant init freebsd/FreeBSD-11.1-RELEASE</code></pre> </div> </div> <div class="paragraph"> <p>After <code>vagrant init</code>, the image is available to create new VMs and a <code>Vagrantfile</code> was created in the current directory. We must edit the file, because the metadata (contained in what Vagrant calls a "box" = disk image + metadata) is missing two pieces of information: base MAC address and shell (see <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201904">bug report</a>). Vagrant&#8217;s default shell is <code>bash -l</code>, but FreeBSD does not ship bash in its base system; hence we use <code>sh</code>. Also, we will disable synced folders as we will not need them here and they do not work out of the box (literally!). Without the commented sample configurations, the file should look as follows:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-ruby" data-lang="ruby">Vagrant.configure("2") do |config| config.vm.box = "freebsd/FreeBSD-11.1-RELEASE" config.ssh.shell = "/bin/sh" config.vm.base_mac = "080027D14C66" config.vm.synced_folder ".", "/vagrant", disabled: true config.vm.network "forwarded_port", guest: 80, host: 8999 end</code></pre> </div> </div> <div class="paragraph"> <p>Now let&#8217;s provision the virtual machine:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">vagrant up</code></pre> </div> </div> <div class="paragraph"> <p>If you see messages like <code>Warning: Connection reset. Retrying&#8230;&#8203;</code> for a while, keep hanging on&#8201;&#8212;&#8201;the official FreeBSD image defaults to connect to the Internet on first startup in order to fetch and install the latest updates. This can take a few minutes and several VM reboots.</p> </div> <div class="paragraph"> <p>Once the VM has fully booted, we can drop into a terminal via SSH. Vagrant handles the connection details for us:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">vagrant ssh</code></pre> </div> </div> <div class="paragraph"> <p>Remember we set <code>/bin/sh</code> as shell in the <code>Vagrantfile</code>? Confusingly, Vagrant 2.0.3 needs this setting to work (else fails while bringing up the virtual machine), but now totally ignores the setting and we find ourselves in <code>csh</code>, the default configured for the connecting user account πŸ™„. You can recognize it from its default <code>vagrant@freebsd:~ % </code> shell prompt (sh uses <code>$ </code> without extra information), or type <code>ps -p []$$</code> to show details about the shell itself (where <code>$$</code> resolves to the shell process ID in all popular shells). If you are more familiar with a different shell, you could for example install and use bash like so: <code>sudo pkg install bash &amp;&amp; chsh &amp;&amp; sudo chsh</code>. If you decide to stick to the default terminal <code>csh</code>, ensure you do not copy-and-paste example shell command lines starting with <code>#</code>, as those are <em>not</em> interpreted as comments in interactive csh shells.</p> </div> </div> </div> <div class="sect1"> <h2 id="_introduction_to_jails">Introduction to jails</h2> <div class="sectionbody"> <div class="paragraph"> <p>FreeBSD has been supporting the concept of jails since the start of its 4.x release series in the year 2000. This is way before its modern competitors LXC/Docker/rkt and&#8201;&#8212;&#8201;like most other mechanisms&#8201;&#8212;&#8201;OS-specific. Some people say that jails are more mature. Since I have not worked with any Linux container mechanisms after OpenVZ many years back, I cannot give any experience or comparison here, and in any case it would probably be apples vs. pears; I like pears when they lay around a little and got soft.</p> </div> <div class="paragraph"> <p>Jails work like a full FreeBSD environment, but access to the outer system&#8217;s resources is restricted. For example, a jail may only listen on a network interface and IP address that was assigned to it. Filesystem access and other permissions like mounting of filesystems is (configurably) limited, as well (similar to a <a href="https://www.freebsd.org/doc/en/books/developers-handbook/secure-chroot.html">chroot environment</a>). The performance difference of running software in a jail vs. directly on the jailhost is usually not noticeable (somewhat related study: <a href="https://people.freebsd.org/~olivier/talks/2017_EuroBSDCon-Tuning_FreeBSD_for_routing_and_firewalling.pdf">packet routing performance analysis</a> by Olivier Cochard-LabbΓ© at EuroBSDcon 2017).</p> </div> <div class="paragraph"> <p>No other operating systems like Linux or Windows can be run in a jail, because the kernel is shared among jailhost (this is what I will call the outer operating system in this article) and all jails. For the same reason, running e.g. FreeBSD 12 in a jail&#8201;&#8212;&#8201;while the host is still on FreeBSD 11&#8201;&#8212;&#8201;might not work because software built for the newer OS version may expect a different kernel interface and crash if run with the older kernel.</p> </div> </div> </div> <div class="sect1"> <h2 id="_overview_of_buildbot">Overview of buildbot</h2> <div class="sectionbody"> <div class="paragraph"> <p>Buildbot is a very versatile software. While I mentioned its main use as CI (Continuous Integration) and probably even CD (Continuous Delivery/Deployment) platform, it could theoretically do <em>any</em> automated task that runs on a computer. It&#8217;s just so that the "batteries included" are mostly related to building software. If you need something else, you can easily write build steps and other things in your Python-based master configuration file.</p> </div> <div class="paragraph"> <p>The main components to understand are the <strong>buildbot master</strong> and <strong>buildbot worker</strong>:</p> </div> <div class="ulist"> <ul> <li> <p><strong>buildbot master</strong>: component which parses all build configuration and other settings (notification e-mails, change sources such as Git repositories, when builds are triggered/scheduled, etc.) and distributes the actual builds to its workers.</p> </li> <li> <p><strong>buildbot worker</strong>: a dumb component which only has connection details as configuration and gets all other commands from the master, namely to run builds. There could be multiple, and in large production setups, it makes a lot of sense to put them onto powerful, separate servers. Ephemeral workers (buildbot calls them "latent workers"), i.e. dynamically created and destroyed instances, are another option and support for several cloud providers and hypervisors is included. In this article, we will start small and set up a single, jailed worker which may be enough for your first steps with buildbot. You can later easily add/move workers somewhere else if you see the need.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_set_up_jails">Set up jails</h2> <div class="sectionbody"> <div class="paragraph"> <p>Jails are a cheap way to semantically (and security-wise) separate applications or groups of them. If we later want to move the buildbot worker component or clone it, it is easiest to have the worker&#8201;&#8212;&#8201;and nothing else&#8201;&#8212;&#8201;in a jail.</p> </div> <div class="paragraph"> <p>We begin by installing <code>ezjail</code>, a very popular and stable wrapper around FreeBSD&#8217;s <a href="https://www.freebsd.org/cgi/man.cgi?jail">jail</a> functionality. It makes creation and administration of jails much easier.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo pkg install ezjail # Create directory structure and "base jail" i.e. extract base # FreeBSD system to /usr/jails/basejail sudo ezjail-admin install</code></pre> </div> </div> <div class="paragraph"> <p>Now it&#8217;s time to actually create the jails. Since the master offers a web UI and the worker talks to the master, both need IP addresses assigned. For simplicity, we choose local-only addresses here (network 10.0.0.0/24).</p> </div> <div class="paragraph"> <p>Jail networking has several gotchas, one of them being how loopback addresses are handled: namely, when accessing the IP addresses <code>127.0.0.1</code> and <code>::1</code> inside the jail, the connection does not end up on the jailhost&#8217;s loopback interface (else jails could access its parent&#8217;s services&#8201;&#8212;&#8201;a security hole), but the kernel rewrites those connections to the first IPv4/IPv6 address assigned to the jail. If the first assigned IP address is public and a service in the jail listens on <code>127.0.0.1:1234</code>, port 1234 will suddenly be publically accessible! Therefore, the <a href="https://www.freebsd.org/doc/handbook/jails-ezjail.html">recommended practice</a> is to have a separate network interface for jails (you could even have one per jail, but in this tutorial we want the jails to communicate with each other directly). This works by "cloning" lo0 into the new interface lo1.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Add a separate network interface for jails and create the cloned # interface (automatically happens at next boot as well, no need # to repeat these steps) sudo sysrc cloned_interfaces+=lo1 sudo service netif cloneup # We can assign an IP to the server ("jailhost") as well. Needed in # this tutorial so jailhost and jails can communicate (we will # serve buildbot's web user interface with nginx later). sudo sysrc ifconfig_lo1="inet 10.0.0.240 netmask 255.255.255.0" # Set default network interface for jails (if not explicitly configured) sudo sysrc jail_interface=lo1 # Start ezjail's configured jails on boot sudo sysrc ezjail_enable=YES # Actually create our jails sudo ezjail-admin create -f example master "10.0.0.2/24" sudo ezjail-admin create -f example worker0 "10.0.0.3/24" # Start all ezjail-managed jails (will also happen on reboot because # of ezjail_enable=YES). Please ignore the warning # "Per-jail configuration via jail_* variables is obsolete" - ezjail # simply has not been changed yet to use another mechanism. sudo ezjail-admin start</code></pre> </div> </div> <div class="paragraph"> <p>The jails have successfully started, but to do something useful&#8201;&#8212;&#8201;like installing packages inside&#8201;&#8212;&#8201;we want Internet access from within the jails (at least if you decide to use the official source pkg.freebsd.org). For that purpose, we set up a NAT networking rule using one of FreeBSD&#8217;s built-in firewalls (or rather: package filters), <code>pf</code>.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo tee /etc/pf.conf &lt;&lt;EOF ext_if = "em0" # external network interface, adapt to your hardware/network if needed jail_if = "lo1" # the interface we chose for communication between jails # Allow jails to access Internet via NAT, but avoid NAT within same network so jails can # communicate with each other no nat on \$ext_if from (\$jail_if:network) to (\$jail_if:network) nat on \$ext_if from (\$jail_if:network) to any -&gt; \$ext_if # Note: above two rules split for clarity -&gt; equivalent to this one-liner: # nat on \$ext_if from (\$jail_if:network) to ! (\$jail_if:network) -&gt; \$ext_if # No restrictions on jail network set skip on \$jail_if # Common recommended pf rules, not exactly related to this article set skip on lo0 block drop in pass out on \$ext_if # Don't lock ourselves out from SSH pass in on \$ext_if proto tcp to \$ext_if port 22 # Allow web access pass in on \$ext_if proto tcp to \$ext_if port 80 EOF # Check firewall rules syntax sudo service pf onecheck sudo sysrc pf_enable=YES sudo service pf start</code></pre> </div> </div> <div class="paragraph"> <p>(mind that <code>$</code> must be escaped in shells and will land in /etc/pf.conf unescaped)</p> </div> <div class="paragraph"> <p>At this point, your SSH connection will stall (and drop after some time) because the firewall does not have a state of your existing connection. To drop out from the hanging terminal, press <code>Enter, ~, .</code> one after another. To understand how this keyboard shortcut closes the SSH session, please read up about escape characters in the <a href="https://www.freebsd.org/cgi/man.cgi?query=ssh&amp;sektion=1#ESCAPE_CHARACTERS">ssh manpage</a>. Now, please reconnect to the VM with <code>vagrant ssh</code>.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Check if Internet connection works at all fetch -o - http://example.com # Copy resolv.conf to every jail to allow resolving hostnames # (note: typically added to your default ezjail flavor) sudo tee /usr/jails/master/etc/resolv.conf &lt; /etc/resolv.conf sudo tee /usr/jails/worker0/etc/resolv.conf &lt; /etc/resolv.conf # Check if Internet connection works from a jail sudo jexec master fetch -o - http://example.com</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_install_buildbot_master">Install buildbot master</h2> <div class="sectionbody"> <div class="paragraph"> <p>Apart from the master, we want to install the web user interface (called "UI" hereinafter) and Git since that is used in buildbot&#8217;s sample configuration for fetching a source project (the smaller package <code>git-lite</code> should be enough for fetching of most typical schemes like ssh and https).</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo pkg -j master install git-lite py36-buildbot py36-buildbot-www # Alternative which requires installing the tool package manager `pkg` # itself inside jail: # sudo jexec master pkg install git-lite py36-buildbot py36-buildbot-www</code></pre> </div> </div> <div class="paragraph"> <p>We create a regular, unprivileged user to run the buildbot master:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Open a shell inside jail sudo jexec master sh # Instead of pw, you can use the interactive command `adduser`. We use a # random password to protect the account. Since we are always root when # doing `jexec` into a jail, we can become the user without entering the # password and therefore can forget which password was automatically generated. pw useradd -n buildbot-master -m -w random # Create directory for master mkdir /var/buildbot-master chown buildbot-master:buildbot-master /var/buildbot-master # Become unprivileged user su -l buildbot-master buildbot create-master /var/buildbot-master cp /var/buildbot-master/master.cfg.sample /var/buildbot-master/master.cfg # Switch to root user again (we did `su -l buildbot-master` earlier) exit</code></pre> </div> </div> <div class="paragraph"> <p>The sample configuration polls a "Hello world" project every few minutes and builds it on changes. Nothing very interesting here, but it explains the principles quite well.</p> </div> <div class="paragraph"> <p>Time to do configure something useful, right? Not so fast! Without a worker, no build could run. For now, we copied the sample configuration to get started. In the next steps, we permanently run the master and set up a worker to actually run the builds.</p> </div> </div> </div> <div class="sect1"> <h2 id="_run_buildbot_master">Run buildbot master</h2> <div class="sectionbody"> <div class="paragraph"> <p>The built-in mechanism for running buildbot is simply <code>buildbot start</code>. Since this starts the master only once, we opt for a permanent solution to start on boot. The package maintainers have thought of this and provide an rc script (such scripts manage service start, stop and other subcommands like restart/reload). It can be executed at boot (or more exactly in this tutorial: when the jail is started) to bring up the service. For that to happen, we only have to enable the service permanently and specify its working directory and user:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Still inside jail shell sysrc buildbot_enable=YES sysrc buildbot_basedir=/var/buildbot-master sysrc buildbot_user=buildbot-master service buildbot start # Check log file if you wish tail /var/buildbot-master/twistd.log</code></pre> </div> </div> <div class="paragraph"> <p>If you are interested how the rc script starts and stops the service, check its code at <code>/usr/local/etc/rc.d/buildbot</code>.</p> </div> </div> </div> <div class="sect1"> <h2 id="_install_buildbot_worker">Install buildbot worker</h2> <div class="sectionbody"> <div class="paragraph"> <p>If you are still in the buildbot master jail&#8217;s shell, drop out with <code>exit</code>, or alternatively create a new session to the jailhost with <code>vagrant ssh</code>.</p> </div> <div class="paragraph"> <p>Like for the master, we first install required packages and then create an unprivileged user. Watch out to not mistype <code>buildbot-master</code> for <code>buildbot-worker</code>&#8201;&#8212;&#8201;below, we will only execute commands related to the worker. Git is used in the example builder to fetch the source code for the build. Not to be confused with the <code>GitPoller</code> on the master which is a "change source" i.e. regularly checks if changes exist in a repository; therefore we need Git on both master and worker for our example usage.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo pkg -j worker0 install git-lite py36-buildbot-worker # Alternative which requires installing the tool package manager `pkg` # itself inside jail: # sudo jexec worker0 pkg install git-lite py36-buildbot-worker # Open a shell inside jail sudo jexec worker0 sh # Instead of pw, you can use the interactive command `adduser`. We use a # random password to protect the account. Since we are always root when # doing `jexec` into a jail, we can become the user without entering the # password and therefore can forget which password was automatically generated. pw useradd -n buildbot-worker -m -w random # Create directory for worker mkdir /var/buildbot-worker chown buildbot-worker:buildbot-worker /var/buildbot-worker # Become unprivileged user su -l buildbot-worker buildbot-worker create-worker /var/buildbot-worker 10.0.0.2 example-worker pass # The output told us do perform some actions manually. Let's obey: cd /var/buildbot-worker # Please fill in yourself or the admin echo "Your Name &lt;your.name@example.test&gt;" &gt; info/admin # Worker description for display in UI echo "worker0" &gt; info/host # Switch to root user again (we did `su -l buildbot-master` earlier) exit</code></pre> </div> </div> <div class="admonitionblock note"> <table> <tr> <td class="icon"> <div class="title">Note</div> </td> <td class="content"> <div class="paragraph"> <p>Buildbot workers were previously called "slaves" and due to the politically unsound meaning, <a href="https://blog.mozilla.org/blog/2015/12/10/mozilla-open-source-support-first-awards-made/">Mozilla assigned a $15000 contribution</a> to take care of the rename, which went from documentation all the way down to source code and package names. So luckily, I do not have to write about a "slave in a jail" here πŸ‘.</p> </div> </td> </tr> </table> </div> </div> </div> <div class="sect1"> <h2 id="_run_buildbot_worker">Run buildbot worker</h2> <div class="sectionbody"> <div class="paragraph"> <p>We are lucky: buildbot workers do not need any configuration other than the connection details because the master handles all logic. Workers are "dumb" and only perform builds locally, reporting progress and results back to the master over the connection we specified (worker connects to master at IP 10.0.0.2 using default port 9989). Most extensibility of buildbot is in the master (and its <code>master.cfg</code> file). However, flexibility for your actual build purposes is in the workers as well, since you have the freedom to choose a different operating system, configuration and installed software for each worker. Since we work with FreeBSD jails in this tutorial, we are "restricted" to the jailhost&#8217;s FreeBSD kernel, but can freely choose any base system and extra packages for the worker as long as the OS release version is not newer than the host (as mentioned in the introduction).</p> </div> <div class="paragraph"> <p>Similar to the buildbot master rc script, you will probably want to run the worker permanently:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Still inside jail shell sysrc buildbot_worker_enable=YES sysrc buildbot_worker_basedir=/var/buildbot-worker sysrc buildbot_worker_uid=buildbot-worker sysrc buildbot_worker_gid=buildbot-worker service buildbot-worker start # if it fails with "cannot run /usr/local/bin/twistd", apply this patch from # https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227675 to the file # `/usr/local/etc/rc.d/buildbot-worker` and try again: # sed -i '' 's|command="/usr/local/bin/twistd"|command="/usr/local/bin/twistd-3.6"|' /usr/local/etc/rc.d/buildbot-worker # Check log file, should show a message "Connected to 10.0.0.2:9989; worker is ready" tail /var/buildbot-worker/twistd.log # Back to jailhost shell exit</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_set_up_web_server_nginx_to_access_buildbot_ui">Set up web server nginx to access buildbot UI</h2> <div class="sectionbody"> <div class="paragraph"> <p>Master and worker have been set up, and if you watch log files, activity will be visible:</p> </div> <div class="listingblock"> <div class="content"> <pre># On jailhost $ tail -F /usr/jails/*/var/buildbot*/twistd.log [...] 2018-04-21 17:23:28+0000 [-] gitpoller: processing changes from "git://github.com/buildbot/hello-world.git"</pre> </div> </div> <div class="paragraph"> <p>Here, "processing changes" means that if a change was detected from the previous build, a new build will be triggered. The change source is explicitly connected to trigger a build in the sample configuration&#8201;&#8212;&#8201;no builds are triggered <em>implicitly</em> only because there is a Git change source; the configuration does only and exactly what you code into it πŸ’ͺ.</p> </div> <div class="paragraph"> <p>There is of course no reason to look into log files to see which build is running. Buildbot features a web-based UI to give an overview, see results, force-trigger builds and more. In the sample master configuration, the <code>www</code> component is already set up to serve HTTP on port 8010. In a real environment, you would not serve unencrypted HTTP or open up the non-standard port 8010 to the outside (mind how listening on port 80 needs superuser privileges). Also, our server contains more than just the buildbot UI: depending on your actual use case for CI/CD, you may also want to serve the build logs and artifacts (such as built software). Hence, we serve the UI with nginx (any other server with HTTP and Web Sockets support would work just as well), and you can later configure yourself which data you are serving to outside users, allowing everyone to see everything and even to trigger builds. By the way, the buildbot UI by default does not perform user authorization. HTTPS is not covered in this tutorial&#8201;&#8212;&#8201;we will use plain HTTP for test purposes. Nevertheless, the nginx configuration presented below works if you enable SSL/TLS.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># On jailhost sudo pkg install nginx sudo tee /usr/local/etc/nginx/nginx.conf &lt;&lt;EOF events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root /usr/local/www/nginx; index index.html index.htm; } location /buildbot/ { proxy_pass http://10.0.0.2:8010/; } location /buildbot/sse/ { # proxy buffering will prevent sse to work proxy_buffering off; proxy_pass http://10.0.0.2:8010/sse/; } # required for websocket location /buildbot/ws { proxy_http_version 1.1; proxy_set_header Upgrade \$http_upgrade; proxy_set_header Connection "upgrade"; proxy_pass http://10.0.0.2:8010/ws; # raise the proxy timeout for the websocket proxy_read_timeout 6000s; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/local/www/nginx-dist; } } } EOF sudo sysrc nginx_enable=YES sudo service nginx start</code></pre> </div> </div> <div class="paragraph"> <p>(mind again that <code>$</code> is escaped in the shell but not in the output file)</p> </div> <div class="paragraph"> <p>Remember the line <code>config.vm.network "forwarded_port", guest: 80, host: 8999</code> in our Vagrantfile? Vagrant&#8217;s networking is a little different in that access to a VM&#8217;s TCP ports is not directly possible, but typically achieved by a port forward which Vagrant establishes for you. You should therefore see a welcoming nginx example page at <a href="http://localhost:8999/" class="bare">http://localhost:8999/</a> (open in your computer&#8217;s browser).</p> </div> <div class="paragraph"> <p>Let us replace the page with an index of what&#8217;s on the server&#8201;&#8212;&#8201;the buildbot master is already active, while as mentioned, other items like serving build artifacts or logs might become important to you later (not in scope of this tutorial).</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo tee /usr/local/www/nginx/index.html &lt;&lt;EOF &lt;html&gt; &lt;body&gt; &lt;a href="/buildbot/"&gt;buildbot&lt;/a&gt; &lt;!-- Since there's only one thing here right now, let's redirect automatically until you figure out which artifacts you want to put here. --&gt; &lt;script&gt; window.location.href = "/buildbot/"; &lt;/script&gt; &lt;/body&gt; &lt;/html&gt; EOF</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_run_your_first_build">Run your first build</h2> <div class="sectionbody"> <div class="paragraph"> <p>Reload the browser page. The buildbot UI should come up. There will be a warning about the configured <code>buildbotURL</code> because we use Vagrant&#8217;s port forwarding; in production, you should have direct access to <code>https://your-ci.your-company.example.com</code> and configure the value accordingly.</p> </div> <div class="paragraph"> <p>Feel free to browse around the UI. You will find the example builder <code>runtests</code>, our single worker on host <code>worker0</code> and some other information already available. Since the example builder has a "force" scheduler configured, you can even trigger a first build now! Click "Builds &gt; Builders &gt; runtests &gt; force &gt; Start Build" and see how the build runs. It will fail when trying to run <code>trial</code>, the example project&#8217;s test runner because we have not installed this software on the worker (at time of writing, it was not available as separate FreeBSD package).</p> </div> <div class="paragraph"> <p><span class="image"><img src="/blog/2018-04-22-buildbot-setup-freebsd-jails/buildbot-www.png" alt="buildbot UI screenshot"></span></p> </div> <div class="paragraph"> <p>We are now ready to do something useful with our buildbot instance. Buildbot configuration and essentials are not covered in here&#8201;&#8212;&#8201;please read the <a href="https://docs.buildbot.net/current/index.html">official documentation</a> to get started. The configuration at <code>/usr/jails/master/var/buildbot-master/master.cfg</code> is right at your fingertips and ready for editing. Here is an edit-and-reload workflow that you may need as "trial and error" strategy until you have successfully learned all the basics:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Open a shell inside jail sudo jexec master sh # Make some changes and reload vi /var/buildbot-master/master.cfg service buildbot reload</code></pre> </div> </div> <div class="paragraph"> <p>The rc script&#8217;s <code>reload</code> command actually calls something like <code>buildbot reconfigure /var/buildbot-master</code> under the hood, telling our master process to reload the configuration.</p> </div> </div> </div> <div class="sect1"> <h2 id="_production_hints">Production hints</h2> <div class="sectionbody"> <div class="paragraph"> <p>We worked in a test virtual machine for this setup, but for production grade, you may still want to adapt a few things:</p> </div> <div class="ulist"> <ul> <li> <p>Think about using ZFS as filesystem so ezjail can take advantage of it (see manpage&#8217;s <a href="https://www.freebsd.org/cgi/man.cgi?query=ezjail#WHAT%27S_IN_A_JAIL">Using ZFS</a> section). Official Vagrant images of FreeBSD are set up using UFS, not ZFS.</p> </li> <li> <p>In my company, I have set up buildbot to run package builds using <a href="https://www.freebsd.org/doc/handbook/ports-poudriere.html">poudriere</a>. Poudriere performs clean builds by means of creating empty jails ("empty" = only FreeBSD base system installed but no packages) and starting the build within. For that to work within our buildbot worker jail, you need to allow it to create subjails, among other settings. At some point, especially if you are a friend of human-readable names and paths, you may run into the current FreeBSD mount point name length limit of 88 characters which will be <a href="https://svnweb.freebsd.org/base?view=revision&amp;revision=318736">fixed in FreeBSD 12</a>. To work around that limitation <em>now</em>, you could set <code>ezjail_jaildir=/j</code> in ezjail.conf (<em>before</em> running <code>ezjail-admin install</code>) instead of using the longer path <code>/usr/jails</code>. Or you could choose shorter jail names like <code>w0</code> instead of <code>my-cool-project-worker0-freebsd-10.3</code>.</p> </li> <li> <p>Store the worker password in a separate file instead of hardcoding it in <code>master.cfg</code> (as done in the sample configuration). This allows you to share the configuration with software developers (e.g. commit to a version-controlled repo) or even allow them to edit it&#8201;&#8212;&#8201;without any security concerns.</p> </li> <li> <p>You should replace the sample worker name and password with own values, obviously.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_finished">Finished!</h2> <div class="sectionbody"> <div class="paragraph"> <p>The tutorial narrated about basics of FreeBSD jails and buildbot, followed by the setup of a test virtual machine featuring a buildbot master and single attached worker. With this in place, you can go on to implement your CI/CD intentions with buildbot&#8217;s explicit and programmable configuration. Good luck!</p> </div> </div> </div> http://andidog.de/blog/2018-04-22-buildbot-setup-freebsd-jails