Andreas Sommer – I'm a software engineer – Blog Writing about research and programming. en-us Sun, 22 Apr 2018 00:00:00 -0000 Andreas Sommer – I'm a software engineer – Blog 144 144 Sun, 22 Apr 2018 00:00:00 -0000 Setting up buildbot in FreeBSD jails <div id="preamble"> <div class="sectionbody"> <div class="paragraph"> <p>In this article, I would like to present a tutorial to set up <a href="">buildbot</a>, a continuous integration (CI) software (like Jenkins, drone, etc.), making use of FreeBSD&#8217;s containerization mechanism "jails". We will cover terminology, rationale for using both buildbot and jails together, and installation steps. At the end, you will have a working buildbot instance using its sample build configuration, ready to play around with your own CI plans (or even CD, it&#8217;s very flexible!). Some hints for production-grade installations are given, but the tutorial steps are meant for a test environment (namely a virtual machine). Buildbot&#8217;s configuration and detailed concepts are not in scope here.</p> </div> <h2 id="_table_of_contents" class="discrete dog-blog-breakpoint">Table of contents</h2> <div id="toc" class="toc"> <div id="toctitle" class="title"></div> <ul class="sectlevel1"> <li><a href="#_choosing_host_operating_system_and_version_for_buildbot">Choosing host operating system and version for buildbot</a></li> <li><a href="#_create_a_freebsd_playground">Create a FreeBSD playground</a></li> <li><a href="#_introduction_to_jails">Introduction to jails</a></li> <li><a href="#_overview_of_buildbot">Overview of buildbot</a></li> <li><a href="#_set_up_jails">Set up jails</a></li> <li><a href="#_install_buildbot_master">Install buildbot master</a></li> <li><a href="#_run_buildbot_master">Run buildbot master</a></li> <li><a href="#_install_buildbot_worker">Install buildbot worker</a></li> <li><a href="#_run_buildbot_worker">Run buildbot worker</a></li> <li><a href="#_set_up_web_server_nginx_to_access_buildbot_ui">Set up web server nginx to access buildbot UI</a></li> <li><a href="#_run_your_first_build">Run your first build</a></li> <li><a href="#_production_hints">Production hints</a></li> <li><a href="#_finished">Finished!</a></li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_choosing_host_operating_system_and_version_for_buildbot">Choosing host operating system and version for buildbot</h2> <div class="sectionbody"> <div class="paragraph"> <p>We choose the released version of FreeBSD (<code>11.1-RELEASE</code> at the moment). There is no particular reason for it, and as a matter of fact buildbot as a Python-based server is very cross-platform; therefore the underlying OS platform and version should not make a large difference.</p> </div> <div class="paragraph"> <p>It will make a difference for what you do with buildbot, however. For instance, <a href="">poudriere</a> is the de-facto standard for building packages from source on FreeBSD. Builds run in jails which may be any FreeBSD base system version older or equal to the host&#8217;s version (reason will be explained below). In other words, if the host is FreeBSD 11.1, build jails created by poudriere could e.g. use 9.1, 10.3, 11.0, 11.1, but potentially not version 12 or newer because of incompatibilities with the host&#8217;s kernel (jails do not run their own kernel as full virtual machines do). To not prolong this article over the intended scope, the details of which nice things could be done or automated with buildbot are not covered.</p> </div> <div class="paragraph"> <p>Package names on the FreeBSD platform are independent of the OS version, since external software (as in: not part of base system) is maintained in <a href="">FreeBSD ports</a>. So, if your chosen FreeBSD version (here: 11) is still <a href="">officially supported</a>, the packages mentioned in this post should work. In the unlikely event of package name changes before you read this article, you should be able to find the actual package names like <code>pkg search buildbot</code>.</p> </div> <div class="paragraph"> <p>Other operating systems like the various Linux distributions will use different package names but might also offer buildbot pre-packaged. If not, the <a href="">buildbot installation manual</a> offers steps to install it manually. In such case, the downside is that you will have to maintain and update the buildbot modules outside the stability and (semi-)automatic updates of your OS packages.</p> </div> </div> </div> <div class="sect1"> <h2 id="_create_a_freebsd_playground">Create a FreeBSD playground</h2> <div class="sectionbody"> <div class="paragraph"> <p><a href="">Vagrant</a> is a popular tool to quickly set up virtual machines from pre-built images. We are using it here for simplicity. Any form of test environment or virtual machine would suffice. If you choose to follow along using Vagrant, please install it and ensure you have a compatible hypervisor installed as well in order to run a virtual machine (for instance <a href="">VirtualBox</a>).</p> </div> <div class="paragraph"> <p>Official and nightly <a href="">FreeBSD images for Vagrant</a> are available. With the following commands, we create a new directory for the playground virtual machine (called "VM" from here on) and then use Vagrant to download the FreeBSD 11.1-RELEASE image. Ensure you have enough disk space: the image presented here has around 1.4 GB, and you additionally need to allocate space for the VM.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">mkdir -p ~/vagrant/freebsd-11.1-buildbot cd ~/vagrant/freebsd-11.1-buildbot vagrant init freebsd/FreeBSD-11.1-RELEASE</code></pre> </div> </div> <div class="paragraph"> <p>After <code>vagrant init</code>, the image is available to create new VMs and a <code>Vagrantfile</code> was created in the current directory. We must edit the file, because the metadata (contained in what Vagrant calls a "box" = disk image + metadata) is missing two pieces of information: base MAC address and shell (see <a href="">bug report</a>). Vagrant&#8217;s default shell is <code>bash -l</code>, but FreeBSD does not ship bash in its base system; hence we use <code>sh</code>. Also, we will disable synced folders as we will not need them here and they do not work out of the box (literally!). Without the commented sample configurations, the file should look as follows:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-ruby" data-lang="ruby">Vagrant.configure("2") do |config| = "freebsd/FreeBSD-11.1-RELEASE" = "/bin/sh" config.vm.base_mac = "080027D14C66" config.vm.synced_folder ".", "/vagrant", disabled: true "forwarded_port", guest: 80, host: 8999 end</code></pre> </div> </div> <div class="paragraph"> <p>Now let&#8217;s provision the virtual machine:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">vagrant up</code></pre> </div> </div> <div class="paragraph"> <p>If you see messages like <code>Warning: Connection reset. Retrying&#8230;&#8203;</code> for a while, keep hanging on&#8201;&#8212;&#8201;the official FreeBSD image defaults to connect to the Internet on first startup in order to fetch and install the latest updates. This can take a few minutes and several VM reboots.</p> </div> <div class="paragraph"> <p>Once the VM has fully booted, we can drop into a terminal via SSH. Vagrant handles the connection details for us:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">vagrant ssh</code></pre> </div> </div> <div class="paragraph"> <p>Remember we set <code>/bin/sh</code> as shell in the <code>Vagrantfile</code>? Confusingly, Vagrant 2.0.3 needs this setting to work (else fails while bringing up the virtual machine), but now totally ignores the setting and we find ourselves in <code>csh</code>, the default configured for the connecting user account πŸ™„. You can recognize it from its default <code>vagrant@freebsd:~ % </code> shell prompt (sh uses <code>$ </code> without extra information), or type <code>ps -p []$$</code> to show details about the shell itself (where <code>$$</code> resolves to the shell process ID in all popular shells). If you are more familiar with a different shell, you could for example install and use bash like so: <code>sudo pkg install bash &amp;&amp; chsh &amp;&amp; sudo chsh</code>. If you decide to stick to the default terminal <code>csh</code>, ensure you do not copy-and-paste example shell command lines starting with <code>#</code>, as those are <em>not</em> interpreted as comments in interactive csh shells.</p> </div> </div> </div> <div class="sect1"> <h2 id="_introduction_to_jails">Introduction to jails</h2> <div class="sectionbody"> <div class="paragraph"> <p>FreeBSD has been supporting the concept of jails since the start of its 4.x release series in the year 2000. This is way before its modern competitors LXC/Docker/rkt and&#8201;&#8212;&#8201;like most other mechanisms&#8201;&#8212;&#8201;OS-specific. Some people say that jails are more mature. Since I have not worked with any Linux container mechanisms after OpenVZ many years back, I cannot give any experience or comparison here, and in any case it would probably be apples vs. pears; I like pears when they lay around a little and got soft.</p> </div> <div class="paragraph"> <p>Jails work like a full FreeBSD environment, but access to the outer system&#8217;s resources is restricted. For example, a jail may only listen on a network interface and IP address that was assigned to it. Filesystem access and other permissions like mounting of filesystems is (configurably) limited, as well (similar to a <a href="">chroot environment</a>). The performance difference of running software in a jail vs. directly on the jailhost is usually not noticeable (somewhat related study: <a href="">packet routing performance analysis</a> by Olivier Cochard-LabbΓ© at EuroBSDcon 2017).</p> </div> <div class="paragraph"> <p>No other operating systems like Linux or Windows can be run in a jail, because the kernel is shared among jailhost (this is what I will call the outer operating system in this article) and all jails. For the same reason, running e.g. FreeBSD 12 in a jail&#8201;&#8212;&#8201;while the host is still on FreeBSD 11&#8201;&#8212;&#8201;might not work because software built for the newer OS version may expect a different kernel interface and crash if run with the older kernel.</p> </div> </div> </div> <div class="sect1"> <h2 id="_overview_of_buildbot">Overview of buildbot</h2> <div class="sectionbody"> <div class="paragraph"> <p>Buildbot is a very versatile software. While I mentioned its main use as CI (Continuous Integration) and probably even CD (Continuous Delivery/Deployment) platform, it could theoretically do <em>any</em> automated task that runs on a computer. It&#8217;s just so that the "batteries included" are mostly related to building software. If you need something else, you can easily write build steps and other things in your Python-based master configuration file.</p> </div> <div class="paragraph"> <p>The main components to understand are the <strong>buildbot master</strong> and <strong>buildbot worker</strong>:</p> </div> <div class="ulist"> <ul> <li> <p><strong>buildbot master</strong>: component which parses all build configuration and other settings (notification e-mails, change sources such as Git repositories, when builds are triggered/scheduled, etc.) and distributes the actual builds to its workers.</p> </li> <li> <p><strong>buildbot worker</strong>: a dumb component which only has connection details as configuration and gets all other commands from the master, namely to run builds. There could be multiple, and in large production setups, it makes a lot of sense to put them onto powerful, separate servers. Ephemeral workers (buildbot calls them "latent workers"), i.e. dynamically created and destroyed instances, are another option and support for several cloud providers and hypervisors is included. In this article, we will start small and set up a single, jailed worker which may be enough for your first steps with buildbot. You can later easily add/move workers somewhere else if you see the need.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_set_up_jails">Set up jails</h2> <div class="sectionbody"> <div class="paragraph"> <p>Jails are a cheap way to semantically (and security-wise) separate applications or groups of them. If we later want to move the buildbot worker component or clone it, it is easiest to have the worker&#8201;&#8212;&#8201;and nothing else&#8201;&#8212;&#8201;in a jail.</p> </div> <div class="paragraph"> <p>We begin by installing <code>ezjail</code>, a very popular and stable wrapper around FreeBSD&#8217;s <a href="">jail</a> functionality. It makes creation and administration of jails much easier.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo pkg install ezjail # Create directory structure and "base jail" i.e. extract base # FreeBSD system to /usr/jails/basejail sudo ezjail-admin install</code></pre> </div> </div> <div class="paragraph"> <p>Now it&#8217;s time to actually create the jails. Since the master offers a web UI and the worker talks to the master, both need IP addresses assigned. For simplicity, we choose local-only addresses here (network</p> </div> <div class="paragraph"> <p>Jail networking has several gotchas, one of them being how loopback addresses are handled: namely, when accessing the IP addresses <code></code> and <code>::1</code> inside the jail, the connection does not end up on the jailhost&#8217;s loopback interface (else jails could access its parent&#8217;s services&#8201;&#8212;&#8201;a security hole), but the kernel rewrites those connections to the first IPv4/IPv6 address assigned to the jail. If the first assigned IP address is public and a service in the jail listens on <code></code>, port 1234 will suddenly be publically accessible! Therefore, the <a href="">recommended practice</a> is to have a separate network interface for jails (you could even have one per jail, but in this tutorial we want the jails to communicate with each other directly). This works by "cloning" lo0 into the new interface lo1.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Add a separate network interface for jails and create the cloned # interface (automatically happens at next boot as well, no need # to repeat these steps) sudo sysrc cloned_interfaces+=lo1 sudo service netif cloneup # We can assign an IP to the server ("jailhost") as well. Needed in # this tutorial so jailhost and jails can communicate (we will # serve buildbot's web user interface with nginx later). sudo sysrc ifconfig_lo1="inet netmask" # Set default network interface for jails (if not explicitly configured) sudo sysrc jail_interface=lo1 # Start ezjail's configured jails on boot sudo sysrc ezjail_enable=YES # Actually create our jails sudo ezjail-admin create -f example master "" sudo ezjail-admin create -f example worker0 "" # Start all ezjail-managed jails (will also happen on reboot because # of ezjail_enable=YES). Please ignore the warning # "Per-jail configuration via jail_* variables is obsolete" - ezjail # simply has not been changed yet to use another mechanism. sudo ezjail-admin start</code></pre> </div> </div> <div class="paragraph"> <p>The jails have successfully started, but to do something useful&#8201;&#8212;&#8201;like installing packages inside&#8201;&#8212;&#8201;we want Internet access from within the jails (at least if you decide to use the official source For that purpose, we set up a NAT networking rule using one of FreeBSD&#8217;s built-in firewalls (or rather: package filters), <code>pf</code>.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo tee /etc/pf.conf &lt;&lt;EOF ext_if = "em0" # external network interface, adapt to your hardware/network if needed jail_if = "lo1" # the interface we chose for communication between jails # Allow jails to access Internet via NAT, but avoid NAT within same network so jails can # communicate with each other no nat on \$ext_if from (\$jail_if:network) to (\$jail_if:network) nat on \$ext_if from (\$jail_if:network) to any -&gt; \$ext_if # Note: above two rules split for clarity -&gt; equivalent to this one-liner: # nat on \$ext_if from (\$jail_if:network) to ! (\$jail_if:network) -&gt; \$ext_if # No restrictions on jail network set skip on \$jail_if # Common recommended pf rules, not exactly related to this article set skip on lo0 block drop in pass out on \$ext_if # Don't lock ourselves out from SSH pass in on \$ext_if proto tcp to \$ext_if port 22 # Allow web access pass in on \$ext_if proto tcp to \$ext_if port 80 EOF # Check firewall rules syntax sudo service pf onecheck sudo sysrc pf_enable=YES sudo service pf start</code></pre> </div> </div> <div class="paragraph"> <p>(mind that <code>$</code> must be escaped in shells and will land in /etc/pf.conf unescaped)</p> </div> <div class="paragraph"> <p>At this point, your SSH connection will stall (and drop after some time) because the firewall does not have a state of your existing connection. To drop out from the hanging terminal, press <code>Enter, ~, .</code> one after another. To understand how this keyboard shortcut closes the SSH session, please read up about escape characters in the <a href=";sektion=1#ESCAPE_CHARACTERS">ssh manpage</a>. Now, please reconnect to the VM with <code>vagrant ssh</code>.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Check if Internet connection works at all fetch -o - # Copy resolv.conf to every jail to allow resolving hostnames # (note: typically added to your default ezjail flavor) sudo tee /usr/jails/master/etc/resolv.conf &lt; /etc/resolv.conf sudo tee /usr/jails/worker0/etc/resolv.conf &lt; /etc/resolv.conf # Check if Internet connection works from a jail sudo jexec master fetch -o -</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_install_buildbot_master">Install buildbot master</h2> <div class="sectionbody"> <div class="paragraph"> <p>Apart from the master, we want to install the web user interface (called "UI" hereinafter) and Git since that is used in buildbot&#8217;s sample configuration for fetching a source project (the smaller package <code>git-lite</code> should be enough for fetching of most typical schemes like ssh and https).</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo pkg -j master install git-lite py36-buildbot py36-buildbot-www # Alternative which requires installing the tool package manager `pkg` # itself inside jail: # sudo jexec master pkg install git-lite py36-buildbot py36-buildbot-www</code></pre> </div> </div> <div class="paragraph"> <p>We create a regular, unprivileged user to run the buildbot master:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Open a shell inside jail sudo jexec master sh # Instead of pw, you can use the interactive command `adduser`. We use a # random password to protect the account. Since we are always root when # doing `jexec` into a jail, we can become the user without entering the # password and therefore can forget which password was automatically generated. pw useradd -n buildbot-master -m -w random # Create directory for master mkdir /var/buildbot-master chown buildbot-master:buildbot-master /var/buildbot-master # Become unprivileged user su -l buildbot-master buildbot create-master /var/buildbot-master cp /var/buildbot-master/master.cfg.sample /var/buildbot-master/master.cfg # Switch to root user again (we did `su -l buildbot-master` earlier) exit</code></pre> </div> </div> <div class="paragraph"> <p>The sample configuration polls a "Hello world" project every few minutes and builds it on changes. Nothing very interesting here, but it explains the principles quite well.</p> </div> <div class="paragraph"> <p>Time to do configure something useful, right? Not so fast! Without a worker, no build could run. For now, we copied the sample configuration to get started. In the next steps, we permanently run the master and set up a worker to actually run the builds.</p> </div> </div> </div> <div class="sect1"> <h2 id="_run_buildbot_master">Run buildbot master</h2> <div class="sectionbody"> <div class="paragraph"> <p>The built-in mechanism for running buildbot is simply <code>buildbot start</code>. Since this starts the master only once, we opt for a permanent solution to start on boot. The package maintainers have thought of this and provide an rc script (such scripts manage service start, stop and other subcommands like restart/reload). It can be executed at boot (or more exactly in this tutorial: when the jail is started) to bring up the service. For that to happen, we only have to enable the service permanently and specify its working directory and user:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Still inside jail shell sysrc buildbot_enable=YES sysrc buildbot_basedir=/var/buildbot-master sysrc buildbot_user=buildbot-master service buildbot start # Check log file if you wish tail /var/buildbot-master/twistd.log</code></pre> </div> </div> <div class="paragraph"> <p>If you are interested how the rc script starts and stops the service, check its code at <code>/usr/local/etc/rc.d/buildbot</code>.</p> </div> </div> </div> <div class="sect1"> <h2 id="_install_buildbot_worker">Install buildbot worker</h2> <div class="sectionbody"> <div class="paragraph"> <p>If you are still in the buildbot master jail&#8217;s shell, drop out with <code>exit</code>, or alternatively create a new session to the jailhost with <code>vagrant ssh</code>.</p> </div> <div class="paragraph"> <p>Like for the master, we first install required packages and then create an unprivileged user. Watch out to not mistype <code>buildbot-master</code> for <code>buildbot-worker</code>&#8201;&#8212;&#8201;below, we will only execute commands related to the worker. Git is used in the example builder to fetch the source code for the build. Not to be confused with the <code>GitPoller</code> on the master which is a "change source" i.e. regularly checks if changes exist in a repository; therefore we need Git on both master and worker for our example usage.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo pkg -j worker0 install git-lite py36-buildbot-worker # Alternative which requires installing the tool package manager `pkg` # itself inside jail: # sudo jexec worker0 pkg install git-lite py36-buildbot-worker # Open a shell inside jail sudo jexec worker0 sh # Instead of pw, you can use the interactive command `adduser`. We use a # random password to protect the account. Since we are always root when # doing `jexec` into a jail, we can become the user without entering the # password and therefore can forget which password was automatically generated. pw useradd -n buildbot-worker -m -w random # Create directory for worker mkdir /var/buildbot-worker chown buildbot-worker:buildbot-worker /var/buildbot-worker # Become unprivileged user su -l buildbot-worker buildbot-worker create-worker /var/buildbot-worker example-worker pass # The output told us do perform some actions manually. Let's obey: cd /var/buildbot-worker # Please fill in yourself or the admin echo "Your Name &lt;;" &gt; info/admin # Worker description for display in UI echo "worker0" &gt; info/host # Switch to root user again (we did `su -l buildbot-master` earlier) exit</code></pre> </div> </div> <div class="admonitionblock note"> <table> <tr> <td class="icon"> <div class="title">Note</div> </td> <td class="content"> <div class="paragraph"> <p>Buildbot workers were previously called "slaves" and due to the politically unsound meaning, <a href="">Mozilla assigned a $15000 contribution</a> to take care of the rename, which went from documentation all the way down to source code and package names. So luckily, I do not have to write about a "slave in a jail" here πŸ‘.</p> </div> </td> </tr> </table> </div> </div> </div> <div class="sect1"> <h2 id="_run_buildbot_worker">Run buildbot worker</h2> <div class="sectionbody"> <div class="paragraph"> <p>We are lucky: buildbot workers do not need any configuration other than the connection details because the master handles all logic. Workers are "dumb" and only perform builds locally, reporting progress and results back to the master over the connection we specified (worker connects to master at IP using default port 9989). Most extensibility of buildbot is in the master (and its <code>master.cfg</code> file). However, flexibility for your actual build purposes is in the workers as well, since you have the freedom to choose a different operating system, configuration and installed software for each worker. Since we work with FreeBSD jails in this tutorial, we are "restricted" to the jailhost&#8217;s FreeBSD kernel, but can freely choose any base system and extra packages for the worker as long as the OS release version is not newer than the host (as mentioned in the introduction).</p> </div> <div class="paragraph"> <p>Similar to the buildbot master rc script, you will probably want to run the worker permanently:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Still inside jail shell sysrc buildbot_worker_enable=YES sysrc buildbot_worker_basedir=/var/buildbot-worker sysrc buildbot_worker_uid=buildbot-worker sysrc buildbot_worker_gid=buildbot-worker service buildbot-worker start # if it fails with "cannot run /usr/local/bin/twistd", apply this patch from # to the file # `/usr/local/etc/rc.d/buildbot-worker` and try again: # sed -i '' 's|command="/usr/local/bin/twistd"|command="/usr/local/bin/twistd-3.6"|' /usr/local/etc/rc.d/buildbot-worker # Check log file, should show a message "Connected to; worker is ready" tail /var/buildbot-worker/twistd.log # Back to jailhost shell exit</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_set_up_web_server_nginx_to_access_buildbot_ui">Set up web server nginx to access buildbot UI</h2> <div class="sectionbody"> <div class="paragraph"> <p>Master and worker have been set up, and if you watch log files, activity will be visible:</p> </div> <div class="listingblock"> <div class="content"> <pre># On jailhost $ tail -F /usr/jails/*/var/buildbot*/twistd.log [...] 2018-04-21 17:23:28+0000 [-] gitpoller: processing changes from "git://"</pre> </div> </div> <div class="paragraph"> <p>Here, "processing changes" means that if a change was detected from the previous build, a new build will be triggered. The change source is explicitly connected to trigger a build in the sample configuration&#8201;&#8212;&#8201;no builds are triggered <em>implicitly</em> only because there is a Git change source; the configuration does only and exactly what you code into it πŸ’ͺ.</p> </div> <div class="paragraph"> <p>There is of course no reason to look into log files to see which build is running. Buildbot features a web-based UI to give an overview, see results, force-trigger builds and more. In the sample master configuration, the <code>www</code> component is already set up to serve HTTP on port 8010. In a real environment, you would not serve unencrypted HTTP or open up the non-standard port 8010 to the outside (mind how listening on port 80 needs superuser privileges). Also, our server contains more than just the buildbot UI: depending on your actual use case for CI/CD, you may also want to serve the build logs and artifacts (such as built software). Hence, we serve the UI with nginx (any other server with HTTP and Web Sockets support would work just as well), and you can later configure yourself which data you are serving to outside users, allowing everyone to see everything and even to trigger builds. By the way, the buildbot UI by default does not perform user authorization. HTTPS is not covered in this tutorial&#8201;&#8212;&#8201;we will use plain HTTP for test purposes. Nevertheless, the nginx configuration presented below works if you enable SSL/TLS.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># On jailhost sudo pkg install nginx sudo tee /usr/local/etc/nginx/nginx.conf &lt;&lt;EOF events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root /usr/local/www/nginx; index index.html index.htm; } location /buildbot/ { proxy_pass; } location /buildbot/sse/ { # proxy buffering will prevent sse to work proxy_buffering off; proxy_pass; } # required for websocket location /buildbot/ws { proxy_http_version 1.1; proxy_set_header Upgrade \$http_upgrade; proxy_set_header Connection "upgrade"; proxy_pass; # raise the proxy timeout for the websocket proxy_read_timeout 6000s; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/local/www/nginx-dist; } } } EOF sudo sysrc nginx_enable=YES sudo service nginx start</code></pre> </div> </div> <div class="paragraph"> <p>(mind again that <code>$</code> is escaped in the shell but not in the output file)</p> </div> <div class="paragraph"> <p>Remember the line <code> "forwarded_port", guest: 80, host: 8999</code> in our Vagrantfile? Vagrant&#8217;s networking is a little different in that access to a VM&#8217;s TCP ports is not directly possible, but typically achieved by a port forward which Vagrant establishes for you. You should therefore see a welcoming nginx example page at <a href="http://localhost:8999/" class="bare">http://localhost:8999/</a> (open in your computer&#8217;s browser).</p> </div> <div class="paragraph"> <p>Let us replace the page with an index of what&#8217;s on the server&#8201;&#8212;&#8201;the buildbot master is already active, while as mentioned, other items like serving build artifacts or logs might become important to you later (not in scope of this tutorial).</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh">sudo tee /usr/local/www/nginx/index.html &lt;&lt;EOF &lt;html&gt; &lt;body&gt; &lt;a href="/buildbot/"&gt;buildbot&lt;/a&gt; &lt;!-- Since there's only one thing here right now, let's redirect automatically until you figure out which artifacts you want to put here. --&gt; &lt;script&gt; window.location.href = "/buildbot/"; &lt;/script&gt; &lt;/body&gt; &lt;/html&gt; EOF</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_run_your_first_build">Run your first build</h2> <div class="sectionbody"> <div class="paragraph"> <p>Reload the browser page. The buildbot UI should come up. There will be a warning about the configured <code>buildbotURL</code> because we use Vagrant&#8217;s port forwarding; in production, you should have direct access to <code></code> and configure the value accordingly.</p> </div> <div class="paragraph"> <p>Feel free to browse around the UI. You will find the example builder <code>runtests</code>, our single worker on host <code>worker0</code> and some other information already available. Since the example builder has a "force" scheduler configured, you can even trigger a first build now! Click "Builds &gt; Builders &gt; runtests &gt; force &gt; Start Build" and see how the build runs. It will fail when trying to run <code>trial</code>, the example project&#8217;s test runner because we have not installed this software on the worker (at time of writing, it was not available as separate FreeBSD package).</p> </div> <div class="paragraph"> <p><span class="image"><img src="/blog/2018-04-22-buildbot-setup-freebsd-jails/buildbot-www.png" alt="buildbot UI screenshot"></span></p> </div> <div class="paragraph"> <p>We are now ready to do something useful with our buildbot instance. Buildbot configuration and essentials are not covered in here&#8201;&#8212;&#8201;please read the <a href="">official documentation</a> to get started. The configuration at <code>/usr/jails/master/var/buildbot-master/master.cfg</code> is right at your fingertips and ready for editing. Here is an edit-and-reload workflow that you may need as "trial and error" strategy until you have successfully learned all the basics:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-sh" data-lang="sh"># Open a shell inside jail sudo jexec master sh # Make some changes and reload vi /var/buildbot-master/master.cfg service buildbot reload</code></pre> </div> </div> <div class="paragraph"> <p>The rc script&#8217;s <code>reload</code> command actually calls something like <code>buildbot reconfigure /var/buildbot-master</code> under the hood, telling our master process to reload the configuration.</p> </div> </div> </div> <div class="sect1"> <h2 id="_production_hints">Production hints</h2> <div class="sectionbody"> <div class="paragraph"> <p>We worked in a test virtual machine for this setup, but for production grade, you may still want to adapt a few things:</p> </div> <div class="ulist"> <ul> <li> <p>Think about using ZFS as filesystem so ezjail can take advantage of it (see manpage&#8217;s <a href="">Using ZFS</a> section). Official Vagrant images of FreeBSD are set up using UFS, not ZFS.</p> </li> <li> <p>In my company, I have set up buildbot to run package builds using <a href="">poudriere</a>. Poudriere performs clean builds by means of creating empty jails ("empty" = only FreeBSD base system installed but no packages) and starting the build within. For that to work within our buildbot worker jail, you need to allow it to create subjails, among other settings. At some point, especially if you are a friend of human-readable names and paths, you may run into the current FreeBSD mount point name length limit of 88 characters which will be <a href=";revision=318736">fixed in FreeBSD 12</a>. To work around that limitation <em>now</em>, you could set <code>ezjail_jaildir=/j</code> in ezjail.conf (<em>before</em> running <code>ezjail-admin install</code>) instead of using the longer path <code>/usr/jails</code>. Or you could choose shorter jail names like <code>w0</code> instead of <code>my-cool-project-worker0-freebsd-10.3</code>.</p> </li> <li> <p>Store the worker password in a separate file instead of hardcoding it in <code>master.cfg</code> (as done in the sample configuration). This allows you to share the configuration with software developers (e.g. commit to a version-controlled repo) or even allow them to edit it&#8201;&#8212;&#8201;without any security concerns.</p> </li> <li> <p>You should replace the sample worker name and password with own values, obviously.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_finished">Finished!</h2> <div class="sectionbody"> <div class="paragraph"> <p>The tutorial narrated about basics of FreeBSD jails and buildbot, followed by the setup of a test virtual machine featuring a buildbot master and single attached worker. With this in place, you can go on to implement your CI/CD intentions with buildbot&#8217;s explicit and programmable configuration. Good luck!</p> </div> </div> </div> Sun, 24 Apr 2017 00:00:00 -0000 Ansible best practices <div id="preamble"> <div class="sectionbody"> <div class="paragraph"> <p>Ansible can be summarized as tool for running automated tasks on servers that require nothing but Python installed on the remote side. Typically used as configuration management framework, Ansible comes with a set of key benefits:</p> </div> <div class="ulist"> <ul> <li> <p>Has simple configuration with YAML, avoiding copy-paste by applying customizable "roles"</p> </li> <li> <p>Uses inventories to scope and define the set of servers</p> </li> <li> <p>Fosters repeatable "playbook" runs, i.e. applying same configuration to a server twice should be idempotent</p> </li> <li> <p>Doesn&#8217;t suffer from feature matrix issues because by design it is a framework, not a full-fledged solution for configuration management. You cannot say "it supports only web servers X and Y, but not Z", as principally Ansible allows you to do <em>anything</em> that is possible through manual server configuration.</p> </li> </ul> </div> <div class="paragraph"> <p>For a full introduction to Ansible, better read the <a href="">documentation</a> first. This article assumes you have already made yourself familiar with the concepts and have some existing attempts of getting Ansible working for a certain use case, but want some guidance on improving the way you are working with Ansible.</p> </div> <div class="paragraph"> <p>The company behind Ansible gives <a href="">some official guidelines</a> which mostly relate to file structure, naming and other common rules. While these are helpful, as they are not immediately common sense for beginners, only a fraction of Ansible&#8217;s features and complexity of larger setups are touched by that small set of guidelines.</p> </div> <div class="paragraph"> <p>I would like to present my experience from roughly over 2 years of Ansible experience, during which I have used it for a test environment at work (allowing developers to test systems like in production), for configuring my laptop and eventually for setting up <em>this</em> server and web application, and also my home server (a Raspberry Pi).</p> </div> <h2 id="_table_of_contents" class="discrete dog-blog-breakpoint">Table of contents</h2> <div id="toc" class="toc"> <div id="toctitle" class="title"></div> <ul class="sectlevel1"> <li><a href="#_why_ansible_over_other_frameworks">Why Ansible over other frameworks?</a></li> <li><a href="#_choose_your_type_of_environment">Choose your type of environment</a> <ul class="sectlevel2"> <li><a href="#_testing">Testing</a></li> <li><a href="#_staging_production">Staging/production</a></li> <li><a href="#_both_non_production_and_production_with_one_ansible_setup">Both non-production and production with one Ansible setup</a></li> </ul> </li> <li><a href="#_careful_when_mixing_manual_and_automated_configuration">Careful when mixing manual and automated configuration</a></li> <li><a href="#_directory_structure">Directory structure</a></li> <li><a href="#_basic_setup">Basic setup</a></li> <li><a href="#inventory-safe-default">Ansible configuration</a></li> <li><a href="#_name_tasks">Name tasks</a></li> <li><a href="#avoid-skipping-items">Avoid skipping items</a></li> <li><a href="#_use_and_abuse_of_variables">Use and abuse of variables</a></li> <li><a href="#_tags">Tags</a></li> <li><a href="#_sudo_only_where_necessary">sudo only where necessary</a></li> <li><a href="#_assertions">Assertions</a></li> <li><a href="#_less_code_by_using_repetition_primitives">Less code by using repetition primitives</a></li> <li><a href="#_idempotency_done_right">Idempotency done right</a></li> <li><a href="#dynamic-inventory">Leverage dynamic inventory</a></li> <li><a href="#_modern_ansible_features">Modern Ansible features</a></li> <li><a href="#storing-sensitive-files">Off-topic: storing sensitive files</a></li> <li><a href="#_conclusion">Conclusion</a></li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_why_ansible_over_other_frameworks">Why Ansible over other frameworks?</h2> <div class="sectionbody"> <div class="ulist"> <ul> <li> <p>Honestly, I did not compare many alternatives because the Ansible environment at work already existed when I joined and soon I believed Ansible to be the best option. The usual suspects Chef and Puppet did not really please me because the recipes do not really look like "infrastructure as code", but are too declarative and hard to understand in detail without looking at many files&#8201;&#8212;&#8201;while in a typical Ansible playbook, the actions taken can be read top-down like code.</p> </li> <li> <p>Many years ago, I built my own solution to deploy my personal web applications (<a href="">"Site Deploy"</a>; UI-based). As hobby project, it never became popular or sophisticated enough, and eventually I learned that it suffers from the aforementioned feature matrix problem. Essentially it only supported the features relevant to me πŸ™„, without providing a framework to support anything on any server. Nevertheless, <em>Site Deploy</em> already had support for configuring hosts with their connection data and services, with the help of variable substitution in most places. Or in other words: the very basic concepts of Ansible.</p> </li> <li> <p>Size of the user-base says a lot (cf. <a href="">their 2016 recap</a>)</p> </li> <li> <p>Ansible aims at simple design, and becomes powerful by all the open-source modules to support services, applications, hardware, network, connections, etc.</p> </li> <li> <p>No server-side, persistent component required. Only Python needed to execute modules. Usual connection type is SSH, but custom modules are available for other types.</p> </li> <li> <p>Flat learning curve: once you understand the basic concepts (define hosts in inventory, set variables on different levels, write tasks in playbooks) and you know the commands/steps to configure a host manually, it&#8217;s easy to get started writing the same steps down in Ansible&#8217;s YAML format.</p> </li> <li> <p>Put simply, Ansible combines a set of hosts (inventory) with a list of applicable tasks (playbooks &amp; roles), customizable with variables (at different places), allowing you to use pre-defined or own task modules and plugins (connection, value lookup, etc.). If you rolled your own, generic configuration management, you probably could not implement its principles much simpler. Since the concepts are so clearly separated, the source code (Python) is easy enough to read, if ever needed. Usually you will only have 2 situations to look into Ansible source code: learning how modules should be implemented and finding out about changed behavior when upgrading Ansible. The latter is not common and only occurred to me when switching from Ansible 1.8/1.9.x to 2.2.x which was quite a big step both in features, deprecations and also Ansible source code architecture itself.</p> </li> <li> <p>Change detection and idempotency. Whenever a task is run, there may be distinct outcomes: successfully changed, failed, skipped, unchanged. After running a playbook, you will have an overview of which tasks actually made changes on the target hosts. Usually, one would design playbooks in a way that running it a second time only gives "unchanged" outcomes, and Ansible&#8217;s modules support this idea of idempotency&#8201;&#8212;&#8201;for example, a <code>command</code> task can be marked as "already done that before, no changes required" by specifying <code>creates: /file/created/by/command</code> β†’ once the file was successfully created, a repeated execution of the task module will not run the command again.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_choose_your_type_of_environment">Choose your type of environment</h2> <div class="sectionbody"> <div class="paragraph"> <p>Before we jump into practice, in the first thought we must consider what kind of Ansible-based setup we want to achieve, which greatly depends on the environment: work/personal, production/staging/testing, mixture of those&#8230;&#8203;</p> </div> <div class="sect2"> <h3 id="_testing">Testing</h3> <div class="paragraph"> <p>A test environment could have many faces: for instance, at my company we manage a separate Git repo for the test environment, unrelated to any production configuration and therefore very quick to modify for developers without lengthy code reviews or approval by devops, as no production system can be affected. Ansible is used to fully configure the system and our software within a virtual machine.</p> </div> <div class="paragraph"> <p>To spin up a VM, many solutions exist already&#8201;&#8212;&#8201;for instance <a href="">Vagrant</a> with a small provisioning script that installs everything required for Ansible (only Python πŸ˜‰) in the VM. We use a small Fabric script to bootstrap a FreeBSD VM and networking before continuing with Ansible.</p> </div> </div> <div class="sect2"> <h3 id="_staging_production">Staging/production</h3> <div class="paragraph"> <p>You should keep separate inventories for staging and production. If you don&#8217;t have staging, you should probably aim at automating staging setup with Ansible, since you already develop the production configuration in playbooks. But if you have both, the below recommendations apply.</p> </div> </div> <div class="sect2"> <h3 id="_both_non_production_and_production_with_one_ansible_setup">Both non-production and production with one Ansible setup</h3> <div class="ulist"> <ul> <li> <p>When deploying both non-production and production environments from the same roles/playbooks, you must take care they don&#8217;t interfere with each other. For instance, you don&#8217;t want to send real e-mails to customers from staging, use different domain names, etc. The main way to decide on applying non-production vs. production properties should be your use of inventories and variables. An example will be discussed below (<a href="#dynamic-inventory">dynamic inventory</a>).</p> </li> <li> <p>Careful&#8201;&#8212;&#8201;developers should not have live credentials such as SSH access to a production server, but probably be able to manage testing/staging systems?!</p> </li> <li> <p>GPG encryption of sensitive files or other protection to disallow unprivileged people from accessing production machines at all (mentioned in section <a href="#storing-sensitive-files">Storing sensitive files</a>)</p> </li> <li> <p>A safe default choice for inventories is required, and the default should most probably <em>not</em> be production. This is described below in the section <a href="#inventory-safe-default">Ansible configuration</a>.</p> </li> </ul> </div> </div> </div> </div> <div class="sect1"> <h2 id="_careful_when_mixing_manual_and_automated_configuration">Careful when mixing manual and automated configuration</h2> <div class="sectionbody"> <div class="paragraph"> <p>If you already have a production system manually set up&#8201;&#8212;&#8201;which is almost always the case, at least for initial OS installation steps which cannot be done via Ansible on physical servers&#8201;&#8212;&#8201;making the switch to fully automated configuration via Ansible is not easy. You may want to introduce automation step-by-step.</p> </div> <div class="paragraph"> <p>There are many imaginable ways to achieve that migration. I want to propose what I would do, admittedly without any real-world experience because I do not manage any production systems as developer.</p> </div> <div class="ulist"> <ul> <li> <p>Develop playbooks and maintain <a href="">check mode and the <code>--diff</code> option</a>. This is not always easy and sometimes unnerving because you have to think both in normal mode (read-write) and check mode (read-only) when writing tasks, and apply appropriate options for modules that can&#8217;t handle it themselves (like <code>command</code>):</p> <div class="ulist"> <ul> <li> <p><code>check_mode: no</code> (previously called <code>always_run: yes</code>)</p> </li> <li> <p><code>changed_when</code></p> </li> <li> <p>If you use tags: apply <code>tags: [ always ]</code> to tasks that e.g. provide results for subsequent tasks</p> </li> </ul> </div> </li> <li> <p>Take care when making manual changes to servers. While often okay and necessary to react quickly, ensure the responsible people (e.g. devops team) can later reproduce the setup rather sooner than later with playbooks.</p> </li> <li> <p>Use <a href=""><code>{{ ansible_managed }}</code></a> to mark auto-generated files as such, so nobody unknowingly edits them manually</p> </li> <li> <p>Automate as much setup as you can, but only the parts that you are able to implement via Ansible without risk. For example, if you fear that an automatic database setup could go horribly wrong (like overwrite the existing production database), then rely on your distrust and do those steps manually.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="_directory_structure">Directory structure</h2> <div class="sectionbody"> <div class="paragraph"> <p>Some <a href="">common directory layouts</a> are already part of the official documentation. In addition, you may want to separate your playbooks in subdirectories of <code>playbooks/</code> once your content grows too large. This cannot really be handled well in best practices because size and purpose of each project varies, so I just leave this on you to decide when time comes to "clean up". Note that if you use several playbook (sub-)directories and files relative to them (such as a custom <code>library</code> folder), you may have to symlink into the each directory containing playbooks.</p> </div> </div> </div> <div class="sect1"> <h2 id="_basic_setup">Basic setup</h2> <div class="sectionbody"> <div class="ulist"> <ul> <li> <p>It should be clear that Ansible uses text files and therefore should be versioned in a VCS like Git. Make sure you ignore files that should not be committed (for example in .gitignore: <code>*.retry</code>).</p> </li> <li> <p>Add something like <code>alias apl=ansible-playbook</code> in your shell. Or do you want to type <code>ansible-playbook</code> all the time?</p> </li> <li> <p>Require users to use at least a certain Ansible version, e.g. the latest version available in OS package managers at the time of starting your endeavors. You could have a little role <code>check-preconditions</code> doing this:</p> </li> </ul> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml"># Check and require certain Ansible version. You should document why that # version is required, for instance: # # We require Ansible 2.2.1 or newer, see changelog # ( # &gt; Fixes a bug where undefined variables in with_* loops would cause a task # &gt; failure even if the when condition would cause the task to be skipped. - name: Check Ansible version assert: that: '(ansible_version.major, ansible_version.minor, ansible_version.revision) &gt;= (2, 2, 1)' msg: 'Please install the recommended version 2.2.1+. You have Ansible {{ ansible_version.string }}.' run_once: yes</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="inventory-safe-default">Ansible configuration</h2> <div class="sectionbody"> <div class="paragraph"> <p><a href=""><code>ansible.cfg</code></a> allows you to tweak many settings to be a little saner than the defaults.</p> </div> <div class="paragraph"> <p>I recommend the following:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-ini" data-lang="ini">[defaults] # Default to no fact gathering because it's slow and "explicit is better # than implicit". Depending how you use variables, you may rather explicitly # define variables instead of relying on facts. You can enable this on # a per-playbook basis with `gather_facts: yes`. gathering = explicit # You should default either 1) to a non-risky inventory (not production) # or 2) point to a nonexistent one so that the person explicitly needs to # specify which one to use. I find the alternative 1) the least risky, # because 2) may lead to people creating shortcuts to deploy to live machines # which defeats the purpose of having a safer default here. inventory = inventories/test # Cows are scared of playbook developers nocows = 1 # Point to your local collection of extras, e.g. roles roles_path = ./roles [ssh_connection] # Enable SSH multiplexing to increase performance pipelining = True control_path = /tmp/ansible-ssh-%%h-%%p-%%r</code></pre> </div> </div> <div class="paragraph"> <p>Choosing a safe default for the inventory is obviously important, thinking about recent catastrophic events like the <a href="">Amazon S3 outage</a> that originated from a typo. Inventory names should not be confusable with each other, e.g. avoid using a prefix (<code>inv_live</code>, <code>inv_test</code>) because people hastily using tab completion may quickly introduce a typo.</p> </div> <div class="paragraph"> <p>If you are annoyed by <code>*.retry</code> files being created next to playbooks which hinders filename tab completion, an environment variable <code>ANSIBLE_RETRY_FILES_SAVE_PATH</code> lets you put them in a different place. For myself, I never use them as I&#8217;m not working with hundreds of hosts matching per playbook, so I just disable them with <code>ANSIBLE_RETRY_FILES_ENABLED=no</code>. Since that is a per-person decision, it should be an environment variable and not go into <code>ansible.cfg</code>.</p> </div> </div> </div> <div class="sect1"> <h2 id="_name_tasks">Name tasks</h2> <div class="sectionbody"> <div class="paragraph"> <p>While already outlined in the <a href="">mentioned best practices article</a>, I&#8217;d like to stress this point: names, comments and readability enable you and others to understand playbooks and roles later on. Ansible output on its own is too concise to really tell you the exact spot which is currently executing, and sometimes in large setups you will be searching that spot where you canceled (Ctrl+C) or a task failed fatally. Naming even the single tasks comes in handy here. Or tooling like <a href="">ARA</a> which I personally did not try yet (overkill for me). After all we&#8217;re doing programming, and no reasonable language would allow you to make public functions unnamed/anonymous.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- name: 'Create directories for service {{ daemontools_service_name }}' file: state: directory dest: '{{ item }}' owner: '{{ daemontools_service_user }}' with_items: '{{ daemontools_service_directories }}'</code></pre> </div> </div> <div class="paragraph"> <p>In recent versions of Ansible, variables in the task <code>name</code> will be correctly substituted by their value in the console output, giving you visual feedback which part of the play is executing. That will be especially important once your configuration management project is growing and you run large collections of playbooks that execute a certain role (this example: <code>daemontools_service</code>) multiple times, for example to create a couple of permanent services.</p> </div> <div class="paragraph"> <p>Another advantage of this technique is that you can start where a play canceled/failed previously using the <code>--start-at-task="Task name"</code> option. That might not always work, e.g. if a task depends on a previously <code>register:</code>-ed variable, but is often helpful to save time by skipping all previously succeeded tasks. If you use static task names like "Install packages", then <code>--start-at-task="Install packages"</code> will start at the first occurrence of that task name in the play instead of a specific one ("Install dependencies for service XYZ").</p> </div> </div> </div> <div class="sect1"> <h2 id="avoid-skipping-items">Avoid skipping items</h2> <div class="sectionbody"> <div class="paragraph"> <p>&#8230;&#8203;because it might hurt idempotency. What if your Ansible playbook adds a cronjob based on a boolean variable, and later you change the value to false? Using <code>when: my_bool</code> (value now changed to <code>no</code>) will skip the task, leaving the cronjob intact even though you expected it to be removed or disabled.</p> </div> <div class="paragraph"> <p>Here&#8217;s a slightly more complicated example: I had to set up a service that should be disabled by default until the developer enables it (because it would log error messages all the time unless the developer had established a required, manual SSH tunnel). Considerations:</p> </div> <div class="ulist"> <ul> <li> <p>When configuring that service (let&#8217;s call the role <code>daemontools_service</code>; <a href="">daemontools</a> are great to set up and manage services on *nix), we cannot simply enable/disable the service conditionally: the service should only be disabled initially (first playbook run = service created for the first time on remote machine) and on boot, but its state should be untouched if the developer had already enabled the service manually. Or in other words (since that fact is not easy to find out), leave state untouched if the service was already configured by a previous playbook run (= idempotency).</p> </li> <li> <p>You might also want an option to toggle enabling/disabling the service by default, so I&#8217;ll show that as well</p> </li> </ul> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- hosts: xyz vars: xyz_service_name: xyz-daemon # Knob to enable/disable service by default (on reboot, and after # initial configuration) xyz_always_enabled: yes roles: - role: daemontools_service daemontools_service_name: '{{ xyz_service_name }}' # Contrived variable, leaving state untouched should be the default # behavior unless you want to risk in production that services are # unintentionally enabled or disabled by a playbook run. daemontools_service_enabled: 'do_not_change_state' daemontools_service_other_variables: ... tasks: - name: Disable XYZ service on boot cron: # We know that the role will symlink into /var/service, # as usual for daemontools job: "svc -d /var/service/{{ xyz_service_name }}" name: "xyz_default_disabled" special_time: "reboot" disabled: "{{ xyz_always_enabled }}" # ...or... # state: "{{ 'absent' if xyz_always_enabled else 'present' }}" tags: [ cron ] - name: Disable XYZ service initially # After *all* initial configuration steps succeeded, take the service # down (`svc -d`) and mark the service as created so we... shell: "svc -d /var/service/{{ xyz_service_name }} &amp;&amp; touch /var/service/{{ xyz_service_name }}/.created" args: # ...don't disable the service again if playbook is run again # (as someone may have enabled the service manually in the meantime). creates: "/var/service/{{ xyz_service_name }}/.created" when: not xyz_always_enabled tags: [ cron ]</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_use_and_abuse_of_variables">Use and abuse of variables</h2> <div class="sectionbody"> <div class="paragraph"> <p>The most important principle for variables is that you should know which variables are used when looking at a portion of "Ansible code" (YAML). As an Ansible beginner, you might have 1) wondered a few times, or looked up, in which <a href="">order of precedence</a> variables are taken into account. Or 2) you might have just given up and asked the author what is happening there. Like in software development, both 1) and 2) are fatal mistakes that hamper productivity&#8201;&#8212;&#8201;code must be readable (hopefully top-down or by looking within the surrounding 100 lines) and understandable by colleagues and other contributors. The case that you even <em>had</em> to check the precedence shows the problem in the first place! <strong>Variables should be specified at exactly one place</strong> (or two places if a variable has a reasonable, overridable default value), <strong>as close as possible to their usage</strong> while still being at the relevant location and <strong>most variables should be ultimately mandatory</strong> so that Ansible loudly complains if a variable is missing. Let us look at a few examples to see what these basic rules mean.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-ini" data-lang="ini">[exampleservers] [all:vars] # Global helper variables. # # I tend to use these specific ones because when inside a role, Ansible 1.9.x # did not correctly find files/templates in some cases (if called from playbook # or dependency of other role). Not sure if that is still required for 2.x, # so don't copy-paste without understanding the need! These are really # just examples. my_playbooks_dir={{ inventory_dir + "/../playbooks" }} my_roles_dir={{ inventory_dir + "/../roles" }} # With dynamic inventories, you can structure your per-host and per-group # variables in a nicer way than this INI file top-down format. If you use # INI files, at least try to create some structure, like alphabetical sorting # for hosts and groups. [exampleservers:vars] # Here, put only variables that belong to matching servers in general, # not to a functional component ansible_ssh_user=dog</code></pre> </div> </div> <div class="paragraph"> <p>Let&#8217;s look at an example role "mysql" which installs a MySQL server, optionally creates a database and then optionally gives privileges to the database (also allows value <code>*</code> for all databases) to a user:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml"># ...contrived excerpt... - name: Ensure database {{ database_name }} exists mysql_db: name: 'ourprefix_{{ database_name }}' when: database_name is defined and database_name != "*" - name: Ensure database user {{ database_user }} exists and has access to {{ database_name }} mysql_user: name: '{{ database_user }}' password: '{{ database_password }}' priv: '{{ database_name }}.*:ALL' host: '%' when: database_user is defined and database_user # ...</code></pre> </div> </div> <div class="paragraph"> <p>The good parts first:</p> </div> <div class="ulist"> <ul> <li> <p>Once <code>database_user</code> is given, the required variable <code>database_password</code> is mandatory, i.e. not checked with another <code>database_password is defined</code>.</p> </li> <li> <p>Variables used in task names, so that Ansible output clearly tells you what <em>exactly</em> is currently happening</p> </li> </ul> </div> <div class="paragraph"> <p>But many things should be fixed here:</p> </div> <div class="ulist"> <ul> <li> <p>Role (I called this example role "mysql") is doing way too many things at once without having a proper name. It should be split up into several roles: MySQL server installation, database creation, user &amp; privilege setup. If you really find yourself doing these three things together repeatedly, you can still create an uber-role "mysql" that depends on the others.</p> </li> <li> <p>Role variables should be prefixed with the role name (e.g. <code>mysql_database_name</code>) because Ansible has no concept of namespaces or scoping these variables only to the role. This helps finding out quickly where a variable comes from. In contrast, host groups in Ansible are a way to scope variables so they are only available to a certain set of hosts.</p> </li> <li> <p>The database name prefix <code>ourprefix_</code> seems to be a hardcoded string. First of all, this led to a bug&#8201;&#8212;&#8201;privileges are not correctly applied to the user in the second task because the prefix was forgotten. The hardcoded string could be an internal variable (mark those with an underscore!) defined in the defaults file <code>roles/mysql/defaults/main.yml</code>: <code>_database_name_prefix: 'ourprefix_' # comment describing why it&#8217;s hardcoded</code>, and must be used wherever applicable. Whenever the value needs changing, you only need to touch one location.</p> </li> <li> <p>The special value <code>database_name: '*'</code> must be considered. Because the role has more than one responsibility (remember software engineering best practices?!), the variables have too many meanings. As said, there had better be a role "mysql_user" that only handles user creation and privileges&#8201;&#8212;&#8201;inside such a scoped role, using <em>one</em> special value turns out to be less bug-prone.</p> </li> <li> <p><code>database_user is defined and database_user</code> is again only necessary because the role is doing too much. In general, you should almost never use such a conditional. For no real reason, an empty value is principally allowed, and the task skipped in that case, and also if the variable is not specified. Once you decide to rename the variable and forget to replace one occurrence, you suddenly always skip the task. Whenever you can, let Ansible complain loudly when a variable is undefined, instead of e.g. skipping a task conditionally. In this example, splitting up the role is the solution to immediately make the variables mandatory. In other cases, you could introduce a default value for a role variable and allow users to override that value.</p> </li> </ul> </div> <div class="paragraph"> <p>Other practices regarding variables and their values and inline templates:</p> </div> <div class="ulist"> <ul> <li> <p>Consistently name your variables. Just like code, Ansible plays should be grep-able. A simple text search through your Ansible setup repo should immediately find the source of a variable and other places where it is used.</p> </li> <li> <p>Avoid indirections like includes or <code>vars_files</code> if possible to keep relevant variables close to their use. In some cases, these helpers can shorten repeated code, but usually they just add one more level of having to jump around between files to grasp where a value comes from.</p> </li> <li> <p>Don&#8217;t use the special one-line dictionary syntax <code>mysql_db: name="{{ database_name }}" state="present" encoding="utf8mb4"</code>. YAML is very readable per se, so why use Ansible&#8217;s crippled syntax instead? It&#8217;s okay to use for single-variable tasks, though.</p> </li> <li> <p>On the same note, remove defaults which are obvious, such as the usual <code>state: present</code>. The "official" blog post on best practices recommends otherwise, but I like to keep code short and boilerplate-less.</p> </li> <li> <p>Decide for one quoting style and use it consistently: double quotes (<code>dest: "/etc/some.conf"</code>), single quotes (<code>dest: '/etc/some.conf'</code>) plus decision if you quote things that don&#8217;t need it (<code>dest: /etc/some.conf</code>). Keep in mind that <code>dest: {{ var }}</code> is not possible (must be quoted), and that <code>mode: 0755</code> (chmod) will give an unexpected result (no octal number support), so recommended practice is of course <code>mode: '0755'</code>.</p> </li> <li> <p>Also decide for one style for spacing and writing Jinja templates. I prefer <code>dest: '{{ var|int + 5 }}'</code> over <code>dest: '{{var | int + 5}}'</code> but only staying consistent is key, not the style you choose.</p> </li> <li> <p>You don&#8217;t need <code>---</code> at the top of YAML files. Just leave them away unless you know what it means.</p> </li> </ul> </div> <div class="paragraph"> <p>More rules can be shown best in a playbook example:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- hosts: web-analytics-database vars: # Under `vars`, only put variables that really must be available in several # roles and tasks below. They have high precedence and therefore are prone # to clash with other variables of the same name (if you didn't follow # the principle of only one definition), or may set a value in one of the # below roles that you didn't want to be set! Therefore the role name # prefix is so important (`mysql_user_name` instead of `username` because # the latter might also be used in many other places and is hard to grep # for if used all over the place). # When writing many playbooks, you probably don't want to hardcode your # DBA's username everywhere, but define a variable `database_admin_username`. # The rule of putting it as close as possible to its use tells you to # create a group "database-servers" containing all database hosts and put # the variable into `group_vars/database-servers.yml` so it's only available # in the limited scope. # Using variable name prefix `wa_` for "web analytics" as example. wa_mysql_user_name_prefix: '{{ database_admin_username }}' roles: - role: mysql_server # [Comment describing why we chose MySQL 5.5...] # Alternatively (but more risky than requiring it to be defined explicitly), # this might have a default value in the role, stating the version you # normally use in production. mysql_server_version: '5.5' # Admin with full privileges - role: mysql_user mysql_user_name: '{{ wa_mysql_user_name_prefix }}_admin' # This should not have a default. Defaulting to `ALL` means that on a # playbook mistake, a new user may get all privileges! mysql_user_privileges: 'ALL' # Production passwords should not be committed to version control # in plaintext. See article section "Storing sensitive files". mysql_user_password: '{{ lookup("gpgfile", "secure/web-analytics-database.password") }}' # Read-only access - role: mysql_user mysql_user_name: '{{ wa_mysql_user_name_prefix }}_readonly' mysql_user_privileges: 'SELECT' mysql_user_password: '{{ lookup("gpgfile", "secure/web-analytics-database.readonly.password") }}' tasks: # With well-developed roles, you don't need extra {pre_}tasks!</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_tags">Tags</h2> <div class="sectionbody"> <div class="paragraph"> <p>Use tags only for limiting to tasks for speed reasons, as in "only update config files". They should not be used to select a "function" of a playbook or perform regular tasks, or else one fine day you may forget to specify <code>-t only-do-xyz</code> and it will take down Amazon S3 or so 😜. It&#8217;s a debug and speed tool and not otherwise necessary. Better make your playbooks smaller and more task-focused if you use playbooks for repeated (maintenance) tasks.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- hosts: webservers pre_tasks: - name: Include some vars (not generally recommended, see rules for variables) include_vars: file: myvars.yml # This must be tagged `always` because otherwise the variables are not available below tags: [ always ] roles: - role: mysql # ... - role: mysql_user # ... tasks: - name: Insert test data into SQL database # Mark with a separate tag that allows you to quickly apply new test # data to the existing MySQL database without having to wait for the # `mysql*` roles to finish (which would probably finish without changes). tags: [ test-sql ] # ...the task... - name: Get system info # Contrived example command - in reality you should use `ansible_*` facts! command: 'uname -a' register: _uname_call # This needs tag `always` because the below task requires the result # `_uname_call`, and also has tags. tags: [ always ] check_mode: no # Just assume this task to be "unchanged"; instead tasks that depend # on the result will detect changes. changed_when: no - name: Write system info copy: content: 'System: {{ _uname_call.stdout }}' dest: '/the/destination/path' tags: [ info ]</code></pre> </div> </div> </div> </div> <div class="sect1"> <h2 id="_sudo_only_where_necessary">sudo only where necessary</h2> <div class="sectionbody"> <div class="quoteblock"> <blockquote> <div class="paragraph"> <p>The command failed, so I used <code>sudo command</code> and it worked fine. I&#8217;m now doing that everywhere because it&#8217;s easier.</p> </div> </blockquote> </div> <div class="paragraph"> <p>It should be obvious to devops people, and hopefully also software developers, how very wrong this is. Just like you would not do that for manual commands, you also should not use <code>become: yes</code> globally for a whole playbook. Better only use it for tasks that actually need root rights. The <code>become</code> flag can be assigned to task blocks, avoiding repetition.</p> </div> <div class="paragraph"> <p>Another downside of "sudo everywhere" is that you have to take care of owner/group membership of directories and files you create, instead of defaulting to creating files owned by the connecting user.</p> </div> </div> </div> <div class="sect1"> <h2 id="_assertions">Assertions</h2> <div class="sectionbody"> <div class="paragraph"> <p>If you ever had a to debug a case where a YAML dictionary was missing a key, you will know how bad Ansible is at telling you where an error came from (does not even tell you the dictionary variable name). I have found my own way to deal with that: assert a condition before actually running into the default error message. Only a very simple plugin is required. I opened a <a href="">pull request</a> already but the maintainers did not like the approach. Still I will recommend it here because of practical experience.</p> </div> <div class="paragraph"> <p>In <code>ansible.cfg</code>, ensure you have:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-ini" data-lang="ini">filter_plugins = ./plugins/filter</code></pre> </div> </div> <div class="paragraph"> <p>Then add the plugin <code>plugins/filter/</code>:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-python" data-lang="python">from ansible import errors def _assert(value, msg=''): # You can leave this condition away if you think it's too strict. # It's supposed to help find typos and type mistakes in assertion conditions. if not isinstance(value, bool): raise errors.AnsibleFilterError('assert filter requires boolean as input, got %s' % type(value)) if not value: raise errors.AnsibleFilterError('assertion failed: %s' % (msg or '&lt;no message given&gt;',)) return '' class FilterModule(object): filter_map = { 'assert': _assert, } def filters(self): return self.filter_map</code></pre> </div> </div> <div class="paragraph"> <p>And use it like so:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- name: My task command: 'somecommand {{ (somevar|int &gt; 5)|assert("somevar must be number &gt; 5") }}{{ somevar }}'</code></pre> </div> </div> <div class="paragraph"> <p>This will only be able to test Jinja expressions, which are mostly but not 100% Python, but that should be enough.</p> </div> </div> </div> <div class="sect1"> <h2 id="_less_code_by_using_repetition_primitives">Less code by using repetition primitives</h2> <div class="sectionbody"> <div class="paragraph"> <p>Ever wrote something like this?</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- name: Do something with A command: dosomething A args: creates: /etc/somethingA when: '{{ is_admin_user["A"] }}' - name: Do something with B command: dosomething --a-little-different B args: creates: /etc/somethingB when: '{{ is_admin_user["B"] }}'</code></pre> </div> </div> <div class="paragraph"> <p>A little exaggerated, but chances are that you suffered from copy-pasting too much Ansible code a few times in your configuration management career, and had the usual share of copy-paste mistakes and typos. Use <a href=""><code>with_items</code> and friends</a> to your advantage:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-yaml" data-lang="yaml">- name: Do something with {{ }} # At a task-level scope, it's totally okay to use non-mandatory variables # because you have to read only these few lines to understand what it's # doing. Use quoting if you want to support e.g. whitespace in values - just # saying, of course it's unusual on *nix... command: 'dosomething {{ item.args|default("") }} "{{ }}"' args: creates: '/etc/something{{ }}' # This is again following the rule of mandatory variables: making dictionary # keys mandatory protects you from typos and, in this case, from forgetting # to add people to a list. Get a good error message instead of just # `KeyError: B` by using the aforementioned assert module. when: '{{ in is_admin_user|assert("User " + + " missing in is_admin_user") }}{{ is_admin_user[] }}' with_items: - name: A - name: B args: '--a-little-different'</code></pre> </div> </div> <div class="paragraph"> <p>More readable (once it gets bigger than my contrived example), and still does the same thing without being prone to copy-paste mistakes and complexity.</p> </div> </div> </div> <div class="sect1"> <h2 id="_idempotency_done_right">Idempotency done right</h2> <div class="sectionbody"> <div class="paragraph"> <p>This term was already mentioned a few times above. I want to give more hints on how to achieve repeatable playbook runs. "Idempotent" effectively means that on the second run, everything is green and no actual changes happened, which Ansible calls "ok" but in a well-developed setup means "unchanged" or "read-only action was performed".</p> </div> <div class="paragraph"> <p>The advantages should be pretty clear: not only can you see the exact <code>--diff</code> of what would happen on remote servers but also it gives visual feedback of what has <em>really</em> changed (even if you don&#8217;t use diff mode).</p> </div> <div class="paragraph"> <p>Only a few considerations are necessary when writing tasks and playbooks, and you can get perfect idempotency in most cases:</p> </div> <div class="ulist"> <ul> <li> <p>Avoid skipping items in certain cases (explained <a href="#avoid-skipping-items">above</a>)</p> </li> <li> <p>Often you need a <code>command</code> or <code>shell</code> task to perform very specific work. These tasks are always considered "changed" unless you define e.g. the <code>creates</code> argument or use <code>changed_when</code>.<br> Example: <code>changed_when: _previously_registered_process_result.stdout == ''</code><br> On the same note, you may want to use <code>failed_when</code> in special cases, like if a program exits with code 0 even on errors.</p> </li> <li> <p>Always use same inputs. For example, don&#8217;t write a new timestamp into a file at every task run, but detect that the file is already up-to-date and does not need to be changed.</p> </li> <li> <p>Use built-in modules like <code>lineinfile</code>, <code>file</code>, <code>synchronize</code>, <code>copy</code> and <code>template</code> which support the relevant arguments to get idempotency if used right. They also typically fully support checked mode and other features that are hard to achieve yourself. Avoid <code>command</code>/<code>shell</code> if built-ins can be used instead.</p> </li> <li> <p>The argument <code>force: no</code> can be used for some modules to ensure that a task is only run once. For instance, you want a configuration template copied once if not existent, but afterwards manage it manually or with other tools, use <code>copy</code> and <code>force: no</code> to only upload the file if not yet existent, but on repeated run don&#8217;t make any changes to the existing remote file. This is not exactly related to idempotency but sometimes a valid use case.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="dynamic-inventory">Leverage dynamic inventory</h2> <div class="sectionbody"> <div class="paragraph"> <p>Who needs to fiddle around carefully in check mode every time you change a production system, if there&#8217;s a staging environment which can bear a downtime if something goes wrong? Dynamic inventories can help separate staging and production in the most readable and&#8201;&#8212;&#8201;you guessed it&#8201;&#8212;&#8201;dynamic way.</p> </div> <div class="paragraph"> <p>Separate environments like test, staging or production of course have different properties like</p> </div> <div class="ulist"> <ul> <li> <p>IP addresses and networks</p> </li> <li> <p>Host and domain names (FQDN)</p> </li> <li> <p>Set of hosts. Production software may be distributed to multiple servers, while your staging may simply be installed on one server or virtual machine.</p> </li> <li> <p>Other values</p> </li> </ul> </div> <div class="paragraph"> <p>Ideally, all of these should be specified in variables, so that you can use different values for each environment in the respective inventory, but with consistent variable names. In your roles and playbooks, you can then mostly ignore the fact that you have different environments&#8201;&#8212;&#8201;except for tasks that e.g. should not or only run in production, but that should also be decided by a variable (β†’ <code>when: not is_production</code>).</p> </div> <div class="paragraph"> <p>Check the official introduction to <a href="">Dynamic Inventories</a> and <a href="">Developing Dynamic Inventory Sources</a> to understand my example inventory script. It forces the domain suffix <code>.test</code> for the "test" environment, and no suffix for the "live" environment.</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-python" data-lang="python">#!/usr/bin/env python from __future__ import print_function import argparse import json import os import sys SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) # One way to go "dynamic": decide inventory type (test, staging, production) # based on inventory directory. Remember that Ansible calls the first file # found if you specify a directory as inventory. Symlinking the same script # into different directories allows you to use one inventory script # for several environments. IS_LIVE = {'live': True, 'test': False}[os.path.basename(SCRIPT_DIR)] DOMAIN_SUFFIX = '' if IS_LIVE else '.test' host_to_vars = { 'first': { 'public_ip': '', 'public_hostname': '', }, 'second': { 'public_ip': '', 'public_hostname': '', }, } groups = { 'webservers': ['first', 'second'], } # Avoid human mistakes by applying test settings everywhere at once (instead # of inline per-variable) for host, variables in host_to_vars.items(): if 'public_hostname' in variables: # Just an example. Realistically you may want to change `public_ip` # as well, plus other variables that differ between test and production. variables['public_hostname'] += DOMAIN_SUFFIX if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--debug', action='store_true', default=False) parser.add_argument('--host') parser.add_argument('--list', action='store_true', default=False) args = parser.parse_args() def printJson(v): print(json.dumps(v, sort_keys=True, indent=4 if args.debug else None, separators=(',', ': ' if args.debug else ':'))) if is not None: printJson(host_to_vars.get(, {})) elif args.list: # Allow Ansible to only make one call to this script instead # of one per host. # See groups['_meta'] = { 'hostvars': host_to_vars, } printJson(groups) else: parser.print_usage(sys.stderr) print('Use either --host or --list', file=sys.stderr) exit(1)</code></pre> </div> </div> <div class="paragraph"> <p>Much more customization is possible with dynamic inventories. Another example: in my company, we use FreeBSD servers with our software installed and managed in jails. For developer testing, we have an Ansible setup to roughly resemble the production configuration. Unfortunately, at the time of writing, Ansible does not directly support configuration of jails or a concept of "child hosts". Therefore, we simply created an SSH connection plugin to connect to jails. Each jail looks like a regular host to Ansible, with the special naming pattern <code>jailname@servername</code>. Our dynamic inventory allows us to easily configure the hierarchy of groups &gt; servers &gt; jails and all their variables.</p> </div> <div class="paragraph"> <p>For personal and simple setups, in which only a few servers are involved, you might as well just use the INI-style inventory file format that Ansible uses by default. For the above example inventory, that would mean to split into two files <code>test.ini</code> and <code>live.ini</code> and managing them separately.</p> </div> <div class="paragraph"> <p>Dynamic inventories have one major downside compared to INI files: they don&#8217;t allow text diffs. Or in other words, you see the script change when looking at your VCS history, not the inventory diff. If you want a more explicit history, you may want a different setup: auto-generate INI inventory files with some script or template, then commit the INI files whenever you change something. Of course you will have to make sure to actually re-generate the files (potential for human mistakes!). I will leave this as exercise to you to decide.</p> </div> </div> </div> <div class="sect1"> <h2 id="_modern_ansible_features">Modern Ansible features</h2> <div class="sectionbody"> <div class="paragraph"> <p>While you may have introduced Ansible years back when it was still in v1.x or earlier stages, the framework is in very active development both by Red Hat and the community. <a href="">Ansible 2.0</a> introduced many powerful features and preparations for future improvements:</p> </div> <div class="ulist"> <ul> <li> <p><a href="">Task blocks (try-except-finally)</a>: useful to perform cleanups if a block of tasks should be applied "either all or none of the tasks". Also can reduce repeated code because you can apply <code>when</code>, <code>become</code> and other flags to a block.</p> </li> <li> <p><a href="">Dynamic includes</a>: you can now use variables in includes, e.g. <code>- include: 'server-setup-{{ environment_name }}.yml'</code></p> </li> <li> <p><a href="">Conditional roles</a> are nothing new. I had some trouble with related bugs in 1.8.x, but those are obviously resolved and <code>role: [&#8230;&#8203;] when: somecondition</code> can help in some use cases to make code cleaner (similar to task blocks).</p> </li> <li> <p>Plugins were refactored to cater for clean, more maintainable APIs, and more changes will come in 2.x updates (like the persistent connections framework). Migrating your own library to 2.x should be simple in most cases.</p> </li> </ul> </div> </div> </div> <div class="sect1"> <h2 id="storing-sensitive-files">Off-topic: storing sensitive files</h2> <div class="sectionbody"> <div class="paragraph"> <p>For this special use case, I don&#8217;t have a recommendation since I never compared different approaches.</p> </div> <div class="paragraph"> <p><a href="">Vault support</a> seems to be a good start but seems to only support protection by a single password&#8201;&#8212;&#8201;a password which you then have to share among the team.</p> </div> <div class="paragraph"> <p>Several <a href="">built-in lookups</a> exist for password retrieval and storage, such as "password" (only supports plaintext) and Ansible 2.3&#8217;s "passwordstore".</p> </div> <div class="paragraph"> <p>In my company, we store somewhat sensitive files (such as passwords for financial test systems) in our developers' Ansible test environment repository, but in GPG-encrypted form. A script contains a list of files and people and encrypts the files. The encrypted .gpg files are committed, while original files should be in <code>.gitignore</code>. Within playbooks, we use a lookup plugin to decrypt the respective files. That way, access can be limited to a "need to know" group of people. While this is not tested for production use, it may be an idea to try and incorporate this extra level of security if you are dealing with sensitive information.</p> </div> </div> </div> <div class="sect1"> <h2 id="_conclusion">Conclusion</h2> <div class="sectionbody"> <div class="paragraph"> <p>Ansible can be complex and overwhelming after developing playbooks in a wrong way for a long time. Just like for source code, readability, simplicity and common practices do not come naturally and yet are important to keep your Ansible code base lean and understandable. I&#8217;ve shown basic and advanced principles and some examples to structure your setup. Many things are left out of this general article, because either I have no experience with it yet (like Ansible Galaxy) or it would just be too much for an introductory article.</p> </div> <div class="paragraph"> <p>Happy automation!</p> </div> </div> </div>