Blog

By first concentrating on monitoring and alerting for the main function of your business and system, you can cover almost all critical problems in subcomponents and infrastructure resources, without having to monitor those explicitly.

That needs explanation…​ The business in this blog article’s example is to process payments. So if payments fail, people cannot pay, and the business is at risk. Anything else is not as important, and therefore not worth to observe as the first thing. If a critical issue arises in our systems or network, the payment_errors_total metric will most likely cover that! To use IT terminology: the metric of payment failures is a significant service level indicator (SLI).

Admittedly, that will not cover if the internet, a customer, or our API are down, because payment requests would not even reach the system and therefore cannot produce logs or error metrics. That can be covered by a metric describing the payment request rate, probably by customer and payment method, since each of those have different typical traffic rates (minimum/maximum, different per timezone or day/night, etc.). We keep this shortcoming out of scope to keep the blog article simple. The point is: choose very few, essential business metrics as a start, not technical metrics.

Often, you would select the most business-relevant Prometheus metric offered by the system you want to monitor. Metrics are stored as time series and therefore very fast to query, as opposed to logs. If you use other observability tools, such as an ELK stack, you can check if Grafana supports the relevant data source. This metric would typically pertain to the methods "RED" (requests/rate, errors, duration) or "USE" (utilization, saturation, errors). The Four Golden Signals of Google’s SRE book additionally distinguishes traffic from saturation. An error metric is a good choice to start, since it is easy to determine which error rate is acceptable, and at which threshold you would consider it a serious problem.

Throughout this blog post, we will use the following example metric and simple terminology from the payments world:

We do not want to have separate metrics credit_card_payment_errors_total and bank_transfer_payment_errors_total! If you have microservices of the same type, as in this example one service per payment method, the metrics really mean the same thing. So rather improve your consistency and use just one metric name. Easy to do if your code is structured in a monorepo, by the way. If you have inconsistent names, it will take extra effort to repeat each dashboard visualization for each of the conventions, instead of just using labels to distinguish and plot the services (or other traits that you define as label dimensions, such as customers).

Metric names should be easy to find in code, so that developers can make sense of them fast in a stressful situation, or find out if/where a metric (still) exists. Here is a bad example where the metric name mysystem_payment_errors_total cannot be found in code: for request_type in ['payment', 'refund', 'status']: prometheus.Counter(name=f’mysystem_{request_type}_errors_total') (Python pseudocode).

Avoid high-cardinality metrics (many label combinations), since those take up lots of space, and queries take longer. Like for the logging rate of your systems, you might want to check for large metrics sporadically, or you may run into unnecessary cost and performance issues.

A graph (nowadays called Time series visualization) for our example metric, showing payment method and error type combinations, looks like this:

Cool graph, right? And we’re already done and have a monitored system! No, this is very, very bad! Great observability requires much more than just clicking together some visuals. The example is not sufficient to monitor a service. A graph visualization is a bad way to get an impression within milliseconds. The eyes have to scan the whole rendered graph, potentially containing multiple lines on varying bounds of the Y axis. You also need to know which thresholds are bad, or configure horizontal lines on the graph which represent warning and errors thresholds. That means lots of lines, colors, and points to look at before getting your question answered: "is this normal or do we have a problem, and where?"

Graphs can be helpful if you set them up nicely, but definitely not in the high-level part of your overview dashboard.

Instead, a Stat visualization, combined with traffic light colors, gives you the answer in milliseconds: green is good, amber (yellow) is noteworthy, red is bad. In addition, I tend to use blue as "noteworthy but may not be problematic" — kind of an early warning sign or unusually high amount of traffic, such as during sales/promotion seasons. So for me personally, I like the order green-blue-amber-red. Grafana allows choosing to color the background instead of the value line, which I recommend since then your whole screen should look green most of the time (click Panel > Display > Color mode > Background), and your eyes do not need to focus on the color of tiny graph lines. Exactly one value is shown — typically a number or human description.

Settings for the above screenshot:

To show the colors, you need to set thresholds on the Field tab. Setting them as static numbers (with the queried unit, e.g. "errors per 2 minutes") may work for the start. That is called Absolute in Grafana.

In our example though, the different payment methods and error types have very different thresholds: for instance, let’s say the credit_card payment method has remote_service_down errors very frequently because the 3rd party provider is unreliable and we cannot help it, so we want to set a higher threshold because it otherwise unnecessarily shows a problem. Or instead of a higher threshold, you could consider querying the error rate increment over 10 minutes, to even out any short spikes. To use relative thresholds, click Percentage and fill some values. They will use the Min/Max settings. For example: if you set Min = 0 and Max = 200, red background color above the 66% threshold will be shown above 66% * 200 = 132 (unit in this example: errors within 2 minutes). Everything above 66% will be red. Everything between 16% and 33% will be blue. And so on.

To set specific thresholds per value combination (here: "payment method / error type"), adjust Max:

Since Grafana live-previews your changes, it should be simple to choose good values for Max. Select a healthy time range for your system, and it should be green (note that the right-most time point is displayed, as we chose Calculation > Last). Select an incident time window, and choose a Max value to make it red. The other values (amber/blue) might then just work, since they are based on percentages. Start with values that work, and adjust them if you later see false positives (red when system is fine) or false negatives (green when system has problems). If you want human descriptions instead of numbers, you can also use the override feature (Field > Value mappings, or for specific fields: Overrides > [create an override] > Add override property > Value Mappings / No value), for instance to replace 0 with the text "no errors".

Example: our query shows the number of errors in 2 minutes. By looking at the graphed data of the last few days (paste query into Explore mode), we might find that 20 errors in 2 minutes are acceptable and should be shown as green. We therefore choose a slightly higher threshold of 25 to still be within the green color. Since we switch to blue color from 16%, we get Max = 25 * 100% / 16% = 156. As a result, red background color — which shouts "something is seriously wrong" — would be shown above Max * 66% = 103 errors in 2 minutes. You should experiment a little so that in healthy times, your dashboard remains green.

Side note: for "higher is better" metrics such as customer conversion rate (100% = all customer purchases succeed), you can just turn around the colors (green on top, red on bottom). The Max setting also defines the upper bound of the graph which is shown as part of the Stat visualization, so if values are higher, the line will cross the top of the Y axis and therefore becomes invisible. Not a big deal if Min/Max cover the expected range. You may also have the rare case of "too high and too low are both bad" metrics, e.g. a counter for payment requests where you always expect some traffic to be made, but also want to be warned if the request rate is soaring. The colors could be adapted to show both low range and high range as red, with green for the expected, normal range.

Pack only few visualizations horizontally, so the font stays large enough. Other people may work on a smaller screen than yours, or do not use full screen sizing. The repeat feature (Panel > Repeat options) makes Grafana create and align them automatically. In our example, the repeat feature is unused, but since we pulled different combinations of payment method and error type out of our (single, consistently named and labeled) metric, that will also show multiple rectangles in one Stat visualization, and try to align them on screen. In the screenshot further above, the titles are barely readable, and the visualization is large and could require scrolling on small screens. To solve that, you could:

Typical pre-built, open source dashboards may show variables like these:

Such low-level investigation dashboards are helpful, but for high-level purposes, your dashboard should show everything at one glance. If you have clusters A and B, of which one serves traffic at a time and the other one is the passive backup, you should not be required to know by heart which cluster is active. Instead, the dashboard should show health across clusters. You can still repeat your visualizations for each cluster, or query for by (cluster) if it proves helpful — but probably rather on low-level dashboards. For our example of an error metric, you want to know if it goes above a threshold, and sum(rate(…​)) does not strictly require distinction by cluster in the high-level visualizations.

Let’s set up the generation of our dashboard from code. First, the necessary tools. jb (jsonnet-bundler) will be used as jsonnet package manager, and go-jsonnet (not the much slower C++ implementation!) for the language itself.

We need jsonnet libraries for the outputs we want to generate. In this case, Grafonnet for Grafana dashboards is enough. If you want to generate non-Grafana resources, consider the kube-prometheus collection which covers much of the Kubernetes landscape (but mind its Kubernetes version compatibility matrix).

Commit jsonnetfile.json and jsonnetfile.lock.json. The lock file ensures that the next user gets the same version of libraries, so you do not need to commit the downloaded modules in the vendor directory. But that probably starts a flame war among Go developers, so please decide yourself…​

Instead of jb, you could also use Git submodules, but probably will regret it after adding more dependencies — I did not test that alternative.

First write a minimal dashboard as code and save as dashboards/payment-gateway.jsonnet:

Play around manually:

The last command outputs a valid Grafana dashboard as JSON. To manually apply it, open the test dashboard in your Grafana instance, then ⚙️ > JSON Model > copy-paste generated JSON > Save Changes. You should now see the dashboard as described by code — containing only a text panel that says "Yippie". This is the simplest development workflow. But it is very tiring to always go and copy-paste some generated blob and save it with many clicks. And the workflow is not visual (WYSIWYG). The next section explains a better way.

While there is no WYSIWYG editor for the whole conversion from jsonnet to a visual dashboard in Grafana, here is an alternative which works right now (in 2022):

Preparation in your shell:

Next, store the following script as watch.sh:

Run the script, passing the source files as argument.

The script listens for changes to the source file and then overwrites the dashboard in your Grafana instance with an API request. Open your Grafana instance and find the dashboard by its title. Mind that the uid field in source code must be set to a fixed value per dashboard in order to overwrite the dashboard instead of creating new ones.

This workflow gives you results within seconds and you only need to refresh in your browser to see saved changes. You may want to keep an eye on your terminal, since jsonnet is a compiled language and therefore spits out errors if you make coding mistakes.

Grafana provisioning allows automatic reloading of dashboards from a certain place. We want to load the committed, generated dashboards. On the other hand, we will not fully recreate Grafana and its database on every commit to some control repository — I’d call that murder by GitOps, and the sheer idea does not sound useful, as users and their settings are stored in the database, so we do not want to manage everything as code.

We can set up Grafana in various ways: via Ansible on a single server, with containers on Docker or Kubernetes, manually run on the company’s historic Raspberry Pi in the CEO’s closet, etc. They luckily all work the same way for configuration: local files.

In the dashboard provisioning section, it says you can put "one or more YAML configuration files in the provisioning/dashboards directory". We try that with a realistic setup of Grafana, and for the sake of simplicity, we assume a GitOps model, meaning that Grafana loads dashboards from committed files. You have to adapt this article yourself to your respective setup.

The following instructions show you how to do this on a self-hosted Kubernetes setup of Grafana (see their setup instructions). As I do not have a Grafana Cloud account right now, I cannot tell if the cloud offering provides this much flexibility, or any good way of using the GitOps model — if they do, the documentation misses this important piece (as of 2022-04). We use kind to simulate a production Kubernetes cluster.

To install from Grafana’s Helm chart, you need to configure it. Store the following content in grafana-values.yaml:

And continue installation:

Now open http://127.0.0.1:7878/ and log in with admin:admin.

Grafana reads dashboards from a directory structure. In the Kubernetes world, we can put generated dashboards into a ConfigMap in order to mount it into the Grafana directory structure. They however have a 1 MiB limit each. An example high-level dashboard of mine takes 115 kiB when Base64-encoded, so you also will not be able to say "we can fix this problem later", since you will reach the limit very soon. We will render all dashboards into one manifest file. That YAML file will contain one ConfigMap object per dashboard. Committing that file in a GitOps fashion is easy to add in your CI (git clone && git add && git commit && git push), but that is out of scope for this article. You could also do CIOps (kubectl apply from CI; not recommended), or just rsync -r --delete if you have Grafana on physical, mutable hardware and not on Kubernetes, or whatever other way of deployment to the directory structure. Make sure you overwrite during deployment instead of only adding new files/dashboards, since deletion and cleanup of technical debt is just as important as it is for writing software. Treat your Grafana instance and database as something that gets reset regularly, from a committed state. This avoids people making manual, unreviewed edits.

As you can see in the configuration, we use k8s-sidecar to automatically collect all dashboard JSON files into one directory for use by Grafana. Each ConfigMap must have the label collect-me: grafana-dashboard to get picked up. The following script creates such ConfigMap manifests. I do not explain here how to integrate it with your specific CI tool, but that should be easy if it works locally. Save the script as render-and-configmap.sh.

Ensure you are still pointing KUBECONFIG to the desired Kubernetes cluster, and test dashboard deployment like so:

Head over to the Grafana instance running in Kubernetes, and you see that the dashboard was already loaded. Integrate this with your CI pipeline, et voilà, you have a GitOps workflow!

Once done with the cluster, you can delete it:

You will surely have different environments, such as dev/staging/prod. They have varying URLs, potentially different set of running systems, and thresholds for production not always make sense in pre-production environments. Pass --ext-str myCompanyEnv=prod to the jsonnet tool to pass in a variable which you can use inside the source code. This allows previewing dashboards with development data before a software feature even goes live, and you will have very consistent views across environments. Example usage:

Alternatively, --ext-code-file seems also a viable option, but I have no experience with it (see external blog post Grafana dashboards and Jsonnet which showcases this parameter).

This can also be interesting if you have a dev/prod split for your Grafana instances.

A simple dashboard should be a matter of only few lines of code. Follow the jsonnet tutorial to learn more how to achieve that. You will see some similarities with Python, for example string formatting with %, slicing, array comprehension, modules/imports and other syntax that can make your code easier and shorter.

If you think your users are not technical enough, jsonnet may not be a good fit unless you either provide high-level functions, or replace the whole jsonnet+Grafonnet rendering idea with your own custom solution that does the same thing: output dashboard definitions as Grafana-compatible JSON.

Use the folder structure to organize your dashboards. You and your colleagues will surely play around with test dashboards, and mixing them with production-ready, usable ones is not helpful. Also, particularly if you have many systems to watch, you want everything categorized for easy access. Product-unrelated dashboards, such as monitoring for Kubernetes clusters or infrastructure, can go into a separate category. Unfortunately, you cannot set the parent folder through jsonnet as of 2022-04, but it has to be achieved as part of deploying the generated dashboards.

For very detailed dashboards, you may have a lot of graphs. While this is typically discouraged, your software may really have so many important metrics. In such case, group them into rows. They are collapsible and ease navigation. Maybe Grafana could consider adding a "Table of contents" feature to jump around quickly on a dashboard, using a navigation sidebar.

In the developer workflow above, we explicitly set dashboards to editable. You may want this for the GitOps/CI workflow as well. This is helpful because incidents sometimes require a bit of playing around with shown data. Users should however not save changes, since they are supposed to be overwritten regularly by deploying dashboards from committed code.

By default, hovering over a graph with many series shows them in a box in alphabetical order of the display label, e.g. sorted by {{customer_name}}. Typically however, such as for error rate metrics, you want the top values shown first, since the bottom of the tooltip may be cut off in case of many entries. Go to setting Display > Hover tooltip > Sort order and adjust to your liking (e.g. Decreasing). With jsonnet, use grafana.graphPanel.new(sort="decreasing") (not documented as of 2022-04).

Do not confuse the tooltip with the legend (which also has a configurable sort order!).

Many metrics only produce non-negative numbers. Graphs with such a metric on the Y axis should therefore have Visualization > Axes > Y-Min set to 0 instead of auto in order to save screen space by not showing the negative area. Another problem is that you often want 0 to be the lower bound, or else Grafana chooses the display range based on the available data. In jsonnet: grafana.graphPanel.new(min=0). Setting the maximum may be helpful if you know the number range (e.g. disk full 0-100%) and want to have a consistent display.

See how the bad example on the left makes you think of a fluctuating metric. The corrected example on the right shows that in reality, the value is quite stable. In general, make trends easier to recognize for the eyes.

Be cautious with your use case, though. If you want to display and warn when a disk gets full, for example, you better extrapolate and display the trend. Or use a Stat visualization with a warning color once the trend reaches "disk becomes full within 30 days". The respective alerts need to be designed in a similar way. Otherwise, if a short time range is selected, the user may not see that the disk usage is going up a lot, as the difference between 200 GiB and 210 GiB may not look dramatic with Y-Min set to zero.

Use grafana.graphPanel.new(…​).addLinks(…​) to create panel links:

For links to detailed dashboards, consistently pre-select a reasonable time frame such as now-30m.

If you consistently tag dashboards, you can use dashboard links to put clickable links to related dashboards on top. You can also add external links such as other company tools. I have not used this feature yet and typically rather repeat the links on each panel since that does not require scrolling all the way to the top. With jsonnet, it is easy to provide a consistent set of links (as dashboard or panel links).

In rare cases, you want a repetitive variable such as datacenter = cluster="dc"\,host=~"server.*" so that queries become less verbose: sum by (payment_method, error_type) (rate(payment_errors_total{$datacenter}[2m])). If the value is used in a label filter of a Prometheus query, as in this example, remember that commas need to be escaped with a backslash, or else Grafana treats the comma as separator between different choices for the variable value.

Even if you use jsonnet, you should use variables instead of filling a hardcoded value into each query. This allows users to change all visualizations on a dashboard at once (at ⚙️ > Variables).

Consider hiding those variables on the dashboard if their sole purpose is to avoid repeated, hardcoded values. See also below for some rationale.

You do not want to be looking at a development dashboard while debugging a production issue, so make that mistake impossible to happen. Possible solutions:

The grid position must be specified explicitly:

See Panel size and position documentation — width is split in 24 columns, height is 30 pixels each. I recommend you use 12 or 24 columns width for readability on small screens, and set a reasonable, consistent height for all visualizations on a dashboard.

As of 2022-04, you cannot easily align visualizations using jsonnet. You can hardcode x/y absolute values to your liking, but that is a hassle since you do not want to develop a user interface in an absolute grid, right? I recommend setting both to 0 in order to automatically align the visualizations on screen.

Choose the right unit, e.g. Seconds, Requests per second, etc.

Mind subtle differences between the built-in choices, e.g. Duration / seconds will show long text such as "412 milliseconds" which makes it hard to put much information on one screen — consider using Time / seconds instead.

Also, do not confuse the order of magnitude: if your data is provided in seconds, do not choose Time / milliseconds since that would show falsified values.

Grafana’s default is to use the browser timezone. Particularly for international companies or those who have international customers, consistent values and avoidance of confusion are important. Employees usually do not open the user settings page, for example to choose light/dark mode or their timezone preference, resulting in inconsistent customer and incident communication regarding dates and times. I am working from Germany and keep seeing confusion between CET/CEST once daylight saving time toggles, and sometimes even do such mistakes myself. Let’s avoid that and communicate only in UTC, and default to UTC in tools such as Grafana.

By writing a jsonnet wrapper function instead of using raw calls to grafana.dashboard.new, you can set that as your default for generated dashboards.

Somewhat related xkcd comic: ISO 8601. Did you know that the Z suffix in 2022-04-21T17:13Z stands for UTC ("Zulu time") and is therefore a pretty good abbreviation? Most non-technical people rather know the suffix "UTC", so that one is preferable.

Even if you rely on one Prometheus-compatible source in the beginning, you will very likely add more data sources, or migrate to another one, in the future. Therefore, explicitly define the source in each visualization. In jsonnet, use for example grafana.statPanel.new(datasource='thanos'). In general, do not ever name something default, anywhere. The same applies to the words "old" and "new", since "new" is always the next "old".

Those are hard to set up since the UI does not give guidance. You have to set several options correctly to see reasonable results:

Instead of a histogram, laying out the information as percentiles on a Stat visualization may give a faster overview and is preferable on high-level dashboards. For example, p50 (median), p95 and p99 percentiles are often useful. Make sure you use the rate function inside histogram_quantile.

Example Prometheus query: histogram_quantile(0.95, sum(rate(prometheus_http_request_duration_seconds_bucket[2m])) by (le))

Grafana annotations can mark interesting time points on graphs. Among many imaginable events to enrich on your dashboards, software and infrastructure deployments are the most interesting ones since change in a technology-driven company usually means risk and the potential for failure. In an incident, the starting point is often known quite soon by looking at dashboards. If graphs additionally show whether and when changes were made, you have better chances to find the cause.

Do not bother adding annotations manually (e.g. time window of every deployment), since people will forget the procedure, get the timezone wrong, and it only adds an unnecessary burden which should be automated.

Here is an example how to consistently show Argo CD sync events on your dashboards. Those mostly relate to real deployments. When I developed that query, no better, human-level event type was available. You may want to tweak this to your own use cases.

You can now use grafana.dashboard.new(…​).addAnnotation(deployments) to add the annotations to your dashboard.

Each visualization’s title should be self-describing. Bad: "Error rate". Good: "Payment methods — error rate of requests to provider".

One reason is because you can link to a single visualization which is helpful during incidents to tell others what exactly you are looking at (or to present one detail on a really large TV):

And again, it helps the eyes to quickly get a full picture instead of having to look at multiple locations on screen.

In Grafana, the dashboard title is always displayed, even for such single-visualization URLs. So if your dashboard is nicely titled "Payment gateway (high-level)", that will already be a good starting point and you may not even need or want verbose visualization titles.

For averaging queries like sum(rate(metric[5m])), which may constitute most of your dashboards, you should consider adding the interval hint (e.g. abbreviated [5m]) to the visualization title — and/or the Y axis — so that users are aware how fast a recovered metric or an error peak will become visible.

The Prometheus naming practices page gives very good guidance, such as to use lower_snake_case, name counters xxx_total or specify the unit such as xxx_seconds.

The 3 current pillars of observability — metrics, logs and traces — may not remain considered the best solution forever. We can expect tooling to try and combine them in the future, such as "metrics from logs" features. You want to observe your applications with minimum instrumentation effort? Then sometimes, a LogQL query such as sum(count_over_time(…​ [15m])) to look for specific log lines may be what you want (temporarily), instead of developing and maintaining a new metric. Beware however that log text tends to change much more frequently than metric names, and how much slower and more expensive it is to query logs. A totally uninstrumented application can easily be monitored if you have access to its logs. Later on, you can make the dashboards more efficient once you learned which indicators are important to show the application health, and which ones are not. Very helpful if you are just getting started with observability.

You can use > 0 or topk(…​) > 5 to display only offenders in your high-level dashboard (please also read Keep panels small on screen above).

For example, the customers with the highest concurrency of API requests. Use Value mappings feature to map "null" to e.g. "currently low concurrency" for better understanding in humans (since Stat visualizations always show something). Together with green/yellow/red thresholds, this explains in 2 seconds what the current value is and whether it is problematic. As explained before, use Calculation > Last if only the latest value is relevant — you do not care about the Average API concurrency over 3 hours while debugging an incident, right?

In our payment example, we could alternatively show the payment methods with the highest rate of errors. Or depending on the business, define each payment method’s business importance in code and then only show the most critical products with a label filter (e.g. importance="boss_says_this_is_super_critical"; or name them "Tier1", "Tier2", etc.).

Note that topk(5, …​) may not show you the top 5 items if evaluated for a graph, since the "top 5" are checked for many time points and all resulting items are shown. The same applies to Stat visualizations — unless you choose Instant to only choose the end time point, but that can falsify the desired data to show. If you really want only up to 5 items to be shown, use the @ end() modifier to evaluate only the latest top-5 values in the range (thanks to a blog reader for pointing this out! – available since Prometheus v2.25.0).

In a modern infrastructure, you might run synthetic test traffic to verify the end-to-end health of your applications. Since those are not from real customers, you should check if that should be shown differently or excluded from certain dashboards or visualizations.

If your business is mostly in a certain region or timezone of the world, such as European payments, traffic goes down at night. Consider different error and request thresholds at day and night, respectively. Visualizations should be clearly distinguished with e.g. 🔆 or 🌒 in the title.

Prometheus allows the time distinction with and/unless hour() >=6 <21. This can be tricky, though: in special cases such as calculations sum(…​)/sum(…​) and hour() >=6 <21, label set matching will surprise you with an empty result. Example to fix that: (sum by (something) (rate(some_metric[15m]))) / sum by (something) (rate(some_metric2[15m])) and ignoring(something) hour() >=6 <21.

This is cumbersome and should be avoided for the start, unless you really need such a strong distinction by time.

The same applies to calculations like rate(…​) / rate(…​). Why? Any change to the labels will make them explode into many series. Combine rate with sum or sum by.

A counter in Prometheus represents a value that can only increase. In contrast, a gauge can take an arbitrary value.

In regular scrape intervals, a metric’s value gets collected by Prometheus. A longer scrape interval means less storage and cost, but can mean that a short spike of a gauge’s value is not stored at all, and therefore also will not produce an alert. So prefer a counter if possible for your for use case, since its value does not lose increments (but on the other hand, it only supports increments).

Good example for using a gauge: queue size. Items can be processed, i.e. removed from the queue, or added. The more interesting metrics for queues however are error rate and per-item processing time.

To find a root cause quickly in case of problems, dashboards must allow drilling down into details. In our example of payment methods as products, each of them could fail separately, or several/all at once. This must be visible in visualizations and alert messages.

Examples why this distinction is important:

Other ideas for details to drill down into: per customer, per Kubernetes cluster, per cloud region, per API endpoint. For some of these, you may be able to leverage variables (mind Avoid dropdowns for variable values), while some value ranges may simply be too large — for instance if you have a million customers — and you should rather show the top N problematic ones (Show only offenders or top N problematic items).

A test environment could have many faces: for instance, at my company we manage a separate Git repo for the test environment, unrelated to any production configuration and therefore very quick to modify for developers without lengthy code reviews or approval by devops, as no production system can be affected. Ansible is used to fully configure the system and our software within a virtual machine.

To spin up a VM, many solutions exist already — for instance Vagrant with a small provisioning script that installs everything required for Ansible (only Python 😉) in the VM. We use a small Fabric script to bootstrap a FreeBSD VM and networking before continuing with Ansible.

You should keep separate inventories for staging and production. If you don’t have staging, you should probably aim at automating staging setup with Ansible, since you already develop the production configuration in playbooks. But if you have both, the below recommendations apply.

Globally familiar words are often clear by themselves, and typically can be looked up in an English dictionary without having more information on the context. The dictionary should usually give no ambiguities. One example is Billing, which expresses that the topic is centered around bills in some sense. If all code had globally intuitive words only, our understanding would be perfect! Reality is that there’s almost always a context which describes (required) details.

Exceptions are truly global names. Think of public trademarks and product names known worldwide. These don’t need explanations anymore, but still the companies behind them have to ensure that the understanding does not get altered through the years. Some people wish for the term "Java" to only relate to an island… A good example is "Microsoft Office". It is familiar all around the globe even for non-tech people (keep this hint in mind).

Even if Billing is easily comprehensible, the example namespace Billing::Aggregation is probably gibberish to someone who is not from the financial sector, or even new hires who don’t know yet how the company handles bills — namely, in this example, by aggregating some key figures. The word may therefore as well be very specific to the company, with a different understanding in other businesses.

A local context also applies to the method name getBankList. Without extra information like a documenting comment, the signature is not enough to find out 1) what kind of list this is, 2) if "get" means downloading, retrieving via remote call or parsing from file, and so on. At best, the surrounding class/module/project is clear enough to the reader of the code to understand the concept later, or provides a clarifying unit test or sample input.

Terms may even be team-specific (or evolve so over time) by various reasons: access to sensitive code may be restricted, the software company grows and splits up teams, topics of teams are not interrelated and there are no intersection points for shared code/guidelines/rules, …

The worst case is when the scope gets so narrow that, earnestly, "it’s all in the code" (only). Below, I will list some recommendations for choosing names, and related tips, to not get to this point (I call it "people lock-in", as in "vendor lock-in").

The aforementioned "local scope" is one context for a term. If one word can be understood in different ways, it is contextually ambiguous. You don’t want to collect many such terms in your code base, or else switching to another project will leave you confused why accountName suddenly means "human description of bank account" for at another point it meant "username of login credentials".

Likewise, even if terms have a unique meaning across the code base, they may be represented as different types and therefore again cause an inconsistent understanding. For example, let’s assume that "account" is always a bank account, but once there’s a variable AccountInfo account and in another spot int64_t account. "The latter is quite obviously the account ID in the database!" — oh no sorry, Mr. Original Code Author, it’s not obvious!

Another popular piece of information stored in data structures is string address (compatible with all international addresses! 🌍) being represented differently elsewhere: struct Address { string address1; string houseNo; […​] }. These are even incompatible in conversion.

The key to overcome ambiguities is to stay consistent in naming. In the majority of cases, the main term (here: "account"), which by itself is not explanatory, can be suffixed to give a meaningful variable name: int64_t accountId and AccountInfo accountInfo (still nicely readable if type is omitted: const auto accountInfo = […​];).

While validateIban very obviously validates IBANs, knowing only the function name doesn’t say anything about how the function works. It requires at least the function signature and possibly a documentation comment to grasp the semantics. A company or development team may have their concept of how all validateXYZ functions should work, e.g. throw specific exception or return false on invalid value, and even if that concept is "well-known", it’s a notion that must be transferred to new hires. Such an induction to the company’s development practices is of course necessary for new developers, but too much will overload those people, resulting in small details being forgotten. Let’s say you forgot what the function validateIban returns for an empty input string? It’s a very important detail, and the most sane way would be to consider an empty value as invalid, because then the caller can decide whether an empty/optional value is allowed depending on use case. Yet this detail is not found in the name (granted, it’s hard in this case without getting wildly over-verbose names).

Here are a few alternative function signatures (C++):

The first influences I want to summarize here are complexity and (spoken) language. International mixes of development team members can be found in almost all companies. The language and culture barrier and gap are the most influential topics to be aware of when it comes to creating a common and mutual understanding of technical and personal themes. Hence it’s no wonder that the English language reigns software development both in coded and spoken words. To account for culture differences, complex English vocabulary should be banned where reading or listening comprehension is important.

To give an example, I want to name the Unicode standard. After reading 20+ articles (incl. the famous Joel Spolsky) about Unicode in the last 15 years, and continuously learning about its updates, its terminology still is only partially burned into my brain. The sheer count of terms is high, but in my opinion not the issue, since the memory of a software developer is quite durable once a term is clear. Can you distinguish UTF-8, UCS-2, UCS-4, UTF-16{BE,LE}, UTF-32, (UTF-7), character, glyph, character set, code point, surrogate pair, BMP, BOM, U+1F4A9? I have no problem recalling their meaning when I see them, but what really makes my brain smoke are the non-technical things mentioned in that list: is a glyph a fully rendered code point, or a partial symbol? How did they define character again — was it the same as a code point? Just look for a minute at their glossary and you’re going to be overwhelmed as well. In summary, the standard, the related myriad of blog posts, true/half-assed/false answers on StackOverflow and other resources are simply an overload for the software industry and simplifying now takes a huge amount of effort. If we were to use UTF-8 everywhere (great simplified glossary there!) already 20 years ago, there probably wouldn’t be crazy inventions like MySQL’s UTF-8 variants (yes, your UTF-8 enabled database probably cannot store all of Unicode!):

See, complexity and amount of terminology is like a growing company — the smart ones can handle growth easily by keeping things simple and stupid, while the typical response to growth is levels of management, performance reviews, more business, less "family" feeling, or in other words: complexity.

Imagine you’re in one well-defined context, have chosen simple English words that need no explanation in your opinion, developers you ask tell you they understand the meaning immediately — what could possible go wrong? You’ve landed a set of terms to be carved in stone. They will call a dictionary after you! Well, probably not… In reality this is long before the finish line.

One area for which I really have a strong opinion are filesystem terms. Those are around since ages but still confused and forcefully en-ambiguated (my opposite of disambiguated) or highly confused in code all the time, to the point where it’s not funny anymore. The problem is that even if the words are clear, and you were given Tanenbaum’s book on operating systems in studies class, the terms are still way too interchangeable. Find below some examples of ambiguous wording, including my proposals and what people also use as alternatives. I’m using lowerCamelCase examples here to also nitpick about spelling differences. This was the motivation to start writing this blog post, so sorry about the lengthy commentary! I’d like to hear comments on this admittedly very opinionated section:

We want to ensure that full PANs cannot be converted directly to masked PANs (among other reasonable restrictions). Look at a beginner attempt to define separate types:

Well, that compiled just fine and led to a fatal bug — we just logged a full PAN to stdout. That action shouldn’t have been possible. You don’t want to sit through and pay for those two extra weeks in the next credit card audit, not to mention the cleanup to get the sensitive data out of the way!

Using an enum wrapper is also ugly, and not really readable like English prose — so probably not a good idea in general.

The C++ standard gives a simple explanation:

Or in other words, typedef A B; seems to be no different in this use case from using B = A; — if there’s a difference at all?! Fortunately right in that quotation we have a proposed solution: declare a new type with struct/class/enum.

While I wouldn’t recommend using an enum for scenario, it apparently has its strong typing benefits:

The std::string case actually is a no-brainer: strings are ubiquitous in business logic of most companies. They are used for

We have to create a new struct or class to have disjoint string-based data types.

Summarizing this solution (one of many):

Strong typing wrappers are not — and will presumably never be — an inherent part of C++. Instead, the above solutions proved simple enough for the mentioned use cases. It’s on developers to decide whether they write a few lines of wrapper code to be very explicit, or choose a library which does the same thing.

Rust has the same notion as C++'s aliasing typedef:

which has the same problems, so no need to repeat that topic.

Syntactically, Rust provides a very lightweight way of creating new types in order to achieve strong typing — tuple structs:

Well, that didn’t help much… Admittedly, tuple structs are more helpful for other use cases, such as struct Point(f32, f32) where it’s clear that X and Y coordinates are meant. A rule of thumb is: if you have to give the tuple fields a name to understand them, or you require input validation at construction time, don’t use a tuple struct. Remember that Rust uses an error model that is different from throwing exceptions, and in the above example there’s not even a constructor involved that could return an error (or panic) on invalid input.

Let’s replicate what we did in C++:

This questions seems to be mostly language-independent, and a matter of taste to some extent. In my experience, there are ups and downs:

Yay:

Nay:

In the end, you must decide per case. Often times, the declaration of functions or types allows for human errors, so before changing to strong typing, you should first consider if the order of parameters, name of fields, choice of constructor(s), et cetera are sane, consistent in their meaning (money amount shouldn’t be 123 cents in one place, but decimal number string 1.23 elsewhere) and follow the principle of least surprise and smallest risk of mistakes.

There’s also no clear winner between the languages — since strong typing is not a built-in feature in either language, you must roll your own or use a library, and that isn’t exactly elegant, but still readable.

Since the Android project will later go in a subdirectory, let me first explain the directory structure that our application will have:

The AndroidSencha directory contains the app scaffolding created by Sencha Touch, i.e. app (models, stores, views, controllers), resources (CSS, images), sdk (necessary Sencha Touch SDK files), .senchasdk (points to the SDK), app.js, app.json, cordova-x.y.z.js and index.html. The android folder will be created manually and contains our Eclipse project. And sencha_wrapper.py is my wrapper script for the sencha command that will be explained later.

If this does not make sense to you, check out the finished application at Github or just bear with me in the rest of the article.

The command should now copy/create some files and directories.

Now it’s time to do some bootstrapping for the Android part. In the newly created application directory (the one that contains app.js), create a folder named android and create a new Android project there using Eclipse:

Note: I use Android 2.3.3 (SDK version 10) as build target and the package name org.dyndns.andidogs.androidsencha. You might run into the error "Can’t find variable: Ext" or similar if you use the 2.2 or older emulator (see here).

PhoneGap is set up almost as usual with Android projects, following the official guide, but some steps differ a bit. In the android folder:

We have a main app directory for the Sencha Touch application and a subdirectory for Android. You should now check if the Sencha Touch application actually works. By default, it should contain a tab bar with two different views. Fire up a web server in that main directory and open it up in a browser (should be a Webkit browser, not Firefox). With Python, it’s as simple as:

Note that it might take some time to load (especially with the single threaded SimpleHTTPServer of Python). It should look something like this:

Important: The fact that it works in the browser does not mean it works on a mobile device/emulator. It took me a while to find out that you have to change "logger": "no" in app.json to "logger": "false". Else you will get an error like "Error: [Ext.Loader] Failed loading 'file:///android_asset/www/sdk/src/log/Logger.js', please verify that the file exists at file:///android_asset/www/sdk/sencha-touch.js:7908".

First of all, add the PhoneGap script as dependency, so copy cordova-x.y.z.js to the main folder and change app.json to include it – therefore you only have to add it to the key "js".

Sencha Touch comes with the sencha command line utility that can build an application, i.e. scan its dependencies, concatenate necessary files into a final app.js, copy resources etc. What we want to accomplish is to put that build output into the Android app’s /assets/www folder. And that’s why I said, don’t edit any files there because they will get overwritten.

I am using a simple builder configuration in Eclipse to run this command (will be explained below). Unfortunately, it seems that Eclipse does not stop the build if the sencha command returns an error – the sencha command actually always returns 0, but even if I wrap it with a script, Eclipse does not stop the build on non-zero return codes. Also, when set up as builder, Eclipse hides the sencha command output after completion (don’t know why?). That is a problem, because if you have a syntax error or other mistake in your Sencha Touch app, then you will see only the loading indicator and some unhelpful "[Ext.Loader] Failed loading…​" error in LogCat once you try and start the app. I work around this problem with a dirty hack wrapper for the sencha command. In the main folder (the one with app.js), add the file sencha_wrapper.py with the following content:

This script runs the sencha command with the passed arguments and checks if the string "[ERROR]" occurs in the output (by the way, they also don’t use stderr as they should) and if so, writes these errors to the end of AndroidManifest.xml and thus stops the Android build process because that XML file is no longer valid. As I said, a dirty hack. These lines are automatically removed once you correct the Sencha Touch app mistakes and run the script again.

So let’s try that out. From the main folder, run python sencha_wrapper.py app build testing -d android/assets/www. If successful, it will show no errors and end with "Embedded microloader into index.html". Now go to Eclipse, refresh the project (select project name and hit F5) and then run it. The app should work without problems (only the emulator is slow as hell):

Great progress! But of course you don’t want to use the command line and refresh manually every time you want to run your app, so let’s automate this. In Eclipse, right click the project, select "Properties" and then "Builders", "New…​" and "Program". Configure it as follows (your Python path will vary):

Click "OK" and use the "Up" button to move that builder to the beginning of the list:

Ensure that AndroidManifest.xml is refreshed after the script is run:

As a bonus, you can set up another builder called "Force recompile" that ensures that every time you click the run button in Eclipse, the Sencha Touch app is recompiled and Eclipse rebuilds the Android app instead of bringing the current intent to the foreground as it would normally do if it doesn’t recognize any changes (which it can’t because Sencha Touch app changes are in the parent directory!). The builder is configured as follows, note that you will need touch.exe (equivalent for the touch command of Linux, for Windows you can use the one from msysgit or the gnuwin32 coreutils package):

Make sure it’s located before "Android Package Builder":

That’s it, now when you click the run button in Eclipse, the app should always be recompiled from the current Sencha Touch source code in the main folder.

Open app/view/Main.js and replace the Getting Started content as follows:

Run the application and when you click the link, you should get a native alert box: