Andreas Sommer – software and health engineer – Blog

Connecting Intermittent Explosive Disorder with diet and histamine intolerance: no more uncontrollable anger and rage

Thu, 24 Jul 2025 00:00:00 +0000

Intermittent Explosive Disorder (IED) is a disease defined by uncontrollable anger and rage, racing thoughts and other exaggerated, abnormal behavior.

Science and medical information only provides symptom descriptions for this disease, and sufferers have basically no chance to solve it. Only psychiatric medication is well-known to improve it to some extent but can have strong side effects and does not solve the cause.

IED destroys families, relationships and probably even lifes. It’s considered a rare condition, but with an estimated lifetime prevalence of 3-5%, it seems not rare at all (for comparison: ADHD is estimated at 5-10% and considered a widespread syndrome nowadays). Guess the impact that a permanent solution could have, even if it only works for a subset of affected people. I found an easily-explainable solution that worked for me ‒ mostly the dietary change of leaving out histamine-rich food items.

My story

…won’t be detailed very much here. It doesn’t matter for resolving the illness, and I want my privacy after all these years of fighting and not knowing what’s going on. The goal of this article is to help others. There are many stories online to read and watch, even scientific case reports, in case you’re unfamiliar with how the disease can present behavior-wise:

Videos about someone’s experience with IED incl. self-harm, pet harm, hitting walls and other people, strong anger outbursts. Got better with medication. Disappointments, loud noises or getting touched mentioned as main triggers. Relation to autistic traits.
Description from someone about her boyfriend and what she did to help. On the other side, most IED stories online can rather be summarized with recommendations like "leave that person". This disease easily destroys family ties if untreated.
As already mentioned in the above links, there are hints about anger symptoms relating to ADHS and autism. For instance, try searching for "anger" on r/ADHD and lots of posts will pop up.
Case reports of Mayo Clinic patients (dated 1987), mostly clinical severity
One study correlates road rage incidents, aggressive criminal acts and other actions with IED (full text not available)
Not about IED, but this r/Anger post reveals various scary stories of what people do while their brain is in anger mode

Another health topic on a software engineering blog? Yes, normally such a switch of topics is unexpected because those computer freaks only write about their stuff. Unless health problems take over your life, you develop an analysis tool specifically to figure it all out, you become a hobby scientist and expert for your own health, ultimately find real solutions, and want to share it for the benefit of everyone out there. My analysis software, yet unpublished, again had an important stake in studying this problem, and below I’ll prove once more how important data analysis and correlations are.

Development of the disease

Intermittent Explosive Disorder didn’t really occur when I was younger. A few signs here and there when I got unnerved for small things, but overall it either didn’t "break out" yet, or wasn’t noticeable. My father had anger and rage outbursts ever since I can remember, and IED is said to possibly be genetic. I didn’t want to be like him.

I got clearer symptoms only at adult age. It’s impossible to say when this really started. Some problems at work annoyed me, particularly if others did stuff without taking care, resulting in me fixing the issues, for example, and I’d sometimes rant about it. This could however be explained as the perfectionism of a young engineer, or by the environment the company gave me. I left the somewhat-toxic environment at my own choice, switched companies and was good for a long time, but still, the same triggers could annoy me. Stupid drivers on the road would also upset me. I can’t say at all when road rage started for me, but I had it to a non-extreme extent, meaning I wouldn’t honk and shout at everyone, like some other people do. It mostly stayed inside my car. And then came family life and failing partnerships, creating strong social triggers for this illness. Yet, all my anger usually had a good explanation: stupid or ignorant people around me, bad relationships, a crying baby, inconsiderate drivers, clumsily forgetting where I had put my belongings in the apartment, things falling down or just "everything goes wrong today" kind of days, having to repeat myself when talking to others, and so on. I kept telling myself that it’s those triggers which caused anger, and didn’t think much of it.

There’s an important distinction to make here: IED presents as uncontrollable anger and rage. Probably every healthy person gets angry about something many times in their life, maybe even many times per week depending on the circumstances. The difference is that someone suffering from IED goes from trigger to rage in milliseconds and doesn’t have the capability to stop the bad behavior at the onset. Specifically, in your mind, you can’t choose an alternative behavior. There’s no time for it, and no other thoughts in your head at that time. You explode, shout, rage, misbehave, and the angry feeling in your gut ‒ sometimes felt literally ‒ might only go away after a few minutes or sometimes hours. I have no clue how and when the disease developed.

Detecting IED in yourself

You came to this article, so you already know or suspect that you or someone close to you has a problem. The first important part now is to know your triggers. I’ve listed a few of mine above. Also, you will likely have non-triggers, that is, bad situations which you can tolerate without getting angry. For me, I was mostly fine at work except for a few occasions, I never had problems among my friends, and my anger was rather in family situations and on the road. Triggers can vary a lot, but I guess the disease is the same.

Second, you need to find out if you have a normal level of anger, for example triggered solely by your surroundings, or rather anger that is uncontrollable, pathological anger, stemming from a mental illness that needs to be fixed in your body and appears even for the smallest issue. It’s super easy to always blame others, especially if you’re really behaving well with other people 80% of the time. But those 80% of time don’t matter if the remaining 20% are killed by your rage outbursts. Mind that it’s very much possible that other people in your environment treat you badly, making you angry, and you still can’t just state it’s only the other person. Try to deal with, and objectively look at, yourself, as you can’t change others. If your current relationship already got intoxicated by anger and stress, there’s always the chance in another one and sometimes things need a fresh start.

There seems to be no objective way to quickly determine if you have the disease. Nevertheless, read on to what my body’s problem and the solution was… you’ll see that anyone can try my dietary solution safely and thereby find out if bad behavior and outbursts turn back to healthy levels ‒ or go to complete remission (no more symptoms), as is the case for me.

Medication and optionally therapy can be a good first step

I had no idea that a mental illness, caused by biochemical reasons in my body, could even be the case. I only got that last-resort idea after many years of struggling, facing life consequences and trying a myriad of interventions to fix this syndrome, all without any improvements. With that idea, the logical next steps to rescue myself were to call the crisis hotline, confirm the idea of getting medical treatment, getting an emergency appointment and a first prescription for an antidepressive from my general practitioner doctor, and then following up with a psychiatrist about the medication. (You can’t imagine the effort and pain it took me to get to that simple-sounding conclusion which finally got the ball rolling.)

The psychiatrist appointment, which in countries like mine usually only happens after several weeks or months on the waiting list even in case of emergency, was honestly useless because it only confirmed that the medication was a good first treatment choice and that the psychiatrist had no idea about IED whatsoever. In fact, IED only appeared in the ICD (International Statistical Classification of Diseases and Related Health Problems) around 2016, and was mainly documented in the USA before that, described for much longer in their DSM (Diagnostic and Statistical Manual of Mental Disorders) and case/medication reports dating back to 1987 and maybe even earlier. Doctor can’t do much about IED apart from prescribing antidepressives and a few other, off-label medication types. Psychiatry is a field that has almost no technical or medical options for root cause analyses, but I was still surprised that not even blood was drawn or any tests made when I was there. The medication I got from my general practitioner, chosen by guidelines, was quetiapine (Americans know it from the brand name Seroquel), a drug affecting neurotransmitters in the brain. My dosage was 2x25 mg a day, the minimal therapeutic dose. Another well-known type of medications that "work" for IED, according to online stories and case report, are SSRIs (selective serotonin reuptake inhibitors). The beta blocker propranolol is also documented to help in many cases.

My point here is: medication has been available for decades, why not try it? For me, trying the pills for a while proved that 1) they do work pretty quickly and 2) it’s in my body and not a psychological problem. That’s why all the talk therapy and other attempts failed in the long run. Therapy has a short-term positive effect that feels like starting with a new mindset, and therefore I’d recommend trying it in combination. It’s not for me though. Since the medication worked to a very good extent, and symptoms reappeared when trying to lower the dose or not taking a pill for a day, I knew that it might be possible to find the actual cause in my body. I think medication is a good start to quickly get rid of IED-triggered behavior. It may even work long-term if you don’t experience intolerable side effects. However, mind that those medications can have serious side effects and don’t treat a cause. Particularly medications affecting neurotransmitters like serotonin, dopamine and norepinephrine are well-described if you search online.

I had researched SSRIs before and was actually hoping to get one of them to try, as they are said to be highly effective within weeks, albeit sometimes with harsh side effects. It turned out that quetiapine was a similar drug I hadn’t read about before, and it worked within days, not weeks. Not only my irritability improved a lot, but also my motivation, insomnia and how good my days felt. Racing thoughts were completely gone. I was finally able to freely decide what my mind thinks. It felt like seeing snow for the first time in life. My thoughts were totally calm. If I did get a small intrusive thought, I could now just decide to stop thinking about that topic! This was suddenly the case in all life situations. In addition, and much more important, things wouldn’t annoy me so easily anymore, and if they did, I could decide within a few seconds whether I react to it and get annoyed. Previously, I had milliseconds and the reaction was like pre-programmed, not decideable. That was the "uncontrollable anger" part of IED ‒ also gone, thanks to medication.

Eventually though, I was able to get rid of the medication after about 4 months. At that time, I had not only, luckily, found the real cause, but also experienced severe and dangerous side effects. There had been a few rough and somewhat-acceptable, known effects such as sedation (which subsided quickly after experimenting with intake timing), insomnia, and exacerbated restless legs syndrome (RLS; I normally have this in summer) due to the influence on dopamine. But those last weeks before quitting, I couldn’t sleep at all anymore, was groggy and very dizzy all day, was ready to pause my gym membership because I couldn’t sustain any sports or much movement anymore, and eventually considered visiting a hospital. Particularly my lower body was physically tired in a way that I’ve never sensed before. I’d like to sensitize you about such risks. Almost every medication with strong main effect can also have strong side effects. Every doctor knows that, normally. I’m especially considerate of anything that crosses the blood-brain barrier. These issues went away after tapering off the medication ‒ which by the way should never be done at once ("cold turkey") because of the corresponding risks.

Measuring symptoms to find the cause

In autumn 2024, I finally picked up the term "Intermittent Explosive Disorder" again for my research even though I remembered that other than some symptom descriptions, there was nothing interesting to find online when I heard of the term first around 2022. This led me to finally write down symptoms in a structured way, long-term, every day. It’s not easy to find a metric that can be objectively measured and consequently written down each day without guessing (was my anger a 7 or a 9 out of 10??). The same applies to other symptoms, though, and fortunately I had years of experience writing down health symptoms from my migraine research (I fixed migraine to a good extent, see my article).

The first metric I started to write down daily is "inner anger" on a scale of 0-3 where 1 would be annoyed feeling, 2 stronger anger, and 3 a freakout, even if it remained only in my head.

And the second one is "nice to person XYZ" on a scale of 1-5, with both the day’s minimum (worst behavior) and maximum (best behavior) written down, so actually two metrics. I did it for multiple close people, so I had even more data. On that scale, 1 stands for screaming, wild, unfair, outrageous behavior, 2 for very unfriendly, 3 for neutral/affirmative, 4 for friendly/nice and 5 for loving/caring/understanding behavior.

Together, this means I have one internal metric that was "measurable" even if I didn’t shout or otherwise react to other people, and two metrics that define my externalizing behavior, or in other words, how bad IED actually affected my behavior towards others, not just my thoughts. It was so incredibly stupid to start the data collection so late, as I knew that the data is the most important input for finding correlations with other conditions.

What could I now do by just having some numbers of how angry and outrageous I was? Standalone, those numbers are useless. They must be correlated with, at best, possible causes. I had lots of conditions already written down for a while due to my migraine research. For example whether I was outside in the sun, did sports, took supplements and ‒ sadly only since that autumn ‒ also what I ate on each day. After a few months, I still had no clue and also didn’t find any success stories online. I tested and analyzed supplements which are supposed to help (e.g. magnesium is said to be calming), among other attempts. I spare you the details.

Histamine intolerance (HIT) is the cause, omitting histamine-rich food items the solution

I had used magnesium for years to try and combat migraine. It’s recommended for a myriad of health conditions and said to have a calming effect, improve sleep and headache, yada yada. It’s the holy grail supplement nowadays. Different magnesium salts have different effects and side effects, but for me, magnesium had exactly one outcome:

Magnesium vs. anger (since writing down data, until roughly the time I figured out this negative effect, ~100 days of data)

Remember my scale of being nice and friendly to "person XYZ"? It’s the main behavioral problem I wanted to improve and magnesium supplementation lowered my niceness score by 0.8 points! That’s vastly significant. (For simplicity, I combined all magnesium types and dosages into one bar in the graph since there was no difference for me.) For the "inner anger" metric, worsening can be seen as well, but it’s not very obvious, so I’m happy to have chosen an externalizing behavior metric as well. The more often I took magnesium, the longer the very bad effect lasted.

This was the first hint of effects from diet and supplements. I also tried the ketogenic diet which is effective for the more well-known mental health disorders (depression, bipolar disorder, anxiety, schizophrenia, epilepsy, etc.). In short: it didn’t help for my IED (and I might have done it wrong by miscalculating the fat ratio). After starting to write down total daily fat, protein, carbohydrates and calories to help get used to keto, I extended that at some point to write down the foods I ate. For example, my spreadsheet has one row for each meal (breakfast 1/2/3, lunch 1/2/3, dinner 1/2/3) and I’d put the cooking ingredients in there, such as "pizza with flour, tomato sauce, champignons, salami, grated gouda cheese".

After several months of recording food items, I found the first food which strongly affected my anger: my beloved peanuts. Then other nuts. Actually, almost all nuts (not all are displayed below). Shit.

Without/with certain types of nuts vs. anger (74 days of data)

The bars for "I ate cashews/almonds" are shown transparent because they have less than 10 days of data each, but the effect is clear: these and even more types of nuts turned out as very bad triggers for my anger. Statistically, one of course has to exclude cofactors like combinations with other foods. I did that. The data didn’t lie. Ripping those favorite snacks out of my diet already helped tremendously.

I still wasn’t sure what the common denominator was, though. That turned out to be high histamine content. My "bad list" of foods grew to over 10 items after a few more weeks of research. It was all easy after getting the initial idea and the written-down data available for every day. It’s just statistical analysis and in the years before, I had already written the tool which produces the above graphs from my spreadsheet of health data (the tool will be released in a few weeks or months, I hope).

As of June 2025, my known incompatible foods were:

avocado
banana
blueberries
chocolate
coconut milk
coffee (incl. decaf)
cream
eggs
green salad
joghurt
kiwi
nuts (most probably all of them, but some I eat so rarely that I don’t have enough data; for example brazil nuts which are anyway to be eaten in very small amounts)
onions
spinache
tea (from the actual tea plant i.e. black/green/white tea, not herbal/fruity water)
vinegar and fermented foods (sauerkraut diarrhea, ewwww never again)

There are very few stories on the internet saying that histamine can have an effect on your mental state and behavior, but there are. And here’s mine. The data proves it. I created a metric in my statistical analysis which tracks how many of those known-incompatible foods I ate on a day, and in retrospective, the number of portions perfectly, linearly (!) correlate with anger symptoms:

Incompatible foods vs. anger

It can’t be more obvious than those graphs: all anger metrics are worse the more high-histamine foods I eat. It’s not gluten, not lactose, not some minor, usually destroyed-by-cooking toxin like oxalates ‒ it’s histamine! Histamine is in many foods, can’t be destroyed or dissolved by cooking and there are great lists available online that show which foods can have a high content.

Mind that not everyone has the same trigger foods. Tomatoes are listed on all web pages that discuss histamine intolerance, but for me, they’re actually very positive. Maybe that’s just a gift so I have at least one favorite food left that I can eat? Meat is good for me as well, luckily, because otherwise I wouldn’t know anymore what to eat to get satiated.

The above data had already convinced me, but there were more signs that hinted at histamine being the actual problem:

Since magnesium was extremely bad for me, which I still can’t understand today, I tried its antagonist calcium. And indeed, calcium is linearly positive regarding anger symptoms! Calcium even seems to work independently from leaving out my trigger foods. I guess that it’s because calcium is a potent antihistamine. It was used as such in the early 20th century. The effect was very easy to observe: if I have a stuffed nose and take calcium, the nose is mostly free within 20 minutes, and the effect could last up to 3 hours. Surely there might also be a risk to high calcium supplementation, so I rather consider diet the main solution for IED, and will be very considerate about using calcium at all.
The more incompatible foods I ate, the more my nose is stuffed. This relationship is also perfectly linear in my data. So the surrounding topic to research here is allergies. Not just food allergies, but also pollen and other triggers. I currently believe that heat is also a trigger. Start reading about "histamine intolerance" (HIT), "mast cell activation syndrome" (MCAS) and related topics, and you’ll get into a huge world of people with extreme health trouble, with various suggested solutions.

So in summary, leaving out food items and taking calcium did the trick. I’m still in full remission, no anger symptoms, after several months on this restrictive diet.

If you ask now: what if the medication has fixed the problem in the long term, and not the dietary change? No, the data and my observations were absolutely clear here. Going lower than the minimal dosage of quetiapine brought back anger symptoms latest after 36 hours. Its effect only lasts while the level remains roughly constant, which is why you need to take two pills each day. While the medication reduced symptoms to a good extent, leaving out foods had a much stronger impact, namely complete remission of symptoms. I don’t consider the medication a solution, or find it in any way comparable to the exclusion diet.

Also, I use the antihistamine cetirizine every now and then against my pollen allergy on days with a runny nose. That didn’t have an effect on the anger symptoms. It’s not surprising: antihistamine medications block certain histamine receptors, and usually not all of them; they don’t influence accumulation of histamine.

This is the end of the informative part of the article. Below you’ll find another optional section with my next steps to try and find the root cause for my body’s overreaction to, or overaccumulation of, histamine. I don’t want to miss out on great foods and drinks forever. No facts or findings in there, yet, but maybe there will be an update later. Good luck to you!

What comes after the anger is solved?

How to ensure the symptoms don’t come back? I don’t really believe in histamine intolerance as standalone problem or it being the root cause. I reckon it’s caused by something else. And I don’t want to skip eating all those great food items forever. I don’t have a clear answer yet, but the research about HIT led to SIBO (small intestinal bacterial overgrowth) as a possible cause. Since SIBO is quite widespread, the possibility for me to have it seemed high, so I did a test. SIBO is a common underlying problem of IBS (irritable bowel syndrome), but I don’t have any digestive or abdominal problems, or frequent flatulence from certain foods. Why do a test then if I don’t have the typical symptoms? Well, let’s compare that to my histamine intolerance: it’s absolutely rare to get mental symptoms from too much histamine. Most people simply get allergy symptoms like skin rashes, runny nose, itchy eyes and so on. I had regular allergy symptoms as well since I’m allergic to trees and grasses. This still doesn’t mean you can’t have other symptoms. Just because something is rare and almost completely undocumented doesn’t mean it cannot happen. I’m ready to do anything to fix my health issues long term. It’s like in software engineering: if you do it the right way, the problem doesn’t come back to you later. This explains why I wanted to test for SIBO despite not having the well-known symptoms like bloating, diarrhea and digestive issues.

The test cost around 100€ incl. return shipment of the glass vials, and debrief phone call about the results. Since doctors and insurances here don’t know or cover SIBO, I paid and did all this myself. There are several labs offering tests, even in other countries. The results turned out strongly positive for hydrogen gas production. In the article The Surprising Connection Between Histamine Intolerance and SIBO, among other sources, I found a reasonable explanation: overgrowth of bacteria in the small intestine, where naturally only a small amount of bacteria should arrive and settle, can damage the gut lining. And that in turn can lead to bad degradation of histamine by the body. There are enzymes involved in the degradation process, as you’ll read in the first 5 minutes about histamine intolerance, yet supplementing the DAO enzyme didn’t help in my case. Anyway, due to this finding, my current attempt is to fix SIBO and thereby maybe the root cause of histamine intolerance. The problem is that I don’t have anger symptoms anymore, so it’s impossible to relate the gut healing with improving symptoms. Yet I still have insomnia as very major health issue, and if my craziest theories prove to be true, then histamine keeps me from falling sleep at night, treating SIBO will fix HIT, and I’ll be happy in life 🤔. Wish me luck, since this isn’t much based on data or other people’s experiences anymore.

I’m following the recommended antimicrobial treatment: oregano oil, berberine, then probiotics, and finally gut restoration with certain supplements. I’ll update this article if it improves anything. Again: I can’t tell if anger symptoms improve since I’m in full remission already. Plus I want to reintroduce avocado, chocolate, eggs, among other items, and not fear food any longer. The exclusion diet is said to be required only temporarily for histamine intolerance, and not required long-term for everyone.

How I mostly fixed my migraine, weather and sports-induced headaches

Wed, 28 May 2025 00:00:00 +0000

After 37 years of migraine, I found a scientifically well-explained solution. It’s hydration with just table salt as electrolyte. And a few additional habits to turn off the remaining headache that the salt couldn’t always prevent.

In this article, I want to explain the background, and since it’s a supplementary intervention without much research behind it, my data about safety and long-term efficacy after trying several variants of salt supplementation.

My last blog post was 3 years ago, as I was spending lots of private time solving health issues. This is still a software engineering blog, and software helped me figure this all out.

Migraine started in childhood

It all started with the early signs of developing a "migraine brain": many nights of my childhood, I had to wake up several times to vomit. Combined with a terrible headache, of course. In adolescent years, I would often come home from school, just wanting to lie down in a dark room until the headache pain, light sensitivity and sometimes sound sensitivity ended the next morning. Surely, I missed out on lots of fun in my childhood. I don’t remember when the puking part ended, but the rest never did… until I found the solution which drastically helped within a matter of weeks.

Doctors couldn’t do or understand much about migraine back then. At the age of 18 or so, I still had no remedy at all. I remember a general doctor telling me: "Strange, aspirine should work after 30 minutes". Of course, aspirine can be effective in some common headache types like tension headache, but barely for migraine headaches. Why would nobody ever tell me? I still tried that medication all the time to no avail. The term "medication overuse headache" only became well-known later. At that age, I already had some theories what’s triggering the illness for me, such as weather and air pressure changes, but my pure observations wouldn’t help because I couldn’t change the weather.

My main symptoms are a headache developing over several hours, then in case of very strong migraine also a neck strain spreading usually on one side of my head. Once developed, I wouldn’t get rid of it until waking up the next morning. Luckily, I have never experienced cluster headaches or migraine lasting several days. A visual aura happened only once, and scared me. Migraine can have many prodromes (early sign of migraine onset), symptoms with and without pain, and is therefore studied as multifaceted, allegedly hard-to-fix sickness.

For a while in my youth, I subjectively believed that drinking coke works against headaches. Even in later years, I tried a few caffeinated and other drinks in this faith. My parents and me never collected any data. It could have proven that the sugary drink didn’t help at all. I was too young and naive to get this idea, and no doctor recommended a headache journal either. All I got was two dead teeth and several root canal surgeries from all those drinks. They were available because my parents bought them all the time. Moms and dads, stop this crap!

Science principles ‒ why even write about this single-person case?

Doctors had no idea of migraines back then. My parents also had the average health knowledge for that era: none. And who’s to blame them ‒ the 1990’s were still a time in which you had no other chance but to trust experts: doctors and unfortunately also newspaper and magazine articles. The internet, open science and wide-spread, free health information by governments or even influencers all happened much later. And despite the existence of these great sources of information, many people, not just our parents' generation, still don’t make use of them. So as I solved my migraine by a convincing extent, I also set out to tell others about the story, and clearly document my case on this public, free website. I believe that free information is a cornerstone of modern science. There are too many commercial interests and corruption conflicting with science and its subsidization. The solution is to simply override scientific malpractice by single people’s experience-based, anecdotal, and in my case data-backed, solutions. That’s my personal rationale behind this article and also behind the things I’ll write about next: free information gives people the power to fix their health.

Obviously, my solution may not work for everyone else. But you’ll read below that salt supplementation wasn’t my own idea. A whole Facebook group is dedicated to a related protocol and has helped an unknown number of people. There are forums and groups for many illnesses, which is a great way to discuss your specific symptoms. These groups are usually structured by questions and comment threads and, only if you’re lucky, clear instructions on what to try against the respective illness. They are however not an easily, publicly accessible and searchable way to find information about migraines. I’m therefore trying to put together the major parts in a concise way, even if I only have data and proofs for my single case.

Failed attempts

French fries: not a healthy, permanent solution

By the age of 25, I hadn’t tried much apart from aspirine and triptans, in the hopes that some unproven "blood pressure in the brain vessels" theory plays out somehow. No medication really worked, and regularly buying a pack of 2 (stupid package size!) triptan pills at a horrendous price therefore wasn’t justified. Instead, I toughened up, so-to-say, not letting migraine days ruin my activities. They of course still did, because you can’t fully focus on work or hobbies, nor do sports with a full-blown migraine. The acceptance however worked somehow on the psychological level to weaken its position as a miserable part of my life.

In 2021, four years ago, me now aged 34, I somehow got the idea of writing down daily symptoms in a spreadsheet, as an attempt to get the headache and other symptoms solved. Not long after, my mouth ulcers were gone after trying out several interventions (hint: use SLS-free toothpaste). So I considered data as a great way to correlate symptoms, interventions and with that, potentially solve health issues. I continued, switched from paper spreadsheets to a huge Google spreadsheet, tried out more and more supplements and stuff in order to solve my main concern at the time ‒ the migraine headache and accompanying neck pain.

My strongest migraine attacks would usually come with strong neck pain, slowly spreading into my head. Many migraineurs have prodromes as first symptoms, and the headache appearing as last, most severe symptom. Prodromes let you predict that you get a headache, so the typical advice "know your triggers" should be extended to "know your triggers and prodromes". The nausea and light sensitivity from my youth are also well-known prodromes. "Feeling" weather changes is another one. I’d know in most cases if a strong headache develops over the next hours, but simply didn’t know how to prevent it.

Example spreadsheet with health data (simplified, contrived)

Before having tried any interventions, when I started to write down how strong my headache was each day, I had a headache (of any severity) every third day, 120 out of 365 days! Life sucks, eh? Many of those were also full-blown migraine attacks. The way I wrote them down in my log was easy: 0=no headache, 1=weak headache, 2=medium and already quite painful, 3=ruining my day. With this data, analyzing the monthly/yearly count of headaches, and their average severity, was the next logical step. As a software developer, I wrote my own small program to analyze my spreadsheet data. That gave answers about whether I have more or less headache if I drink apple juice, take supplements, do sports, had alcohol, etc.

I also got enthusiastic about reading scientific studies around the topic. Which remedies exist? What supplements and dietary interventions could I try? I became focused on trying vitamin D, magnesium, eating enough food because skipping meals is a trigger for me, etc. And indeed, the year 2022 turned out to have 22 headache days less than the year before. Or in words: from every third day to roughly every fourth day! Quite a success, I thought, not yet knowing that long-term health issues can be solved much better.

In 2023, I tried many more supplements from scientific literature and ideas from the internet. For migraine, you quickly find a plethora of things to try. Apart from r/migraine’s top recommendation of salty McDonald’s fries, you’ll find quite some substantiated evidence in favor of several supplements, for instance coenzyme Q10, vitamin B2 (riboflavin), vitamin D or magnesium. And of course modern medication such as triptans or monoclonal antibody medications like Erenumab. Food triggers are also listed for a small percentage of migraineurs, but I hadn’t considered that at the time. Of those data-backed suggestions from science, and countless comments from other people on Reddit, I picked the ones which made sense to me. Since either or both of Vitamin D and magnesium seemed to help, I was eager to find other supplementary or lifestyle interventions to help my pain. Regular sports was one of the things I kept evaluating. I did neck stretches, neck strengthening with weight training, avoiding the valsalva maneuver (breath-holding) during training, different magnesium supplements and higher dosage, caffeine and electrolyte drinks, other supplements. My improvements from the previous year remained, likely because I was still using vitamin D and magnesium, but boy did this take a vengeance… higher doses of vitamin D (2000-5000 IU/day), taken over a long time, left me with permanent, severe dizziness over several months, useless neurologist appointments, their new distracting theories of "vestibular migraine", and so on. By looking into my spreadsheet, I could luckily see that I had raised the dosage before this occurred, and fully tapered off vitamin D to get rid of the dizziness. The body buffers vitamin D, so even after taking a small daily dose, I kept seeing symptoms. If you check online information, vitamin D is hyped as the world savior and high doses often mentioned as safe. That’s simply not true for everyone! This definitely made me trust N>1 science (studies on many subjects) less in the future, now equally taking single case studies and experience reports ("N=1 science") more into account. Reddit in particular turned out as great source for real people reporting about their health problems and solutions. Unfortunately, the average internet user doesn’t really detail their story at all, and health still remains individual. Anyway, internet answers can be great hints of possible illness causes that you didn’t consider before.

My research on migraine-related neck pain also misled me: my neck training sessions turned out as potential trigger of migraine, mostly since I was lifting heavy weights (up to 15 kg / 33 lbs). DOMS in the neck is a one-of-a-kind experience that not everybody gets in their life, and it surely made me stronger, but it didn’t make the headache much better in the long run, and definitely worse on training days. Another wasted effort to follow this idea, as so many before. The below section on the scientific theory will explain why migraine doesn’t originate in the neck. I consider neck pain as secondary symptom or prodrome of my migraine.

But I insisted to follow the obvious observations: intense sports were still triggering my headache. A common term for this is exercise-induced headache and it can have various causes. After a lower back injury in the gym end of 2023, I reconsidered how I do my training. I wasn’t really following common advice, even though it was known to me. I was weak in my back, but luckily smart enough to use the safety bars. Therefore, I didn’t destroy my life but was able to recover the muscle which just shut off during squats. Upon resuming strength training, I now always warm up for 5 minutes on the treadmill. I switched to exercises on safer machines and did Andy Galpin’s "3 by 5" protocol (as advertised by Andrew Huberman), meaning 3-5 repetitions, sets, minutes of break, etc. On top, I started with real progressive overload training, increasing the weight whenever I was ready to do more than 5 reps of the previous weight. Writing down the weights in a spreadsheet was again a good solution. A digital spreadsheet makes it easy to change exercises, hide the ones currently not in my plan, and so on. Google Spreadsheet comes with a good mobile app, so that’s my preferred tool. With the warmup, breaks and safety changes, the average headache severity reduced even more. Hard to quantify how much, since just a few months later, beginning of 2024, I had finally found the best-yet fix ‒ the salt intervention. Overall, in 2023, I was down to 83 headache days, i.e. one every 4.4 days.

An electrolyte product I tried

Electrolytes were an often mentioned topic on the internet and in some studies about exercise-induced headache, so I researched it further. I tried apple juice soda and other rather natural sources which didn’t help, and later sports mixtures containing sodium, potassium and sometimes magnesium. Unfortunately, here comes a crux of analyzing preventative drugs or supplements: if you only take them when there’s a trigger (for me: sports) or you feel a prodrome (for me: noticing a weather change), you can’t really find out if it helps unless there’s a super-significant difference. Interventions must be tried in a preventative fashion over the course of at least some weeks. Headache frequency without electrolytes after sports was 20.1% in 2023, but with electrolytes it was worse at 36.8%. This proves nothing. In order to look at long-term effects, I also looked at "with/without supplement on at least X out of the last Y days". And indeed, if I took electrolytes after sports on 3 out of the last 7 days (this also means sports 3x a week), the risk of a headache (20.8% in 2023) would even be slightly less compared to not taking them (23.1%). Fine, those results wouldn’t make anyone jump and say that electrolytes are any effective. Such a small difference just isn’t significant.
However, I was lucky to also have analyzed only the highest headache severity separately ‒ and of those debilitating days, I had zero when taking electrolytes after sports in 3 out of the last 7 days. Getting rid of the worst days of my daily life was a big finding, end of 2022. This was the first time when I had a clear hint about electrolytes being involved. The finding was in a small time frame and only on sports days, though. It yet didn’t prevent migraine overall.

Regarding weather-induced headaches: the triggering weather pattern never became clear to me until I analyzed air pressure information of my location. What if it was air pressure change? That would turn out as true in my data: days with pressure increase had somewhat higher risk of headache. But it doesn’t matter, as attacks could happen on days with sunshine turning to rain, rain turning to sunshine, air pressure changing in any direction. Only the probability is different, but every day is a potential migraine day! I stopped writing down the useless weather information at some point. We can’t change the weather.

Those years of reading scientific papers, social media reports of real people trying interventions took a lot of my private time. It became a full-blown hobby. The average severity (my daily headache score between 0 and 3) went down from 0.57 in the year 2021, to only 0.38 in 2023. Significant, but not life-saving. However, I wouldn’t know how to do it differently. Doing nothing would have changed nothing at all. Was it all a failure? I don’t think so. Learning about a hundred failed attempts that don’t work taught me a lot about scientific methods, real-world statistics, making logical conclusions, following the obvious instead of miniscule and pseudoscientific hints, and that apple juice isn’t really a working electrolyte even if I had the feeling that it works. Subjective feelings all turned out as false or even negative, so writing down and objectively evaluating and graphing my statistics was a good course of action. As of 2023, my homegrown analysis app had to analyze many symptoms against many conditions and produced a huge table of correlation percentages that I starred at for hours to see significant effects of supplements and other attempts. This made no sense anymore. Each analysis took 10 minutes already as the program was poorly developed and slow, not meant for long-term usage. I set out to rewrite it in another programming language, add a web page to control what I want to analyze, and to visually show graphs for correlations of symptoms vs. interventions. I’m going to publish that application in the next months, and believe it’s a quite good biohacking and health optimization tool for people who are happy entering daily data into a spreadsheet.

Despite the failure to fix headache in those previous years, I consider these years a success. They turned me into a hobby scientist dedicated to my own health, and I became interested to use this for the good of other people, too, resulting in this blog article, for instance. The fix for migraine came in 2024, and later, in 2025, I would even be able to put a decades-long mental disease into remission with the help of health data collected in my ever-growing spreadsheet. But that’s for a later blog post, if I ever dare to share that story.

The solution for my migraine: salt at the right amount

The results in the years 2021-2024:

The graph doesn’t show the severity of the headache days, only the count. You can see the dip from 83 headache days in 2023 to only 32 headache days in 2024, the year in which I started supplementing salt. That’s only one headache every 11.4 days! The average severity, not shown here, reduced from 0.38 to 0.15 (as mentioned above, I wrote down severity as score between 0 and 3 where 0=no headache, 1=weak headache, 2=medium and already quite painful, 3=ruining my day). In 2024, I only had 7 days of full-blown migraine. Numbers, numbers… they are critical to health science, but don’t nicely translate to what they mean for my life. The successful change in 2024 allowed me to finally consider migraine fixed to an extent that doesn’t make me fear it anymore. I am so happy and thankful for that.

I can only imagine what this could mean for people who have a really strong migraine problem. I have almost never experienced multi-day migraine. Mine mostly resolved overnight, but some folks even have episodes lasting weeks! Whether this solution works for you or not, I think it’s well worth the little effort to try it for a few weeks.

As mentioned, the main change is to drink a tiny amount of salt each day. If the scientific explanation holds true, there’s a good chance that certain types of migraine may have a root cause related to sodium-potassium imbalance in nerve cells and therefore supplementing with a safe dose of table salt could work for many people. There are unfortunately many types of headache in the International Classification of Headache Disorders (ICHD) for which I cannot provide any insights. There’s however enough information on the internet to distinguish the vastly different "regular" tension headache from migraines, so I don’t explain that here. This article is about migraines, and headache as their most typical and most painful symptom. I’m not a doctor and not giving medical advice here, so treat and try all information at your own risk. Below, you’ll find a section about safety, for good reasons.

Science provides many theories and even more recommendations for migraine, with some of them making obvious sense (such as weather changes and exercise inducing headache), but no clear "cause X? use remedy Y!" findings, or explanations why there are so many different triggers. Most people with migraines are left with barely working medicine such as painkillers, aspirine, triptans or supplements (e.g. magnesium, CoQ10). With those, they get a few percent of improvement in headache days, as I experienced in my own story above, but no life-changing outcomes. And most importantly, those treatments aren’t preventative, but curative and often applied only on pain days, usually too late after the root cause has already developed in the body. Funnily, if you do statistics like me, you may get surprising results such as headache being worse on days when you took ‒ for example ‒ aspirine. That’s easily explained: you don’t take aspirine as preventative measure (I hope you don’t; ever heard of medication overuse headaches??!), but only when you feel pain already. So statistically, headache severity will very likely be higher on those days, making the statistic totally useless because the data can’t answer if aspirine worked for you. You need to try a preventative, at best cause-treating intervention over a longer period of time, and can probably ditch medication at some point.

This is the intervention that works for me now, long-term:

Drink around least 2.5 liters (85 oz) of plain water every day. I live in a country with clean, unchlorinated tap water, so I’m using that. See below for my data about headache vs. water intake ‒ I highly recommend finding your own optimum amount! On sports days, my body automatically demands more. I mostly don’t drink tea or coffee. I have an insufficient amount of data hinting that tea (black/green/white i.e. from tea plant, not herb/fruit tea) might be very positive for my headache, but such diuretic drinks make me pee very often, and both tea and coffee contain histamine which isn’t for me. Hence I’m skipping them for non-migraine reasons.
I add less than a quarter tea spoon of table salt (¼ tea spoon ≈ 1.25 g NaCl alias sodium chloride ≈ 0.5 g sodium) to every 1-liter bottle that I drink. After some days, it doesn’t taste salty at all anymore. I found that the body buffers sodium, so the absolute amount over several days is more important than weighing the exact mass.
Watch out for concerning symptoms, see the section "Safety of the salt intervention" below!

That’s why I reduced from ¼ tea spoon to basically just a pinch of salt per liter:

Less than a quarter table spoon suffices for me

At first, it took some weeks to notice much fewer days with headache or headache onset. I started the intervention in January 2024 and still got 7 headache days in that month. In March and April, I was down to only 2 headache days each month. Incredible! Since the effect doesn’t come from one day to another, it doesn’t feel like a miracle cure but I only realized after some weeks that something was strongly improved in my life. This wasn’t the end of the story though, so please read on about the safety research I did. And if you’re still interested by then, there’s a section at the end explaining what the theory behind this solution is, and how a large group of people may already have known, and benefited from this for over 10 years. It felt bad for me that the knowledge was out there on the internet all the time, me having no clue about it. My migraine could have been solved many years ago.

A common recommendation is to drink lots of water together with electrolytes, but my own data says that drinking too much is bad for my headaches:

Headache severity (0-3) vs. water intake in milliliters (data from 2024-06-01 to 2025-05-27; insufficient data for the [3500,4000) ml bucket)

For the probability of feeling any migraine prodrome on a day, the results are similar:

Prodrome probability (0 or 1) vs. water intake in milliliters (data from 2024-06-01 to 2025-05-27; insufficient data for the [3500,4000) ml bucket)

For me, 2000-2500 ml of water might be optimal, with more water making it worse. I can only guess that dilution could be the problem. Important: I only analyzed the same-day water intake. For example, if I analyze headache vs. yesterday’s water intake, the results are similar, but less pronounced. Also, I only consider salted water here and not which other electrolytes I did or didn’t take with it, such as magnesium. Significant linear-looking relationships in my data however often turned out as meaningful, so I go with roughly 2500 ml as my optimal amount, with more on sports days.

Intense sports or a full-day in a sauna and thermal spa can still evoke rather strong headaches for me. I haven’t figured this out fully and can’t claim a great remedy just yet. I’m wondering if electrolyte imbalance could explain those triggers as well.

Here are some additional recommendations:

Hydration means water + electrolytes. Electrolytes include sodium, potassium, magnesium and possibly other micronutrients if you’re deficient. It’s important to hydrate the day before sports or another intense event, and then also during that day. Pee color and times-on-toilet are said to be good hints about over-hydrated vs. under-hydrated status, but I didn’t research if that’s true. Sugar-free electrolyte supplements might work as well, but don’t overdo potassium supplementation since getting that from food is safer than drinking it all at once (article about potassium toxicity).
Warm up before sports. My data doesn’t show much benefit of warmup regarding migraine risk, but it reduces injury risk for sure. It’s a no-brainer. Overall, my gym sessions are now long, structured, and with breaks, rather than intense 30-minute workouts that make me totally tired for the rest of the day.
I take breaks during sports, so that my heart rate doesn’t go over a certain threshold (for me: 170 🤷). This idea isn’t data-backed; I only have a subjective clue that this could raise migraine risk on that day. I have a pretty high heart rate overall, typically at least 160 for any speed of running and also with other cardio sports. The more volume (~ repetitions multiplied by weight), the higher the heart rate spikes will be. I’m avoiding that by taking breaks, especially for compound/leg exercises, since those seem to require the most energy and therefore strongest heart pumping.
Another trigger: skipping a meal. If intermittent fasting isn’t good for you, leave it.
In a sauna/thermal spa, there are often very warm temperatures. Some indoor pools offer "warm water days". Both warm air and water can quickly raise your body temperature, while the body tries to stay in a small range of feel-good temperatures. So cool down outside or in the ice bath more often. Don’t hop from sauna to sauna, and don’t remain in the warm water or warm air without a cooldown break.
Make typical modern health improvements: ditch alcohol and carb-loaded drinks (also alcohol-free beer), eat fresh, little-processed and rather low-carb food. Alcohol screws up sleep and muscle growth. In my experience, the headache from alcohol hangover seems to be a different type from migraine, and should be avoided in addition. I noticed that certain, highly-processed foods gave me a headache within minutes. That only happened rarely to me, but the trigger was obvious.

I haven’t written down food macros for long enough, but my current data shows that I should probably avoid a high-carb diet:

Headache severity (0-3) vs. eaten carbohydrate amount on a day (data from 61 days of food data, year 2025, scores 0-3)

In fact, the Stanton Migraine Protocol which I’ll mention in the next sections, recommends low-carb nutrition. I tried going low-carb/keto and it’s definitely helpful to feel satiated for longer and not having to cook all the time. I’d recommend it independently of my migraine research and data.
If you feel a headache coming up despite salt supplementation, there’s still the chance to stop its progression. Put the aforementioned amount of salt on a table spoon, place it under your tongue and let it dissolve. The strong salty taste is a special experience you shouldn’t miss. I’m joking. If you’re not too late, this can still make a difference on the same day. For me, it can sometimes still fight the onset, but it adds to the absolute amount of ingested salt, so safety is a concern.
Sunstroke can lead to a similar, unstoppable headache as with migraine, but this one is actually dangerous. It’s often also associated with nausea, similar to alcohol hangover. The solution is simple: avoid long exposure of the head to strong sunlight. A baseball cap can be enough on certain days. Shadow is best. And mind that air temperature and sunlight intensity aren’t much related. You can easily get a sunburn or sunstroke in winter, just like solar panels can create lots of electrical energy despite freezing temperatures. Estimate sunlight energy, not temperature.

That’s it ‒ some changes are simple, especially the salt part, and some take more effort and knowledge, like the dietary part. Mind that the intervention does not cure your migraine illness; it only prevents it, sorry!

Safety of the salt intervention

The so-called Stanton Migraine Protocol, one of the sources recommending salt ingestion in water as migraine remedy, prescribes a larger amount of salt. I initially salted each glass of water as suggested. At some point though, dizziness symptoms started. With my previous dizziness phase caused by vitamin D (see my story above), I was already aware that the attempted interventions could be the likely culprit. And indeed, it turned out as too much salt which triggered the dizziness! (I’m not taking vitamin D anymore, so that was ruled out.)

Eating more potassium, or supplementing with a 2:1 sodium-potassium salt mix instead of table salt, didn’t change that.

Regarding the safety of sodium salt in general: too much and too little sodium both correlate with higher mortality. The WHO recommends reducing salt intake from the statistical average to less than 5 g/day salt (= 2 g elementary sodium). With processed foods, it’s easy to get over that threshold, with fresh foods not at all. You’ll need to judge your individual intake and read food labels.

Since experiencing dizziness, I tried a few variations of salt dosing. For a while, I turned back to not salting my water while still drinking 2.5 to 3 liters of water a day and keeping other things roughly the same as well. I tried without salt supplementation several times, as a crossover experiment, and it seems that my body has roughly a 10-day buffer. After that, my headaches come back, but dizziness subsides. These repeated, identical results also prove that the salt really works as preventative intervention. I didn’t want dangerous side effects like dizziness, so I reckoned that taking much less salt, or every second day, might do it as well. So what works for me right now, as written above, is to dissolve only a pinch of salt per liter, several times a day. Less than a quarter tea spoon of table salt.

I’ll keep testing this. In contrast, I’m not writing down and analyzing dietary salt intake, but that’s an option, too. Overall, my recommendation is to not treat the salt intervention as a perfect, final solution, and to observe whether it works for you without side effects.

By the way, very intense gym sessions still give me temporary dizziness, independent from salt intake. It’s a complex symptom.

Scientific theory of electrolyte imbalance behind migraine

When I found electrolytes as possibly related to my headache, I didn’t know much about brain neurology yet. That changed quickly. Angela Stanton, with her neuroscience background, pioneered both in creating a reasonable scientific theory and in bringing awareness to people about what works. Being a decades-long migraine sufferer herself, she’s much more believable than strange healing stories on the internet. Creating a Facebook group for healing migraine, in which, as of 2025-05, around 19000 people have joined, takes dedication.

That said, I particularly like the neurons' electrolyte imbalance theory as described in her book and articles. Here, I’m summarizing a few resources:

Definition of Migraine
- Migraine is genetic, and a "migraine brain" is different than a healthy one. Problems with sodium/electrolyte homeostasis (balance) leads to different electrical properties, and that is critical for nerve cells (neurons).
- Restricting carbohydrate consumption can improve electrolyte balance
- Hydration isn’t drinking lots of water
- Other possible triggers, e.g. weather changes, menstrual cycles, etc.
- Recommendations for salt intervention during air pressure increase/decrease
Specifically formulated ketogenic, low carbohydrate, and carnivore diets can prevent migraine: a perspective

Front. Nutr., 30 April 2024, Sec. Nutrition, Psychology and Brain Health, Volume 11 - 2024, https://doi.org/10.3389/fnut.2024.1367570

An overview paper for her theory and what’s behind it. The abstract greatly summarizes it and I have nothing to add to this great article.

This article presents a hypothesis explaining the cause of migraines, suggesting that electrolyte imbalance, specifically a lack of sufficient sodium in the extracellular space of sensory neurons, leads to failed action potentials. The author argues that migraines are triggered when sodium channels fail to initiate action potentials, preventing communication between neurons. The article discusses the evolutionary perspective of the migraine brain, stating that migraineurs have a hypersensitive brain with more sensory neuronal connections, making them more reactive to environmental stimuli and in need of more minerals for the increased sensory neuronal communication. Since glucose is often used to reduce serum hypernatremia, it follows that a high carbohydrate diet reduces sodium availability for use in the brain, causing an electrolyte imbalance. Low carbohydrate diets, such as ketogenic, low carb-high fat (LCHF), and carnivore (all animal products), can be beneficial for migraineurs by reducing/eliminating carbohydrate intake, thereby increasing sodium availability. In support, many research papers and some anecdotal evidences are referred to. The article concludes by proposing lifestyle modifications, such as dietary changes and sodium intake management. These will provide migraineurs with a long-term healthy metabolic foundation helping them to maintain strong nutritional adherence and with that aiding continued proper neuronal functioning and migraine free life.
The book "Fighting The Migraine Epidemic: A Complete Guide: How To Treat & Prevent Migraines Without Medicine" was the first thing I’ve read. It’s much longer than the above resources, but worth a read for migraineurs or relatives, such as parents trying to help their kids, as the book is written for everyone (not just scientists). If you want to check out the complete protocol, the go-to places would be the book and the Facebook group.

Hint: I’m not advertising. Science should be as open, free and without financial interests as possible. It’s just fair and important to mention my sources of information. If there’s a sold book along a plethora of freely available resources from one scientist, that’s totally fine. I could offer my endless writeup of other resources from the last years. I must have read thousands of web pages and full-text studies. This article is already too long to list and explain everything. Therefore I refer to sources that give explanations in a single place, and which influenced me the most.

The sodium-potassium homeostasis explanation makes sense to me. But I can’t tell if it’s the correct explanation. And it doesn’t matter. The result ‒ getting rid of migraine pain ‒ is more important, as long as the intervention is safe to use. That said, I’m not ruling out other explanations. Just like I’m not excluding that I might have multiple causes for headache and other symptoms.

The end of the fight after 37 years of pain

The whole endeavor was worth the effort. I became a hobby scientist, fixed more health issues afterwards, and invest even more effort into free and open science now: I’ll publish my free data analysis program soon. I learned that some of my observation-derived ideas turned out to be true: I had my own theory that electrolytes were involved, and that’s the case. Those observations should be followed first rather than trying hundreds of likely unrelated ideas from the internet or from studies that average out effects on a large population. Most importantly, my life improved a lot by solving migraine. And even though it’s not cured, and also not 100% fixed, the pain I have remaining is definitely something I can live with. And I’m happy being able to help others now by sharing my story, solution and specific instructions, fully backed by my own efficacy and safety data.

With the problem now solved closer to the cause, rather than only treating symptoms, certain triggers are fully gone: Flights usually gave me nausea and headache; even before a flight, I would already have a bad feeling in my body. Also high-speed train rides used to give me a high headache risk, likely due to the fast and countless changes of air pressure and weather along the route. With the salt intervention, these are no longer a trigger.

The result is incredible. According to the recommended treatment goals, I’m under "optimal control" with 1-2 monthly moderate-to-severe headaches (mid of 2025, much over a year into the intervention).

From: Sacco S, Ashina M, Diener H-C, et al. Setting higher standards for migraine prevention: A position statement of the International Headache Society. Cephalalgia. 2025;45(2). doi:10.1177/03331024251320608, published by Sage here. Figure 2. Aspirational goals of migraine prevention according to the position statement of the International Headache Society.

Thank you for reading, and best of luck to all sufferers!

My research on migraine will continue. I’ll probably try again to sustain a strict low-carb diet for a longer time. This article will definitely be updated once I have new insights.

Grafana dashboards — best practices and dashboards-as-code

Thu, 21 Apr 2022 00:00:00 +0000

Grafana is a web-based visualization tool for observability, and also part of a whole stack of related technologies, all based on open source. You can configure various data sources — time series sources like Prometheus, databases, cloud providers, Loki, Tempo, Jaeger — and use or even combine them for your observability needs. As of April 2022, that means metrics, logs, traces, and visualizations based on those concepts. That is where dashboards come into play.

Dashboards are the typical first solution that even small companies or hobbyists can use to quickly get insights of their running software, infrastructure, network, and other data-generating things such as edge devices. In case of incidents, or for just glancing over them at will, dashboards are supposed to give a good overview to find and solve problems. Great dashboards make the difference of understanding an incident in 10 seconds vs. digging into red herrings for over 5 minutes. Meaningful dashboards also ease the path to setting correct alerts that do not wake you up unnecessarily.

On top of dashboards, you can leverage alerts, logs and tracing which together form a simple and helpful look at your stuff, if done right — or a complicated mess like your legacy software code, if done wrong 😏. Those other concepts are out of scope in this article. I only focus on how to achieve helpful, up-to-date dashboards, using private research and my production experience, where I extensively used Grafana dashboards — and also logs — to resolve incidents fast, and often with ease.

Typical examples on the internet present dashboards as a huge screen full of graphs, sometimes a collection of numbers or percentages. A lot to look at for human eyes. I will show you how to visualize better, with realistic examples. This article showcases solutions for Grafana and Prometheus, but can also be applied generically for other platforms.

Goals for all your dashboards

You cannot look only at best practices for a single dashboard. If you do that, you will end up with 100 dashboards, each for a single, distinct purpose, such as one per microservice. But altogether, that becomes an unmanageable mess.

Think of your company: who is the main audience of the dashboards? If you are using the "you build it, you run it" concept, it may be mostly developers (e.g. on-call engineers during an incident). For other organizational concepts, it might be SREs or infrastructure engineers. Or it may be multiple technical departments (operations + engineering). This article will focus on examples how to monitor the health of software systems (easy to apply to network and infrastructure likewise). If however your organization looks totally different and dashboards are for sales, management, compliance, information security, then the goals can be different — the article may still be helpful nevertheless. Most importantly, you as author should be part of the audience yourself, or else you are not a good fit to develop a reasonable dashboard. On the same note: great dashboards help in many ways, such as against following red herrings, but nevertheless you need application experts to resolve incidents, so those should definitely be part of the audience.

This is what we want to achieve for users:

Fast resolution of incidents, by finding the root cause and impacted services/customers fast
Users should have one browser bookmark, leading to a main, high-level dashboard. It will be the first page you open when you get called for an incident. Within seconds, it tells you which parts could be problematic, and which ones are okay (as operators in Star Trek say: "operating within normal parameters").
Show health at a glance, with a simple indicator that the human eyes can quickly consume (e.g. green or red background color), for each component
Allow drilling down into more detail (low-level) in order to come closer to the root cause if the high-level dashboard is not enough

How this can roughly work:

Create a high-level overview dashboard. It depends on your company how many of those make sense. The scope could be one system, service, product domain, or for small companies even the whole landscape at once.
Represent each component or microservice of a system in the high-level overview
For fine-grained analysis, it also often makes sense to create a separate, detailed (low-level) dashboard for each component. The high-level dashboard links to those.
From each visualization (those are the rectangles on your dashboard, such as graphs), link to detailed dashboards, prepared log queries, debugging tools on your intranet, the system/website itself, etc.
Create dashboards solely through code, to avoid having a mess of manually created, unreviewed, inconsistent dashboards after a few weeks, and the need for a company-wide "tooling switch" after a few months or years, only to clean up all of that. Users nevertheless get write access to Grafana, since that allows temporarily adapting a dashboard for own usage, such as special investigation during incidents. They should however be trained that changes should not be saved, and any saved changes will regularly be overwritten by the dashboards committed as code. Those get automatically deployed, for example by a CI pipeline.
You give no training to users. Yes, you heard right! A well-designed dashboard is 100% obvious and requires no explanation to use it, given the user knows the relevant terminology of your monitored system. Training for incidents anyway mostly happens through practice. Therefore, I recommend you present the dashboards on screen during incidents, so that other users see the capabilities they offer, and less obvious features such as the hyperlinks that can be added to the clickable top-left corner of each visualization. Code review for dashboards, and reviewers who are application experts (who understand the meaning of the displayed metrics), are essential to keep up good quality and really make the dashboards plain simple to use without explanations.
For medium to large companies in terms of head count, introducing such a consistent concept will be impossible unless technical leadership supports the full switch from the old or non-existing monitoring solution to Grafana with dashboards-as-code. It is key to document and communicate that codifying dashboards is the only way to go, listing the rationale for your company and also how a developer can start creating or modifying dashboards. This requires no more than one-page documentation/guide and an introduction by leadership or engineering managers.

Spinning up a Grafana playground in a minute

You only need this if you want to follow along with the blog article recommendations, play around with random sample data, and do not have a live Grafana instance with real application metrics at hand.

Some solutions exist — most of them using docker-compose — which allow you to easily and quickly spin up Grafana and data sources in a minute. Here, I describe the official devenv that Grafana developers and contributors use (see Grafana’s devenv README):

git clone --depth 1 https://github.com/grafana/grafana.git
cd grafana/devenv

# As of 2022-04, these are the instructions.
# Check `README.md` files for more information.
./setup.sh

cd ..

# See directory `devenv/docker/blocks` for more supported sources.
# "grafana" is not a source - this value ensures you don't have to
# build Grafana, and an official image is used instead.
make devenv sources=grafana,loki,prometheus

# To tear down later: `make devenv-down`

Now open http://localhost:3001/ and log in with user admin and password admin. Navigate to Explore mode on the lefthand navigation bar, choose gdev-prometheus as data source and query an example metric such as counters_logins. If it shows data, you are ready to play around.

Getting started with a playground dashboard

Mind that a good dashboard takes hours or days to create! As a start, the graphical way of clicking one together is the fastest. Once you have found out a good concept and layout, codifying it for the first time is some work, but worth the effort — more on that later.

If you have to experiment on a live instance, start by adding a name and hint so that your testing dashboard will not be touched by others. You can use a text visualization for that. Save the dashboard with a meaningful name. Do not use filler words like "monitoring" or "dashboard" — of course it is a dashboard… the screenshot only does this for the temporary "it’s a test" hint in the title. Now you can follow along with the recommendations and examples in this blog post.

High-level dashboard creation guidelines

Choose main input for the high-level dashboard

By first concentrating on monitoring and alerting for the main function of your business and system, you can cover almost all critical problems in subcomponents and infrastructure resources, without having to monitor those explicitly.

That needs explanation… The business in this blog article’s example is to process payments. So if payments fail, people cannot pay, and the business is at risk. Anything else is not as important, and therefore not worth to observe as the first thing. If a critical issue arises in our systems or network, the payment_errors_total metric will most likely cover that! To use IT terminology: the metric of payment failures is a significant service level indicator (SLI).

Admittedly, that will not cover if the internet, a customer, or our API are down, because payment requests would not even reach the system and therefore cannot produce logs or error metrics. That can be covered by a metric describing the payment request rate, probably by customer and payment method, since each of those have different typical traffic rates (minimum/maximum, different per timezone or day/night, etc.). We keep this shortcoming out of scope to keep the blog article simple. The point is: choose very few, essential business metrics as a start, not technical metrics.

Often, you would select the most business-relevant Prometheus metric offered by the system you want to monitor. Metrics are stored as time series and therefore very fast to query, as opposed to logs. If you use other observability tools, such as an ELK stack, you can check if Grafana supports the relevant data source. This metric would typically pertain to the methods "RED" (requests/rate, errors, duration) or "USE" (utilization, saturation, errors). The Four Golden Signals of Google’s SRE book additionally distinguishes traffic from saturation. An error metric is a good choice to start, since it is easy to determine which error rate is acceptable, and at which threshold you would consider it a serious problem.

Throughout this blog post, we will use the following example metric and simple terminology from the payments world:

Imagine we are in a company that processes payments, offering different payment methods, with each of those methods (e.g. credit card, voucher, bank transfer) having a separate microservice implementation
Counter payment_errors_total
Cardinality — the counter has these labels:
- payment_method (example values credit_card, voucher, bank_transfer)
- error_type (example values connectivity_through_internet, remote_service_down, local_configuration_error)

Metric naming and cardinality

We do not want to have separate metrics credit_card_payment_errors_total and bank_transfer_payment_errors_total! If you have microservices of the same type, as in this example one service per payment method, the metrics really mean the same thing. So rather improve your consistency and use just one metric name. Easy to do if your code is structured in a monorepo, by the way. If you have inconsistent names, it will take extra effort to repeat each dashboard visualization for each of the conventions, instead of just using labels to distinguish and plot the services (or other traits that you define as label dimensions, such as customers).

Metric names should be easy to find in code, so that developers can make sense of them fast in a stressful situation, or find out if/where a metric (still) exists. Here is a bad example where the metric name mysystem_payment_errors_total cannot be found in code: for request_type in ['payment', 'refund', 'status']: prometheus.Counter(name=f’mysystem_{request_type}_errors_total') (Python pseudocode).

Avoid high-cardinality metrics (many label combinations), since those take up lots of space, and queries take longer. Like for the logging rate of your systems, you might want to check for large metrics sporadically, or you may run into unnecessary cost and performance issues.

Stat instead of Graph for human understanding within milliseconds

A graph (nowadays called Time series visualization) for our example metric, showing payment method and error type combinations, looks like this:

Cool graph, right? And we’re already done and have a monitored system! No, this is very, very bad! Great observability requires much more than just clicking together some visuals. The example is not sufficient to monitor a service. A graph visualization is a bad way to get an impression within milliseconds. The eyes have to scan the whole rendered graph, potentially containing multiple lines on varying bounds of the Y axis. You also need to know which thresholds are bad, or configure horizontal lines on the graph which represent warning and errors thresholds. That means lots of lines, colors, and points to look at before getting your question answered: "is this normal or do we have a problem, and where?"

Graphs can be helpful if you set them up nicely, but definitely not in the high-level part of your overview dashboard.

Instead, a Stat visualization, combined with traffic light colors, gives you the answer in milliseconds: green is good, amber (yellow) is noteworthy, red is bad. In addition, I tend to use blue as "noteworthy but may not be problematic" — kind of an early warning sign or unusually high amount of traffic, such as during sales/promotion seasons. So for me personally, I like the order green-blue-amber-red. Grafana allows choosing to color the background instead of the value line, which I recommend since then your whole screen should look green most of the time (click Panel > Display > Color mode > Background), and your eyes do not need to focus on the color of tiny graph lines. Exactly one value is shown — typically a number or human description.

Settings for the above screenshot:

Prometheus query: sum by (payment_method, error_type) (increment(payment_errors_total[2m]))
Legend: {{payment_method}} / {{error_type}}
Choose Calculation > Last. That will give the latest metric value, since now is the most interesting time point to show. Aggregations such as Mean may be a useless "all problems averaged away" view if you pick a big time range such as Last 24 hours, and would therefore show different values to different people. Since the Last setting does not average at all, your query should do that instead of sampling a single raw value: Prometheus queries such as increment(the_metric[2m]), or rate(the_metric[2m]) if you prefer a consistent unit to work with, will average for you. The [2m] in there should be selected depending on how stable the metric is and how fast you need to react once the metric reaches a threshold (mind the averaging!). Magic variables like $__rate_interval may sound promising, but also have the issue that a different time range selection shows different results, and that could lead to confusion if you exchange links to dashboard views with other people during an incident.

To show the colors, you need to set thresholds on the Field tab. Setting them as static numbers (with the queried unit, e.g. "errors per 2 minutes") may work for the start. That is called Absolute in Grafana.

In our example though, the different payment methods and error types have very different thresholds: for instance, let’s say the credit_card payment method has remote_service_down errors very frequently because the 3rd party provider is unreliable and we cannot help it, so we want to set a higher threshold because it otherwise unnecessarily shows a problem. Or instead of a higher threshold, you could consider querying the error rate increment over 10 minutes, to even out any short spikes. To use relative thresholds, click Percentage and fill some values. They will use the Min/Max settings. For example: if you set Min = 0 and Max = 200, red background color above the 66% threshold will be shown above 66% * 200 = 132 (unit in this example: errors within 2 minutes). Everything above 66% will be red. Everything between 16% and 33% will be blue. And so on.

To set specific thresholds per value combination (here: "payment method / error type"), adjust Max:

Since Grafana live-previews your changes, it should be simple to choose good values for Max. Select a healthy time range for your system, and it should be green (note that the right-most time point is displayed, as we chose Calculation > Last). Select an incident time window, and choose a Max value to make it red. The other values (amber/blue) might then just work, since they are based on percentages. Start with values that work, and adjust them if you later see false positives (red when system is fine) or false negatives (green when system has problems). If you want human descriptions instead of numbers, you can also use the override feature (Field > Value mappings, or for specific fields: Overrides > [create an override] > Add override property > Value Mappings / No value), for instance to replace 0 with the text "no errors".

Example: our query shows the number of errors in 2 minutes. By looking at the graphed data of the last few days (paste query into Explore mode), we might find that 20 errors in 2 minutes are acceptable and should be shown as green. We therefore choose a slightly higher threshold of 25 to still be within the green color. Since we switch to blue color from 16%, we get Max = 25 * 100% / 16% = 156. As a result, red background color — which shouts "something is seriously wrong" — would be shown above Max * 66% = 103 errors in 2 minutes. You should experiment a little so that in healthy times, your dashboard remains green.

Side note: for "higher is better" metrics such as customer conversion rate (100% = all customer purchases succeed), you can just turn around the colors (green on top, red on bottom). The Max setting also defines the upper bound of the graph which is shown as part of the Stat visualization, so if values are higher, the line will cross the top of the Y axis and therefore becomes invisible. Not a big deal if Min/Max cover the expected range. You may also have the rare case of "too high and too low are both bad" metrics, e.g. a counter for payment requests where you always expect some traffic to be made, but also want to be warned if the request rate is soaring. The colors could be adapted to show both low range and high range as red, with green for the expected, normal range.

Keep panels small on screen

Pack only few visualizations horizontally, so the font stays large enough. Other people may work on a smaller screen than yours, or do not use full screen sizing. The repeat feature (Panel > Repeat options) makes Grafana create and align them automatically. In our example, the repeat feature is unused, but since we pulled different combinations of payment method and error type out of our (single, consistently named and labeled) metric, that will also show multiple rectangles in one Stat visualization, and try to align them on screen. In the screenshot further above, the titles are barely readable, and the visualization is large and could require scrolling on small screens. To solve that, you could:

Keep as-is and show a separate rectangle for each combination. With each added or newly monitored product/feature (here: payment methods), the whole dashboard size grows, so the page does not always look the same or fit on one screen. I’m not telling you it should fit on one screen, but a high-level dashboard must not be an endless scrolling experience.
Show only problematic items — for instance, only yellow and worse. The downside is that in healthy cases, nothing gets shown, making users unaware of how it should normally look like. See below for a better option.
Show the top 10 highest error rates (Prometheus: topk). This can be combined with traffic light coloring. If only one shows red, you will think that one payment method is down, while if multiple show red, you may think of a larger issue. With this solution, the visualization will never show as empty, so you’ll see ~10 green rectangles in healthy scenarios (the section Show only offenders or top N problematic items later explains why it may not be exactly 10, and how to fix that). Compared to the above option, this ensures that the visualization remains at the same size and does not jump around on the web page, and you know that the dashboard is still working. Just like software, dashboards can become buggy and not show you the right things, for example if someone renames a metric!

Avoid dropdowns for variable values

Typical pre-built, open source dashboards may show variables like these:

Such low-level investigation dashboards are helpful, but for high-level purposes, your dashboard should show everything at one glance. If you have clusters A and B, of which one serves traffic at a time and the other one is the passive backup, you should not be required to know by heart which cluster is active. Instead, the dashboard should show health across clusters. You can still repeat your visualizations for each cluster, or query for by (cluster) if it proves helpful — but probably rather on low-level dashboards. For our example of an error metric, you want to know if it goes above a threshold, and sum(rate(…)) does not strictly require distinction by cluster in the high-level visualizations.

Dashboards-as-code and GitOps (committed dashboards)

The Grafana provisioning documentation describes how to use a local directory which Grafana will watch and load dashboards from. However, you cannot just write dashboards as plain JSON as a human. Also, more ergonomic ways of importing dashboards, e.g. from a Git repo, are not supported yet but Grafana Labs is considering improvements.

Here is the rough plan:

Generate Grafana-compatible JSON containing dashboard objects. The Grafonnet library is the official way to develop dashboards using the Jsonnet language. There is also Weaveworks' grafanalib for Python which is not presented in this article.
Add that build to CI. Deployment means that you have to make the JSON files available to Grafana in some directory.
Configure Grafana to pick up files from that directory
Also, support a developer workflow

Advantages of not creating dashboards visually through the Grafana UI:

You get a developer workflow. The later section Fast develop-deploy-view cycle explains how that works. It is surely worse than WYSIWYG. But a programming language such as jsonnet makes the repetitive parts so much more expressive and consistent. With file watching tools and a Grafana API key, you can deploy each saved change within a second and only need to reload in your browser. Very soon, it will be a great experience, once you have assembled some basic functionality and learned the language. While jsonnet is not the best or most well-known language, better alternatives such as CDK integration or other libraries may arise in the future. And once you have developed a dashboard, it is easy to improve in small increments, similar to software code. In fact, just like the main parts of your software code, dashboard code will typically be written once and then not touched for a long time. Codifying dashboards therefore leads to long-term consistency, yet making large changes easy. If you tell people to visually create dashboards instead of dashboard-as-code, after a few months you will see a bloat of outdated, awful, non-informative and unreviewed dashboards, with lots of them probably unused. Coded dashboards improve quality and allow you to throw out old stuff easily and with the needed 4-eye principle. You can still allow people to visually author dashboards, but tell them they will be automatically destroyed every Deleteday. Changes can be tested visually (at best with production data!) but should then be ported back into code. Once coded, a dashboard should go through review, and it is very likely that most changes reuse homegrown jsonnet functions instead of reinventing each dashboard from scratch. Simple improvements should become one-line changes. Reviewed dashboards are much more robust, stable and avoid outdated parts.
Consistent annotations such as deployment events, for instance by reusing a custom function which adds them everywhere
With such a custom base library of functionality, nobody needs to be an expert to get started making changes to monitoring
Vendor lock-in can be avoided to some extent. The Grafonnet library and custom object properties are highly Grafana-specific. No way around that. Even if you use WYSIWYG editing and storage, the dashboard is stored as Grafana-specific JSON, not transferable at all to other providers. I recommend to choose one observability platform and stick with it for years — just like you would for an infrastructure platform such as Kubernetes and its ecosystem. This article shows you how to cleanly manage your dashboards as code. That way, improving or fixing all dashboards at once is done within minutes, and you get the benefit of code review. If you write some high-level jsonnet functions such as addDashboard(prometheusQuery, yellowThreshold, redThreshold) (pseudocode), you can even abstract Grafana-specific stuff to some extent and later port more easily once the company switches to another observability provider or cloud product. I cannot provide experience or examples (yet) whether such an abstraction layer is worth the effort.
Old and unused stuff is easy to detect and delete. For example, you can grep for metric names or other things that do not exist anymore in your software, and delete those dashboards or visualizations from the code. Likewise, it is easy for a developer to find out where and if a metric is actually used for monitoring.
A monorepo keeps all observability-related things in one place. You do not want to copy the solution into every engineering team’s projects, since it may take some effort upfront to include it with your existing CI/CD/GitOps tooling, and decentralizing the solution defeats many advantages (consistency, shared base library functions, availability of good examples for new joiners to learn from). If you can reuse your software (mono)repository, even better, since that makes it easier to put relevant changes together — such as adding/removing/extending metrics or log fields.

Dashboard generation from jsonnet code using the grafonnet library

Let’s set up the generation of our dashboard from code. First, the necessary tools. jb (jsonnet-bundler) will be used as jsonnet package manager, and go-jsonnet (not the much slower C++ implementation!) for the language itself.

# macOS + Homebrew
brew install go-jsonnet jsonnet-bundler
# Any other OS / package manager combination
go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
go install github.com/google/go-jsonnet/cmd/jsonnet@latest

We need jsonnet libraries for the outputs we want to generate. In this case, Grafonnet for Grafana dashboards is enough. If you want to generate non-Grafana resources, consider the kube-prometheus collection which covers much of the Kubernetes landscape (but mind its Kubernetes version compatibility matrix).

cd my/version/control/repository
jb init
jb install https://github.com/grafana/grafonnet-lib/grafonnet
echo "/vendor" >>.gitignore

Commit jsonnetfile.json and jsonnetfile.lock.json. The lock file ensures that the next user gets the same version of libraries, so you do not need to commit the downloaded modules in the vendor directory. But that probably starts a flame war among Go developers, so please decide yourself…

Instead of jb, you could also use Git submodules, but probably will regret it after adding more dependencies — I did not test that alternative.

First write a minimal dashboard as code and save as dashboards/payment-gateway.jsonnet:

local grafana = import 'grafonnet/grafana.libsonnet';

grafana.dashboard.new(
  timezone='utc',
  title='Payment gateway (high-level)',
  uid='payment-gateway',
)
.addPanel(
  grafana.text.new(
    content='Yippie',
    mode='markdown',
  ),
  gridPos={
    x: 0,
    y: 0,
    w: 24,
    h: 2,
  },
)

Play around manually:

# This may fail if you forget the newline at the end of the file, or the code is
# otherwise not compatible with jsonnetfmt's expectations
jsonnetfmt --test dashboards/payment-gateway.jsonnet || echo "ERROR: File must be reformatted" >&2 # use in CI and IDE ;)

export JSONNET_PATH="$(realpath vendor)" # same as using `-J vendor` argument for the below commands
jsonnet-lint dashboards/payment-gateway.jsonnet # use in CI and IDE ;)
jsonnet dashboards/payment-gateway.jsonnet

The last command outputs a valid Grafana dashboard as JSON. To manually apply it, open the test dashboard in your Grafana instance, then ⚙️ > JSON Model > copy-paste generated JSON > Save Changes. You should now see the dashboard as described by code — containing only a text panel that says "Yippie". This is the simplest development workflow. But it is very tiring to always go and copy-paste some generated blob and save it with many clicks. And the workflow is not visual (WYSIWYG). The next section explains a better way.

Fast develop-deploy-view cycle

While there is no WYSIWYG editor for the whole conversion from jsonnet to a visual dashboard in Grafana, here is an alternative which works right now (in 2022):

Create a personal API key (side bar > Configuration > API Keys) with Editor permission
Use entr or other file watching tool to execute a script whenever you save your jsonnet files. IDEs may offer this action-on-save feature as well.
That script uses your API key to overwrite dashboards in your Grafana instance

Preparation in your shell:

export GRAFANA_API_KEY="THE_API_KEY_YOU_CREATED_WITH_EDITOR_PERMISSION"
export GRAFANA_URL="THE_GRAFANA_URL" # if you use devenv: `export GRAFANA_URL="http://localhost:3001/"`

Next, store the following script as watch.sh:

#!/usr/bin/env bash
set -eu -o pipefail

error() {
	>&2 echo "ERROR:" "${@}"
	exit 1
}

[ -n "${GRAFANA_API_KEY:-}" ] || error "Invalid GRAFANA_API_KEY"
[[ "${GRAFANA_URL:-}" =~ ^https?://[^/]+/$ ]] || error "Invalid GRAFANA_URL (example: 'http://localhost:3001/' incl. slash at end)"

[ $# = 1 ] || error "Usage: $(basename "${0}") JSONNET_FILE_OF_DASHBOARD"
dashboard_jsonnet_file="${1}"

rendered_json_file="/tmp/$(basename "${dashboard_jsonnet_file%.jsonnet}").rendered.json"

cat >/tmp/render-and-upload-dashboard.sh <<-EOF
	#!/usr/bin/env bash
	set -euo pipefail
	clear

	# Render
	echo "Will render to \${2}"
	JSONNET_PATH="\$(realpath vendor)"
	export JSONNET_PATH
	jsonnet-lint "\${1}"
	jsonnet -o "\${2}" "\${1}"

	# Enable editable flag and upload via Grafana API
	cat "\${2}" \
		| jq '{"dashboard":.,"folderId":0,"overwrite":true} | .dashboard.editable = true' \
		| curl \
			--fail-with-body \
			-sS \
			-X POST \
			-H "Authorization: Bearer \${GRAFANA_API_KEY}" \
			-H "Content-Type: application/json" \
			--data-binary @- "${GRAFANA_URL}api/dashboards/db" \
		&& printf '\nDashboard uploaded at: %s\n' "$(date)" \
		|| { >&2 printf '\nERROR: Failed to upload dashboard\n'; exit 1; }
EOF
chmod +x /tmp/render-and-upload-dashboard.sh

echo "${dashboard_jsonnet_file}" | entr /tmp/render-and-upload-dashboard.sh /_ "${rendered_json_file}"

Run the script, passing the source files as argument.

# Make script executable
chmod +x watch.sh

./watch.sh dashboards/payment-gateway.jsonnet

The script listens for changes to the source file and then overwrites the dashboard in your Grafana instance with an API request. Open your Grafana instance and find the dashboard by its title. Mind that the uid field in source code must be set to a fixed value per dashboard in order to overwrite the dashboard instead of creating new ones.

This workflow gives you results within seconds and you only need to refresh in your browser to see saved changes. You may want to keep an eye on your terminal, since jsonnet is a compiled language and therefore spits out errors if you make coding mistakes.

Automatic provisioning of generated dashboards into Grafana instance

Grafana provisioning allows automatic reloading of dashboards from a certain place. We want to load the committed, generated dashboards. On the other hand, we will not fully recreate Grafana and its database on every commit to some control repository — I’d call that murder by GitOps, and the sheer idea does not sound useful, as users and their settings are stored in the database, so we do not want to manage everything as code.

We can set up Grafana in various ways: via Ansible on a single server, with containers on Docker or Kubernetes, manually run on the company’s historic Raspberry Pi in the CEO’s closet, etc. They luckily all work the same way for configuration: local files.

In the dashboard provisioning section, it says you can put "one or more YAML configuration files in the provisioning/dashboards directory". We try that with a realistic setup of Grafana, and for the sake of simplicity, we assume a GitOps model, meaning that Grafana loads dashboards from committed files. You have to adapt this article yourself to your respective setup.

The following instructions show you how to do this on a self-hosted Kubernetes setup of Grafana (see their setup instructions). As I do not have a Grafana Cloud account right now, I cannot tell if the cloud offering provides this much flexibility, or any good way of using the GitOps model — if they do, the documentation misses this important piece (as of 2022-04). We use kind to simulate a production Kubernetes cluster.

# Create Kubernetes cluster and a target namespace
kind create cluster --kubeconfig ~/.kube/config-kind
export KUBECONFIG=~/.kube/config-kind
kubectl create ns monitoring

# Install Prometheus so we have a data source to play with
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install -n monitoring prometheus prometheus-community/prometheus

To install from Grafana’s Helm chart, you need to configure it. Store the following content in grafana-values.yaml:

adminUser: admin
adminPassword: admin

# Disable persistence so all data is lost on restart. That's a little unfair to your users,
# though, so you may want to instead combine GitOps with a delete-every-Sunday concept.
# A real production setup would provide persistence, but that is out of scope for this article.
persistence:
  enabled: false

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: prometheus
        type: prometheus
        url: http://prometheus-server
        access: server
        isDefault: true

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: default
        orgId: 1
        folder: ""
        type: file
        disableDeletion: false
        updateIntervalSeconds: 10 # how often Grafana will scan for changed dashboards
        allowUiUpdates: true
        options:
          path: /var/lib/grafana/dashboards/default
          foldersFromFilesStructure: false

rbac:
  extraRoleRules:
    # Allow k8s-sidecar image to read ConfigMap objects in same namespace
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["get", "watch", "list"]

extraContainers: |
  - name: collect-dashboard-configmaps-in-directory
    image: kiwigrid/k8s-sidecar:latest
    volumeMounts:
      - name: collection
        mountPath: /tmp/collection
    env:
      - name: LABEL
        value: "collect-me"
      - name: LABEL_VALUE
        value: "grafana-dashboard"
      - name: FOLDER
        value: /tmp/collection
      - name: RESOURCE
        value: configmap

extraVolumeMounts:
  # This will implicitly create an `emptyDir` volume as well (quite surprising),
  # so we do not require the value `extraContainerVolumes`.
  - name: collection
    mountPath: /var/lib/grafana/dashboards/default
    readOnly: true

And continue installation:

# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install -f grafana-values.yaml -n monitoring grafana grafana/grafana

# Wait until the installation is ready, then keep this running in order to access
# Grafana in the browser
kubectl port-forward -n monitoring svc/grafana 7878:80

Now open http://127.0.0.1:7878/ and log in with admin:admin.

Grafana reads dashboards from a directory structure. In the Kubernetes world, we can put generated dashboards into a ConfigMap in order to mount it into the Grafana directory structure. They however have a 1 MiB limit each. An example high-level dashboard of mine takes 115 kiB when Base64-encoded, so you also will not be able to say "we can fix this problem later", since you will reach the limit very soon. We will render all dashboards into one manifest file. That YAML file will contain one ConfigMap object per dashboard. Committing that file in a GitOps fashion is easy to add in your CI (git clone && git add && git commit && git push), but that is out of scope for this article. You could also do CIOps (kubectl apply from CI; not recommended), or just rsync -r --delete if you have Grafana on physical, mutable hardware and not on Kubernetes, or whatever other way of deployment to the directory structure. Make sure you overwrite during deployment instead of only adding new files/dashboards, since deletion and cleanup of technical debt is just as important as it is for writing software. Treat your Grafana instance and database as something that gets reset regularly, from a committed state. This avoids people making manual, unreviewed edits.

As you can see in the configuration, we use k8s-sidecar to automatically collect all dashboard JSON files into one directory for use by Grafana. Each ConfigMap must have the label collect-me: grafana-dashboard to get picked up. The following script creates such ConfigMap manifests. I do not explain here how to integrate it with your specific CI tool, but that should be easy if it works locally. Save the script as render-and-configmap.sh.

#!/usr/bin/env bash
set -eu -o pipefail

error() {
	>&2 echo "ERROR:" "${@}"
	exit 1
}

[ $# = 1 ] || error "Usage: $(basename "${0}") JSONNET_FILE_OF_DASHBOARD"
dashboard_jsonnet_file="${1}"

rendered_json_file="/tmp/$(basename "${dashboard_jsonnet_file%.jsonnet}").rendered.json"

# Render
JSONNET_PATH="$(realpath vendor)"
export JSONNET_PATH
jsonnet-lint "${dashboard_jsonnet_file}"
jsonnet -o "${rendered_json_file}" "${dashboard_jsonnet_file}"

# Grafana wants `.json` file extension to pick up dashboards
kubectl create configmap "$(echo "${dashboard_jsonnet_file}" | openssl sha1)" \
	--from-file="$(basename "${dashboard_jsonnet_file%.jsonnet}").json"="${rendered_json_file}" \
	--dry-run=client -o json \
	| jq '.metadata.labels["collect-me"]="grafana-dashboard"'

Ensure you are still pointing KUBECONFIG to the desired Kubernetes cluster, and test dashboard deployment like so:

# Make script executable
chmod +x render-and-configmap.sh

./render-and-configmap.sh dashboards/payment-gateway.jsonnet | kubectl apply -n monitoring -f -

Head over to the Grafana instance running in Kubernetes, and you see that the dashboard was already loaded. Integrate this with your CI pipeline, et voilà, you have a GitOps workflow!

Once done with the cluster, you can delete it:

kind delete cluster

jsonnet for dashboards — some tips

Recommendations specifically for the jsonnet language.

Distinguish environments

You will surely have different environments, such as dev/staging/prod. They have varying URLs, potentially different set of running systems, and thresholds for production not always make sense in pre-production environments. Pass --ext-str myCompanyEnv=prod to the jsonnet tool to pass in a variable which you can use inside the source code. This allows previewing dashboards with development data before a software feature even goes live, and you will have very consistent views across environments. Example usage:

assert std.extVar('myCompanyEnv') == 'dev' || std.extVar('myCompanyEnv') == 'prod';

{
  my_company_environment_config:: {
    dev: {
      environment_title: 'Development',
      prometheus_cluster_selector: 'k8s_cluster="dev.mycompany.example.com"',
    },
    prod: {
      environment_title: 'Production',
      prometheus_cluster_selector: 'k8s_cluster="prod.mycompany.example.com"',
    },
  }[std.extVar('myCompanyEnvironment')],
}

Alternatively, --ext-code-file seems also a viable option, but I have no experience with it (see external blog post Grafana dashboards and Jsonnet which showcases this parameter).

This can also be interesting if you have a dev/prod split for your Grafana instances.

Consider jsonnet as full programming language

A simple dashboard should be a matter of only few lines of code. Follow the jsonnet tutorial to learn more how to achieve that. You will see some similarities with Python, for example string formatting with %, slicing, array comprehension, modules/imports and other syntax that can make your code easier and shorter.

If you think your users are not technical enough, jsonnet may not be a good fit unless you either provide high-level functions, or replace the whole jsonnet+Grafonnet rendering idea with your own custom solution that does the same thing: output dashboard definitions as Grafana-compatible JSON.

Grafana-specific dashboard tips

Some small tips and their solution, some with jsonnet examples.

Categorize, do not make a dashboard mess

Use the folder structure to organize your dashboards. You and your colleagues will surely play around with test dashboards, and mixing them with production-ready, usable ones is not helpful. Also, particularly if you have many systems to watch, you want everything categorized for easy access. Product-unrelated dashboards, such as monitoring for Kubernetes clusters or infrastructure, can go into a separate category. Unfortunately, you cannot set the parent folder through jsonnet as of 2022-04, but it has to be achieved as part of deploying the generated dashboards.

Grouping within a dashboard

For very detailed dashboards, you may have a lot of graphs. While this is typically discouraged, your software may really have so many important metrics. In such case, group them into rows. They are collapsible and ease navigation. Maybe Grafana could consider adding a "Table of contents" feature to jump around quickly on a dashboard, using a navigation sidebar.

Make dashboards editable

In the developer workflow above, we explicitly set dashboards to editable. You may want this for the GitOps/CI workflow as well. This is helpful because incidents sometimes require a bit of playing around with shown data. Users should however not save changes, since they are supposed to be overwritten regularly by deploying dashboards from committed code.

By default, hovering over a graph with many series shows them in a box in alphabetical order of the display label, e.g. sorted by {{customer_name}}. Typically however, such as for error rate metrics, you want the top values shown first, since the bottom of the tooltip may be cut off in case of many entries. Go to setting Display > Hover tooltip > Sort order and adjust to your liking (e.g. Decreasing). With jsonnet, use grafana.graphPanel.new(sort="decreasing") (not documented as of 2022-04).

Do not confuse the tooltip with the legend (which also has a configurable sort order!).

Y axis display range

Many metrics only produce non-negative numbers. Graphs with such a metric on the Y axis should therefore have Visualization > Axes > Y-Min set to 0 instead of auto in order to save screen space by not showing the negative area. Another problem is that you often want 0 to be the lower bound, or else Grafana chooses the display range based on the available data. In jsonnet: grafana.graphPanel.new(min=0). Setting the maximum may be helpful if you know the number range (e.g. disk full 0-100%) and want to have a consistent display.

See how the bad example on the left makes you think of a fluctuating metric. The corrected example on the right shows that in reality, the value is quite stable. In general, make trends easier to recognize for the eyes.

Be cautious with your use case, though. If you want to display and warn when a disk gets full, for example, you better extrapolate and display the trend. Or use a Stat visualization with a warning color once the trend reaches "disk becomes full within 30 days". The respective alerts need to be designed in a similar way. Otherwise, if a short time range is selected, the user may not see that the disk usage is going up a lot, as the difference between 200 GiB and 210 GiB may not look dramatic with Y-Min set to zero.

Link to detailed dashboards, logs, other observability tools

Use grafana.graphPanel.new(…).addLinks(…) to create panel links:

grafana.graphPanel.new(
  // ...
).addTarget(
  prometheus.target(
    // ...
  )
).addLinks([
  {
    title: 'Logs - Payment gateway',
    url: ...,
  },
  // ...
])

For links to detailed dashboards, consistently pre-select a reasonable time frame such as now-30m.

If you consistently tag dashboards, you can use dashboard links to put clickable links to related dashboards on top. You can also add external links such as other company tools. I have not used this feature yet and typically rather repeat the links on each panel since that does not require scrolling all the way to the top. With jsonnet, it is easy to provide a consistent set of links (as dashboard or panel links).

Use variables for repetitive values

In rare cases, you want a repetitive variable such as datacenter = cluster="dc"\,host=~"server.*" so that queries become less verbose: sum by (payment_method, error_type) (rate(payment_errors_total{$datacenter}[2m])). If the value is used in a label filter of a Prometheus query, as in this example, remember that commas need to be escaped with a backslash, or else Grafana treats the comma as separator between different choices for the variable value.

Even if you use jsonnet, you should use variables instead of filling a hardcoded value into each query. This allows users to change all visualizations on a dashboard at once (at ⚙️ > Variables).

Consider hiding those variables on the dashboard if their sole purpose is to avoid repeated, hardcoded values. See also below for some rationale.

Clearly differentiate environments

You do not want to be looking at a development dashboard while debugging a production issue, so make that mistake impossible to happen. Possible solutions:

Separate Grafana instance per environment. See this Reddit thread for some options.
Set the category and title of each dashboard so that non-production ones show a clear hint
Different colors and backgrounds
Different Grafana UI theme per environment. I am not aware of an official way to customize styles using CSS or external themes. You could patch built-in themes and build Grafana yourself, or use the Boomtheme plugin. I did not test those options. Users can change their own preference (light vs. dark), so this idea anyway does not really help unless you hardcode one fixed, customized theme. The feature request Custom UI themes discusses solutions and describes drawbacks of the available plugin.

Grid positioning

The grid position must be specified explicitly:

.addPanel(
  [...],
  gridPos={
    x: 0,
    y: 0,
    w: 24,
    h: 12,
  },
)

See Panel size and position documentation — width is split in 24 columns, height is 30 pixels each. I recommend you use 12 or 24 columns width for readability on small screens, and set a reasonable, consistent height for all visualizations on a dashboard.

As of 2022-04, you cannot easily align visualizations using jsonnet. You can hardcode x/y absolute values to your liking, but that is a hassle since you do not want to develop a user interface in an absolute grid, right? I recommend setting both to 0 in order to automatically align the visualizations on screen.

Choose the right data unit

Choose the right unit, e.g. Seconds, Requests per second, etc.

Mind subtle differences between the built-in choices, e.g. Duration / seconds will show long text such as "412 milliseconds" which makes it hard to put much information on one screen — consider using Time / seconds instead.

Also, do not confuse the order of magnitude: if your data is provided in seconds, do not choose Time / milliseconds since that would show falsified values.

UTC timezone everywhere

Grafana’s default is to use the browser timezone. Particularly for international companies or those who have international customers, consistent values and avoidance of confusion are important. Employees usually do not open the user settings page, for example to choose light/dark mode or their timezone preference, resulting in inconsistent customer and incident communication regarding dates and times. I am working from Germany and keep seeing confusion between CET/CEST once daylight saving time toggles, and sometimes even do such mistakes myself. Let’s avoid that and communicate only in UTC, and default to UTC in tools such as Grafana.

By writing a jsonnet wrapper function instead of using raw calls to grafana.dashboard.new, you can set that as your default for generated dashboards.

Somewhat related xkcd comic: ISO 8601. Did you know that the Z suffix in 2022-04-21T17:13Z stands for UTC ("Zulu time") and is therefore a pretty good abbreviation? Most non-technical people rather know the suffix "UTC", so that one is preferable.

Do not rely on `default` data source

Even if you rely on one Prometheus-compatible source in the beginning, you will very likely add more data sources, or migrate to another one, in the future. Therefore, explicitly define the source in each visualization. In jsonnet, use for example grafana.statPanel.new(datasource='thanos'). In general, do not ever name something default, anywhere. The same applies to the words "old" and "new", since "new" is always the next "old".

Heatmaps

Those are hard to set up since the UI does not give guidance. You have to set several options correctly to see reasonable results:

Prometheus query example: sum by (le) (increase(prometheus_http_request_duration_seconds_bucket[1m]))
Query > Format: Set to Heatmap instead of Time series
Visualization: Choose type Heatmap
Visualization > Y Axis > Data format: Time series buckets
Visualization > Y Axis > Unit: Choose according to the metric, typically seconds (s)
Visualization > Y Axis > Decimals: For seconds (s) or other time unit, use 0 decimals, as Grafana automatically shows the appropriate text "ms"/"s"/"min", so the .0 decimal after each axis label is useless. This would best be fixed within Grafana source code.
The Y axis will only be sorted numerically once you change the query legend to {{le}}
Visualization > Display > Colors: I recommend opacity-based coloring with full blue (rgb(0,0,255)) as strongest color, in order to see things without getting eye strain or having to come close to the monitor. Use a color that is visible with light and dark theme. I would love to see the thresholds feature for the heatmap visualization as well, so that good values can be colored green, and bad ones yellow or red. Right now, colors are assigned by how often a value range ("bucket") appeared, not by the value itself — that means your eyes have to rest on the visualization for some seconds to understand it.
Visualization > Tooltip: Enable, and optionally show the histogram for fast interpretation of the value distribution on mouseover
This option seems not available through the UI anymore for Prometheus data sources, but let me put this here for the record: if the visualization is showing too detailed information (too many faded bars), limit Query > Query options > Max data points to e.g. 24.

Instead of a histogram, laying out the information as percentiles on a Stat visualization may give a faster overview and is preferable on high-level dashboards. For example, p50 (median), p95 and p99 percentiles are often useful. Make sure you use the rate function inside histogram_quantile.

Example Prometheus query: histogram_quantile(0.95, sum(rate(prometheus_http_request_duration_seconds_bucket[2m])) by (le))

Display interesting events as annotations

Grafana annotations can mark interesting time points on graphs. Among many imaginable events to enrich on your dashboards, software and infrastructure deployments are the most interesting ones since change in a technology-driven company usually means risk and the potential for failure. In an incident, the starting point is often known quite soon by looking at dashboards. If graphs additionally show whether and when changes were made, you have better chances to find the cause.

Do not bother adding annotations manually (e.g. time window of every deployment), since people will forget the procedure, get the timezone wrong, and it only adds an unnecessary burden which should be automated.

Here is an example how to consistently show Argo CD sync events on your dashboards. Those mostly relate to real deployments. When I developed that query, no better, human-level event type was available. You may want to tweak this to your own use cases.

local grafana = import 'grafonnet/grafana.libsonnet';

{
  deployments:: grafana.annotation.datasource(
    name='Deployments related to payment methods and their infrastructure',
    datasource='loki',
    expr=|||
      {app_kubernetes_io_name="argocd-application-controller"}
      |~ "reason=(OperationStarted|ResourceUpdated)"
      | logfmt
      | dest_namespace =~ "payment-methods|ingress-nginx"
        and
        msg =~ "(?i).*?(?:initiated.*(?:sync|rollback)|sync status: OutOfSync -> Synced|health status: Progressing ->).*"
      | line_format `App {{.application}}, namespace {{.dest_namespace}}, cluster {{.cluster}}: {{.msg}}`
    |||,
    iconColor='blue',
  ),
}

You can now use grafana.dashboard.new(…).addAnnotation(deployments) to add the annotations to your dashboard.

Self-describing visualization titles

Each visualization’s title should be self-describing. Bad: "Error rate". Good: "Payment methods — error rate of requests to provider".

One reason is because you can link to a single visualization which is helpful during incidents to tell others what exactly you are looking at (or to present one detail on a really large TV):

And again, it helps the eyes to quickly get a full picture instead of having to look at multiple locations on screen.

In Grafana, the dashboard title is always displayed, even for such single-visualization URLs. So if your dashboard is nicely titled "Payment gateway (high-level)", that will already be a good starting point and you may not even need or want verbose visualization titles.

For averaging queries like sum(rate(metric[5m])), which may constitute most of your dashboards, you should consider adding the interval hint (e.g. abbreviated [5m]) to the visualization title — and/or the Y axis — so that users are aware how fast a recovered metric or an error peak will become visible.

Observability tips not specific to Grafana

These tips relate for example to Prometheus query practices and other things that do not require Grafana in the monitoring stack per se.

Stay consistent in naming metrics and labels

The Prometheus naming practices page gives very good guidance, such as to use lower_snake_case, name counters xxx_total or specify the unit such as xxx_seconds.

No need to create a metric for everything / how to easily get started monitoring an uninstrumented application

The 3 current pillars of observability — metrics, logs and traces — may not remain considered the best solution forever. We can expect tooling to try and combine them in the future, such as "metrics from logs" features. You want to observe your applications with minimum instrumentation effort? Then sometimes, a LogQL query such as sum(count_over_time(… [15m])) to look for specific log lines may be what you want (temporarily), instead of developing and maintaining a new metric. Beware however that log text tends to change much more frequently than metric names, and how much slower and more expensive it is to query logs. A totally uninstrumented application can easily be monitored if you have access to its logs. Later on, you can make the dashboards more efficient once you learned which indicators are important to show the application health, and which ones are not. Very helpful if you are just getting started with observability.

Show only offenders or top N problematic items

You can use > 0 or topk(…) > 5 to display only offenders in your high-level dashboard (please also read Keep panels small on screen above).

For example, the customers with the highest concurrency of API requests. Use Value mappings feature to map "null" to e.g. "currently low concurrency" for better understanding in humans (since Stat visualizations always show something). Together with green/yellow/red thresholds, this explains in 2 seconds what the current value is and whether it is problematic. As explained before, use Calculation > Last if only the latest value is relevant — you do not care about the Average API concurrency over 3 hours while debugging an incident, right?

In our payment example, we could alternatively show the payment methods with the highest rate of errors. Or depending on the business, define each payment method’s business importance in code and then only show the most critical products with a label filter (e.g. importance="boss_says_this_is_super_critical"; or name them "Tier1", "Tier2", etc.).

Note that topk(5, …) may not show you the top 5 items if evaluated for a graph, since the "top 5" are checked for many time points and all resulting items are shown. The same applies to Stat visualizations — unless you choose Instant to only choose the end time point, but that can falsify the desired data to show. If you really want only up to 5 items to be shown, use the @ end() modifier to evaluate only the latest top-5 values in the range (thanks to a blog reader for pointing this out! – available since Prometheus v2.25.0).

Mind test and synthetic traffic

In a modern infrastructure, you might run synthetic test traffic to verify the end-to-end health of your applications. Since those are not from real customers, you should check if that should be shown differently or excluded from certain dashboards or visualizations.

Daytime vs. nighttime

If your business is mostly in a certain region or timezone of the world, such as European payments, traffic goes down at night. Consider different error and request thresholds at day and night, respectively. Visualizations should be clearly distinguished with e.g. 🔆 or 🌒 in the title.

Prometheus allows the time distinction with and/unless hour() >=6 <21. This can be tricky, though: in special cases such as calculations sum(…)/sum(…) and hour() >=6 <21, label set matching will surprise you with an empty result. Example to fix that: (sum by (something) (rate(some_metric[15m]))) / sum by (something) (rate(some_metric2[15m])) and ignoring(something) hour() >=6 <21.

This is cumbersome and should be avoided for the start, unless you really need such a strong distinction by time.

Do not use `rate(…)` alone

The same applies to calculations like rate(…) / rate(…). Why? Any change to the labels will make them explode into many series. Combine rate with sum or sum by.

Prefer counters over gauges

A counter in Prometheus represents a value that can only increase. In contrast, a gauge can take an arbitrary value.

In regular scrape intervals, a metric’s value gets collected by Prometheus. A longer scrape interval means less storage and cost, but can mean that a short spike of a gauge’s value is not stored at all, and therefore also will not produce an alert. So prefer a counter if possible for your for use case, since its value does not lose increments (but on the other hand, it only supports increments).

Good example for using a gauge: queue size. Items can be processed, i.e. removed from the queue, or added. The more interesting metrics for queues however are error rate and per-item processing time.

Make observed components distinguishable

To find a root cause quickly in case of problems, dashboards must allow drilling down into details. In our example of payment methods as products, each of them could fail separately, or several/all at once. This must be visible in visualizations and alert messages.

Examples why this distinction is important:

1 payment method failing — only that application’s code might be affected, for example from a bad change recently deployed
Multiple payment methods failing — perhaps those have something in common, such as serving traffic from a certain cloud region or Kubernetes cluster, or which are otherwise special (in the middle of a migration, feature toggled, traffic pattern changed, rate limit of database reached, etc.)
All payment methods failing — bad code change affecting all those applications was introduced, networking issues, infrastructure down, other catastrophic scenario

Other ideas for details to drill down into: per customer, per Kubernetes cluster, per cloud region, per API endpoint. For some of these, you may be able to leverage variables (mind Avoid dropdowns for variable values), while some value ranges may simply be too large — for instance if you have a million customers — and you should rather show the top N problematic ones (Show only offenders or top N problematic items).

Summary

I showed how high-level dashboards and main business metrics cover most of your monitoring and incident resolution needs. On top of that, the article explains the advantages of dashboards as code and how to apply that concept, using jsonnet, the Grafonnet library and working scripts to integrate in your developer and CI/CD/GitOps workflow. Lastly, I listed the best practices for dashboard creation and visualization so that your monitoring becomes easier and faster to use.

Out of scope

Detailed relation to logs, traces, alerting, and other tools. Great dashboards can help you shape alerting — particularly, I mean that if you have built an understandable and quickly navigable dashboard without any clutter, then alerts should cover those observed areas. For example, if your revenue is driven by successful outcomes of payments, that should be on your main dashboard of the payment gateway, and represented in alerts. Such a high-level alert can replace a hundred fine-grained alerts. How? Here’s an example alert: "for payment method SuperFastPay, alert if there are more than 50 failed payments per minute" (set this value based on an expected failure rate). Once such an alert is received, and the on-call engineer opens the main dashboard, or the SuperFastPay-specific dashboard (if that even makes sense), it should show red for that component. The detailed dashboard may show things like failure type statistics based on metrics, or the most common recent errors in logs. If it shows you mainly internet/connectivity issues, follow your way to the payment logs and infrastructure dashboards, for example (which at best would be linked). In the end, you may find that one of your cloud availability zones A/B/C, in which the software runs, does not have internet access. And that only by getting alerted about the most important business symptom, not because you had put large effort into monitoring internet connectivity from those availability zones. If only your dashboards make sense, allowing you to navigate quickly from symptom to root cause, you can probably live with fewer alert definitions overall. This example is not from production, but a wild dream of mine if all the suggestions are optimally applied. Surely you still want alerts for symptoms in infrastructure/platform/network, particularly if the company reaches a scale where those are handled by separate teams, but those alerts then may not need highest priority ("P1") — while business-critical symptoms like failing payments of your customers should be P1 alerts. The fewer high priority alerts you have, the better people’s work life, sleep and therefore productivity will be.
How hard it is to convince people of doing dashboards in code. There are very valid points against it, such as the missing WYSIWYG support as of 2022-04. Those can mostly be resolved with good tooling or a reasonable "how to develop a dashboard" README file. Other concerns are often just opinionated, and you will simply need to take the decision "do we allow it to become a mess or not". I recommend vendors to make codifying resources easier, so that even less technical people will be able to work with this concept. Exporting a visually-crafted dashboard to JSON is unfortunately not a solution, since that diminishes many of the advantages explained in this article (such as consistency).
The article is all about live monitoring of a service/system which could have incidents at any time. For example, an API serving requests for customers. There are lots of other use cases where monitoring, alerting and tracing may help, such as performance issues, SLOs, business statistics and intelligence.
Accessibility / color blindness support. Red and green may not be the best options, but I do not have the experience to give help here.
Installation and maintenance of the observability stack does not belong in this article. Dashboards are most helpful if you can also look at historical data and not only use them for short-term review of incidents. Therefore, prefer using long-term storage such as Thanos or Cortex (and since recently in March 2022: Mimir). Those solutions provide a very good backup solution as well.
The current jsonnet+Grafonnet solution for generated dashboards is not the final stage of evolution. Tooling like CDK could be adapted so dashboards can be written in a Grafana-agnostic way, using great languages like TypeScript. For now, if you go with jsonnet, I recommend you implement common functions that abstract the Grafana details away and set reasonable defaults everywhere.
Observing only the main business metric(s) is not sufficient. Particularly when you have split into several engineering teams or even have a platform infrastructure / DevOps / SRE team, specific monitoring depending on the teams' respective responsibility makes a lot of sense. In our example business, watching the health of partner or provider companies can make sense, since they may not have the most modern health monitoring in place. For examples, Grafana Labs has acquired k6 which can be used for load tests, but in the future hopefully also to monitor TLS certificate expiry (until that feature exists, Blackbox exporter is a reasonable tool). Try a "pre-mortem" brainstorming session to think of what could go wrong, and you will find many things to monitor which are not covered by the main metrics. Consider also "value under threshold" checks, since an error rate of zero could simply come from zero requests per second, and that can mean a whole service or feature is not working, or customers cannot reach your API.
Recommendations in this article were collected mainly in 2020-2021, before Tempo/tracing, exemplars and k6 were in wide-spread use. All these can prove helpful in combination with metrics-based monitoring.
Training. As mentioned, I think a good solution survives without training, but instead has proper and concise documentation, and the code speaks for itself. There are very few professional training and recommendation videos on the internet around dashboarding, and the available beginner content often showcases "The more metrics/graphs on a dashboard, the better" 😬. I cannot disagree more, so please try my "high-level dashboard + most important business metric" approach first and see if you prefer that, or rather a jungle of messy, unreviewed stuff which fosters a useless and long-winded tooling replacement every 2-3 years. See also the Grafana webinar Getting started with Grafana dashboard design for a gentle introduction which requires less upfront knowledge about Grafana compared to my article. The video however has some examples where dashboards are too crowded for my taste.
Advanced observability features. Grafana and its competitors offer quite interesting features such as anomaly detection (Datadog) (also possible with Prometheus — interesting blog post), error/crash tracking (Sentry) and others. Those deserve a place on dashboards if reasonably applicable to your products.
Auto-deletion of manually authored changes: Remember my term Deleteday from above? Make sure to implement that. At best, your deployment from CI simply takes care to replace all existing dashboards, including those not created by code. Think of the deployment like rsync -a --delete committed-dashboards production-grafana-instance.

Andreas Sommer – software and health engineer – Blog

Connecting Intermittent Explosive Disorder with diet and histamine intolerance: no more uncontrollable anger and rage

Table of contents

My story

Development of the disease

Detecting IED in yourself

Medication and optionally therapy can be a good first step

Measuring symptoms to find the cause

Histamine intolerance (HIT) is the cause, omitting histamine-rich food items the solution

What comes after the anger is solved?

Related thoughts and reading

How I mostly fixed my migraine, weather and sports-induced headaches

Table of contents

Migraine started in childhood

Science principles ‒ why even write about this single-person case?

Failed attempts

The solution for my migraine: salt at the right amount

Safety of the salt intervention

Scientific theory of electrolyte imbalance behind migraine

The end of the fight after 37 years of pain

Related reading

Grafana dashboards — best practices and dashboards-as-code

Table of contents

Goals for all your dashboards

Spinning up a Grafana playground in a minute

Getting started with a playground dashboard

High-level dashboard creation guidelines

Choose main input for the high-level dashboard

Metric naming and cardinality

Stat instead of Graph for human understanding within milliseconds

Keep panels small on screen

Avoid dropdowns for variable values

Dashboards-as-code and GitOps (committed dashboards)

Dashboard generation from jsonnet code using the grafonnet library

Fast develop-deploy-view cycle

Automatic provisioning of generated dashboards into Grafana instance

jsonnet for dashboards — some tips

Distinguish environments

Consider jsonnet as full programming language

Grafana-specific dashboard tips

Categorize, do not make a dashboard mess

Grouping within a dashboard

Make dashboards editable

Tooltip sort order

Y axis display range

Link to detailed dashboards, logs, other observability tools

Use variables for repetitive values

Clearly differentiate environments

Grid positioning

Choose the right data unit

UTC timezone everywhere

Do not rely on default data source

Heatmaps

Display interesting events as annotations

Self-describing visualization titles

Observability tips not specific to Grafana

Stay consistent in naming metrics and labels

No need to create a metric for everything / how to easily get started monitoring an uninstrumented application

Show only offenders or top N problematic items

Mind test and synthetic traffic

Daytime vs. nighttime

Do not use rate(…​) alone

Prefer counters over gauges

Make observed components distinguishable

Summary

Out of scope

Related reading

Do not rely on `default` data source

Do not use `rate(…)` alone