Installing Prometheus and Grafana on a Raspberry Pi

Recently I've gone through something of a revelation. I'm a fan of the Raspberry Pi (duh!) and I'm a fan of connecting them and presenting the data from sensors. I've used the experience that I have with d3.js to present collected data and even written a standalone monitoring system that I really enjoyed.

However, I was intoduced to Prometheus and Grafana recently and pretty early on in the process I could see that I was re-inventing the wheel for monitoring. The combination of the two of them created a very polished experience. As a result, I dove in as I will normall do and wrote a book (you can download it for free by the way) outlining the process for setting up a Raspberry Pi and installing / configuring Prometheus and Grafana.

Below you can find the steps that I took.

About Prometheus

Prometheus is an open source application used for monitoring and alerting. It records real-time metrics in a time series database built using a HTTP pull model.

It is licensed under the Apache 2 License, with source code available on GitHub.

It was was created because of the need to monitor multiple microservices that might be running in a system. It employs a modular architecture and employs modules called exporters, which allow the capture of metrics from a range of platforms, IT hardware and software. Prometheus is written with easily distributed binaries which allow it to run standalone with no external dependencies.

Prometheus’s ‘pull model’ of metrics gathering means that it will actively request information for recording. The alternative (and both have strengths) is a ‘push model’ which occurs when information is pushed to a recording service without being asked. InfluxDB is a popular piece of software that employs that model.

Prometheus collects metrics at regular intervals and stores them locally. These metrics are pulled from nodes that run ‘exporters’. An exporter can be defined as a module that extracts information and translates it into the Prometheus format.

Prometheus has an alert manager that can notify a follow on end point if something is awry and it has a visualisation component that is useful for testing. However, it is commonly used in combination with the Grafana platform which has a very powerful visualisation capability.

Prometheus data is stored as metrics, with each having a name that is used for referencing and querying. This is what makes it very good at recording time series data. To add dimensionality, each metric can be drilled down by an arbitrary number of key=value pairs (labels). Labels can include information on the data source and other application-specific information.

About Grafana

Grafana is an open-source, general purpose dashboard and visualisation tool, which runs as a web application. It supports a range of data inputs such as InfluxDB or Prometheus.

It allows you to visualize and alert on your metrics as well as allowing for the creation of dynamic & reusable dashboards.

Grafana is open source and covered by the Apache 2.0 license and its source code is available on GitHub.

Three of the primary strengths of Grafana are;

1. A powerful engine for the building of dashboards that can contain a wide range of different visualisation techniques.
2. The ability to display dynamic data from multiple sources in a way that allows for multi-dimensional integration.
3. An alerting engine that provides the ability to attach rules to dashboard panels. These rules provide the facility to trigger alerts and notifications.

Imported Dashboard

Installation

While the Raspberry Pi comes with a range of software already installed on the Raspbian distribution (even the Lite version) we will need to download and install Prometheus and Grafana separately

If you’re sneakily starting reading from this point, make sure that you update and upgrade Raspbian before continuing.

sudo apt-get update
sudo apt-get upgrade

Installing Prometheus

The first thing that we will want to do before installing Prometheus is to determine what the latest version is. To do this browse to the download page here - https://prometheus.io/download/. There are a number of different software packages available, but it’s important before looking at any of them that we select the correct architecture for our Raspberry Pi. As the Pi 4 (which I’m using for this install) uses a CPU based on the Arm v7 architecture, use the drop-down box to show armv7 options.

Note the name or copy the URL for the file that is presented. On the 6th of January, the version that was available was 2.15.2. The full URL was something like;

https://github.com/prometheus/prometheus/releases/download/v2.15.2/prometheus-2.15.2.linux-armv7.tar.gz. 

We can see that ‘armv7’ is even in the name. That’s a great way to confirm that we’re on the right track.

On our Pi we will start the process in the pi users home directory (/home/pi/). We will initiate the download process with the wget command as follows ;

wget https://github.com/prometheus/prometheus/releases/download/v2.15.2/prometheus-2.15.2.linux-armv7.tar.gz

The file that is downloaded is compressed so once the download is finished we will want to expand our file. For this we use the tar command;

tar xfz prometheus-2.15.2.linux-armv7.tar.gz   

Let’s do some housekeeping and remove the original compressed file with the rm (remove) command;

rm prometheus-2.15.2.linux-armv7.tar.gz   

We now have a directory called prometheus-2.15.2.linux-armv7. While that’s nice and descriptive, for the purposes of simplicity it will be easier to deal with a directory with a simpler name. We will therefore use the mv (move) command to rename the directory to just ‘prometheus’ thusly;

mv prometheus-2.15.2.linux-armv7/ prometheus/   

Believe it or not, that is as hard as the installation gets. Everything from here is configuration in one form or another. However, the first part of that involves making sure that Prometheus starts up simply at boot. We will do this by setting it up as a service so that it can be easily managed and started.

The first step in this process is to create a service file which we will call prometheus.service. We will have this in the /etc/systemd/system/ directory.

sudo nano /etc/systemd/system/prometheus.service   

Paste the following text into the file and save and exit.

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=pi
Restart=on-failure

#Change this line if Prometheus is somewhere different
ExecStart=/home/pi/prometheus/prometheus \
  --config.file=/home/pi/prometheus/prometheus.yml \
  --storage.tsdb.path=/home/pi/prometheus/data

[Install]
WantedBy=multi-user.target

The service file can contain a wide range of configuration information and in our case there are only a few details. The most interesting being the ‘ExecStart’ details which describe where to find the prometheus executable and what options it should use when starting. In particular we should note the location of the prometheus.yml file which we will use in the future when adding things to monitor to Prometheus.

Before starting our new service we will need to reload the systemd manager configuration. This essentially takes changed configurations from our file system and makes them ready to be used. We have added a service, so systemd needs to know about it before it can start it.

sudo systemctl daemon-reload   

Now we can start the Prometheus service. .

sudo systemctl start prometheus 

You shouldn’t see any indication at the terminal that things have gone well, so it’s a good idea to check Prometheus’s status as follows;

sudo systemctl status prometheus 

We should see a report back that indicates (amongst other things) that Prometheus is active and running.

Now we will enable it to start on boot.

sudo systemctl enable prometheus 

To check that this is all working well we can use a browser to verify that Prometheus is serving metrics about itself by navigating to its own metrics endpoint at http://10.1.1.110:9090/metrics (or at least at the IP address of your installation).

A long list of information should be presented in the browser that will look a little like the following;

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.4297e-05
go_gc_duration_seconds{quantile="0.25"} 5.476e-05
go_gc_duration_seconds{quantile="0.5"} 0.000157445
go_gc_duration_seconds{quantile="0.75"} 0.000249149
go_gc_duration_seconds{quantile="1"} 0.000375409
go_gc_duration_seconds_sum 0.001187673
go_gc_duration_seconds_count 7
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 47
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.5"} 1

We can now go to a browser and enter the IP address of our installation with the port :9090 to confirm that Prometheus is operating. In our case that’s http://10.1.1.110:9090.

Grafana login

If we now go to the ‘Status’ drop down menu and select ‘Targets’ we can see the list of targets that Prometheus is currently scraping metrics from.

Grafana login

From the information above, we can see that it is already including itself as a metric monitoring point!

There’s still plenty more to do on configuring Prometheus (specifically setting up exporters), but for the mean time we will leave the process here and set up Grafana.

Installing Grafana

In much the same way that we installed Prometheus, the first thing we need to do is to find the right version of Grafana to download. To do this browse to the download page here - https://grafana.com/grafana/download?platform=arm. There are a number of different ways to install Grafana. We are going to opt for the simple method of installing a standalone binary in much the same way that we installed Prometheus.

The download page noted above goes straight to the ARM download page. We will be looking for the ‘Standalone Linux Binaries’ for ARMv7. Note the name or copy the URL for the file that is presented. On the 24th of January, the version that was available was 6.5.3. The full URL was something like https://dl.grafana.com/oss/release/grafana-6.5.3.linux-armv7.tar.gz;

On our Pi we will start the process in the pi users home directory (/home/pi/). We will initiate the download process with the wget command as follows (the command below will break across lines, don’t just copy - paste it);

wget https://dl.grafana.com/oss/release/grafana-6.5.3.linux-armv7.tar.gz

The file that is downloaded is compressed so once the download is finished we will want to expand our file. For this we use the tar command;

tar xfz grafana-6.5.3.linux-armv7.tar.gz   

Housekeeping time again. Remove the original compressed file with the rm (remove) command;

rm grafana-6.5.3.linux-armv7.tar.gz   

We now have a directory called grafana-6.5.3. While that’s nice and descriptive, for the purposes of simplicity it will be easier to deal with a directory with a simpler name. We will therefore use the mv (move) command to rename the directory to just ‘grafana’ thusly;

mv grafana-6.5.3/ grafana/   

Again, now we need to make sure that Grafana starts up simply at boot. We will do this by setting it up as a service so that it can be easily managed and started.

The first step in this process is to create a service file which we will call grafana.service. We will have this in the /etc/systemd/system/ directory.

sudo nano /etc/systemd/system/grafana.service   

Paste the following text into the file and save and exit.

[Unit]
Description=Grafana Server
After=network.target

[Service]
Type=simple
User=pi
ExecStart=/home/pi/grafana/bin/grafana-server
WorkingDirectory=/home/pi/grafana/
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

The service file can contain a wide range of configuration information and in our case there are only a few details. The most interesting being the ‘ExecStart’ details which describe where to find the Grafana executable.

Before starting our new service we will need to reload the systemd manager configuration again.

sudo systemctl daemon-reload   

Now we can start the Grafana service.

sudo systemctl start grafana 

You shouldn’t see any indication at the terminal that things have gone well, so it’s a good idea to check Grafana’s status as follows;

sudo systemctl status grafana 

We should see a report back that indicates (amongst other things) that Grafana is active and running.

Now we will enable it to start on boot.

sudo systemctl enable grafana 

At this point we have Grafana installed and configured to start on boot. Let’s start exploring!

Using a browser, connect to the Grafana server: http://10.1.1.110:3000.

Grafana login

The account and password are: admin/admin. Grafana will ask you to change this password.

The first configuration to be made will be to create a data source that Grafana will use to collect metrics. Of course, this will be our Prometheus instance

Go to the configuration menu (the little gear icon) and select ‘Data Sources’.

Data Sources Menu

The select ‘Add Data Source’ and then select Prometheus as that data source.

Add Data Source

Now we get to configure some of the settings for our connection to Prometheus.

In this case we can set the URL as ‘http://localhost:9090’ (since both Prometheus and Grafana are installed on the same server), the scrape interval as ’15s’, the Query timeout as 60s and the http method as ‘GET’.

Then click on the ‘Save & Test button’.

Save & Test

We should get some nice ticks to indicate that the data source is working.

Data Sources Tick

Great!

Now things start to get just a little bit exciting. Remember the metrics that were being sent out by Prometheus? Those were the metrics that report back on how Prometheus is operating. In other words, it’s the monitoring system being monitored. We are going to use that to show our first dashboard.

Here lies another strength of Grafana. Dashboards can be built and shared by users so that everyone benefits. We will import one such dashboard to show us how Prometheus is operating.

At the top of our ‘Data Sources / Prometheus’ settings screen there is a ‘Dashboards tab. Click on that to show some available dashboards for our Prometheus instance;

Import Prometheus Dashboard

Click on the ‘Import’ button for the ‘Prometheus 2.0 Stats’ dashboard. We should recieve a little green tick notification telling us that it has been successful. Now click on the ‘Prometheus 2.0 Stats’ label itself and the dashboard will open.

First Dashboard

How about that?

We should now be looking at a range of metrics that Grafana is scraping from Prometheus and presenting in a nice fashion.

Trust me, this is just the start of our journey. As simple as the process of getting up and running and looking at a pretty graph is, there are plenty more pleasant surprises to come as we look at the flexibility and power of these two services.