The following are steps that I work through when upgrading Prometheus on my Raspberry Pi based system. It was installed as a standalone binary, so there is a little more to the process. The information is an excerpt from the book 'Raspberry Pi Computing: Monitoring with Prometheus and Grafana'. You can download the full book for free or read online if you wish. There are no catches :-).
Upgrading Prometheus is something that we should do as new versions with new features become available. Have installed a system by downloading and running it as a standalone binary, the simple method such as using the
apt-get track won’t work for us.
However, that doesn’t mean that it’s a difficult task. In fact, it’s blissfully easy.
We can make the process fairly straight forward and painless by installing the new version alongside our older version and then just copying over the configuration and database.
What we will do is;
- Download and decompress our new version
- Copy the configuration and data from our old version to our new version.
- Stop Prometheus and Grafana
- Run our new version of Prometheus manually (not as a service) and test it.
- Stop the newer version of Prometheus
- Change the directory name of our old and new versions
- Start the Prometheus and Grafana services.
Download
In much the same way that we installed Prometheus the first time, the first thing we need to do is to find the right version to download. To do this browse to the download page here - https://prometheus.io/download/. Select the architecture as ‘armv7’ (assuming that we are installing on a Pi 2,3 or 4)
Note the name or copy the URL for the Prometheus file that is presented. On the 10th of May 2020, the version that was available was 2.18.1. The full URL was something like;
Note that we can see that ‘armv7’ is in the name. That’s a great way to confirm that we’re on the right track.
Just for reference, the previous version that we are upgrading from is 2.17.1
On our Pi we will start the process in the pi users home directory (
/home/pi/). We will initiate the download process with the wget command as follows;
The file that is downloaded is compressed so once the download is finished we will want to expand our file;
Remove the original compressed file with the
rm (remove) command;
We now have a directory called
prometheus-2.18.1.linux-armv7. During a new installation we would have renamed this with the mv (move) command to change the directory name to just ‘prometheus’. However, in this case we will work with our new version in this default folder till we’ve tested that it works correctly. This way we can back out of the upgrade if (for whatever reason) it doesn’t go smoothly.Stop the services
Stopping the Prometheus service means that we need to think a bit about implications. While we have the program stopped, there won’t be any data available for the Grafana service. This means that anything that will be affected by an absence of data will get triggered. For example, if we had an alert set up in Grafana to notify when a metric was absent, that alert will get triggered. If we have other users that are relying on the service to be operating we will need to ensure that we discuss the plans with them ahead of time. To remove the possibility of Grafana thinking that something horribly wrong has happened, we will stop the Grafana service as well.
Stopping both of the services is nice and easy;
Copy the configuration and data
The two things that we will want to copy over from our old installation are our configuration file
prometheus.yml and our collected data.
Since we haven’t started our new version of Prometheus yet, it won’t have a data folder, so we can just copy that straight into the appropriate place.
Then copy in our configuration file from our current Prometheus instance
Run the new version manually and test
We can now run Prometheus manually. We will need to ;
To test that it is working correctly, we can open our browser to the Prometheus web page and check that all the things that are in there seem good. Once we are happy we can move on.
Stop the newer version
In your terminal use ‘ctrl-c’ to stop the running manual instance of
prometheus.
We should also change back to the home directory.
Change the directory names
Now that we’re happy that everything is in order we can change the name of our older
Prometheus instance to reflect its version number (in other words, we won’t get rid of it just yet, because it’s always prudent to keep things about just in case.
And now we can rename the directory of our new version of Prometheus as simply
prometheus.Start the services.
With everything in it’s proper place we can restart the services again and they will automatically start our new version of Prometheus.
Just as a final check, we should go back to our browser and go over the system (both Prometheus and Grafana) to confirm that everything is good.
If you have any users of the system you can advise them that everything is operating well again.
No comments:
Post a Comment