Proxmox itself has a very good interface to monitor all kinds of resources like virtual machines, containers and storages. Because I use prometheus and grafana for other services, I decided to also use it for proxmox. Let's dive right into the installation.

Install prometheus #

Step 1: Create a user #

First of all we create a dedicated user for prometheus. This way we have a better encapsulation of the services.

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

This user does not need any shell, that's why we added the -s /sbin/nologin flag.

Step 2: Create directories for data and configuration #

Next, we create the required folders for Prometheus in order to store data and configuration files.

sudo mkdir /var/lib/prometheus
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done

Step 3: Download and install prometheus #

Now we are ready to download the latest release of Prometheus directly from Github.

mkdir -p /tmp/prometheus && cd /tmp/prometheus
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
  | grep browser_download_url \
  | grep linux-amd64 \
  | cut -d '"' -f 4 \
  | wget -qi -

We have to extract the file and move all files to the correct directory.

tar xvf prometheus*.tar.gz
cd prometheus*/
mv prometheus promtool /usr/local/bin/
mv prometheus.yml  /etc/prometheus/prometheus.yml
mv consoles/ console_libraries/ /etc/prometheus/

Cleanup the /tmp directory.

cd ~/
rm -rf /tmp/prometheus

Step 4: Create a systemd configuration file #

We want to manage Prometheus with systemd. Therefore we have to create a config file using the following command.

sudo tee /etc/systemd/system/prometheus.service<<EOF

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Once you did this, set the correct file permissions and start Prometheus.

for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

This is all we have to do in order to get Prometheus up and running. In the next step we install and configure the proxmox-pve-exporter.

Install proxmox-pve-exporter #

As the name already indicates, the proxmox-pve-exporter scrapes data from the proxmox api and provides it to prometheus.

Step 1: Install proxmox-pve-exporter #

If you haven't it already installed, install python and pip using the following command.

sudo apt install python python-pip

Once this command has finished we are ready to install the proxmox-pve-exporter via pip.

sudo pip install prometheus-pve-exporter

Step 2: Create an authentication file #

In order for proxmox-pve-exporter to connect to the Proxmox api we need to create a file with the credentials.

sudo vim /etc/prometheus/pve.yml

Paste the following lines into the file and be sure to replace the credentials with yours.

default:
    user: user@pve
    password: your_password_here
    verify_ssl: false

Step 3: Create a systemd configuration file #

Like in the first chapter we need to create a systemd config file to control proxmox-pve-exporter.

sudo tee /etc/systemd/system/prometheus-pve-exporter.service<<EOF
[Unit]
Description=Prometheus exporter for Proxmox VE
Documentation=https://github.com/znerol/prometheus-pve-exporter

[Service]
Restart=always
User=prometheus
ExecStart=/usr/local/bin/pve_exporter /etc/prometheus/pve.yml

[Install]
WantedBy=multi-user.target
EOF

Step 4: Add proxmox-pve-exporter to prometheus #

Now we have to add proxmox-pve-exporter to prometheus.

sudo vim /etc/prometheus/prometheus.yml

- job_name: 'proxmox'
  metrics_path: /pve
  static_configs:
  - targets: ['localhost:9221']

Save the file and restart prometheus.

sudo systemctl restart prometheus

Install grafana #

In all previous steps we started collecting data from our system. Now we need to install grafana to get a beautiful overview of all the data.

Step 1: Install Grafana #

First of all, add the following line to /etc/apt/sources.list.d/grafana.list

deb https://packages.grafana.com/oss/deb stable main

Now we have to install the proper key.

curl https://packages.grafana.com/gpg.key | sudo apt-key add - 

Update the apt index and install grafana.

sudo apt update && sudo apt install -y apt-transport-https grafana

Once grafana was installed, enable it and check if it is running.

sudo systemctl enable --now grafana-server

systemctl status grafana-server.service

Step 2: Test the installation #

Grafana was successfully installed. Open up your browser and navigate to http://ipaddress:3000 and login with the default credentials.

username: admin
password: admin

Step 3: Add the Prometheus data source #

After logging into grafana, click on "Add data source".

Select "Prometheus" as template.

When adding the data source, be sure to select the URL from the dropdown. Click on "Save & Test".

Next, navigate to the sidebar select "Create" -> "Import".

In this step, Grafana asks you for a dashboard. Luckily, Pietro Saccardi already created a nice dashboard that we can use to display our data. Paste in the dashboard id (10347) and click on "Load".

In the last step, select "Prometheus" from the dropdown menu and click on "Import".

And this is what our new Grafana dashboard look like! Isn't it nice?

Security #

One thing you may noticed while reading this blog post is the following. We have a couple of endpoints that are available for everyone using a webbrowser. Although these endpoints do not leak highly sensitive information, I encourage you to block these ports for the rest of the world using a firewall like ufw.

Conclusion #

With not too much effort we were able to install a beautiful monitoring dashboard using Prometheus, proxmox-pve-exporter and Grafana. Special thanks goes to Pietro Saccardi, who built this amazing Grafana dashboard (which saves us a lot of work!).

If you have any questions or problems following this post, please feel free to contact me.