One Grafana Dashboard With Multiple Prometheus Datasources

In this article, the following aspects of using Prometheus and Grafana will be demonstrated:

TL;TR

1. One Grafana server presenting data from multiple Prometheus resources.
2. Each dashboard would show only selected Prometheus datasources (not all configured datasources are relevant to all of the dashboards)
3. Present only the relevant data from each datasource according to the dashboard content. (For example in case the dashboard panel present one storage mount which has a different mount requirements per server).
4. Useful dashboards for your needs:
   - Host / VM Resources ( CPU, RAM, Storage and I/O, Network).
   - Docker Containers (Use of resources per container).

In the next article I will present monitoring Jenkins and JVM with Prometheus and Grafana. Stay tuned ;-)
- Jenkins Builds (Total, Successful, Unstable, Failed, Queue etc.).
- Jenkins JVM (Memory and GC stats for performance analysis).

Note:

This solution was implemented according the requirement of my internal project. This may not be the best practice. You may consider solutions like Prometheus Federation that may fit better your needs.

Requirements:

• Basic knowledge of Prometheus, Grafana, Docker and Docker-compose
• Docker and Docker-compose installed

For a quick start I would recommend using the dockprom docker-compose that will setup all the Prometheus and Grafana needs in no time (optional).

dockprom installation:

git clone https://github.com/stefanprodan/dockprom
cd dockprom
ADMIN_USER=admin
ADMIN_PASSWORD=admin
docker-compose up -d

And you are all set to go with Prometheus and Grafana up in the air with neat Docker Host and Docker Containers dashboards up and running.

http://localhost:3000

However the dashboards shows only the data from the current server.
Our goal now it to present in one dashboard the content of all the relevant servers from one centralised Grafana.

1. Use one Grafana server to present data from multiple Prometheus resources.

Default Docker Container dashboard:

A datasource is the server from where the Grafana collects the metrics from, that is the server which we want to monitor.

The default data is taken from the local machine:

To monitor a remote server, install Prometheus on that server. You can use the example above using the dockprom.

Now add the remote server as a new datasource:

Click “Add data source” (Blue button).

Select Prometheus (first on the list):

Edit:

• Name: Whatever
• URL: Server (DNS or IP) and port 9090
• Basic auth: On
• With Credentials: On
• User: admin
• Password: admin (Keep the default, unless you know what you are doing).
• Click “Save & Test”. If you have issues, check iptables / firewall settings.

The new datasource will be added to the list:

Now we will add a datasource dropdown to switch between the servers on the dashboard.

Click on the settings button on the top left.

On the Variables tab click “Add variable”.

Add:

• General > Name: datasource
• General > Type: Datasource
• Data source options > Type: Prometheus

Click “Add”

You will see the added datasource dropdown:

I have added a few more datasources for different types:

2. Each dashboard would show only selected Prometheus datasources.

Now that multiple datasources were accumulated, they will all appear in each dashboard.

What if you want each dashboard to filter only the relevant datasources?

For example for the Docker dashboard only show the docker servers? Or for the Jenkins dashboard show only the Jenkins datasources?

Click on the settings button > Variables as seen above. You will see the Datasource variable we created before. Click on prometheus:

So since we are dealing with the Docker dashboard, we want to filter all Datasources that include “Docker” in their name:

In the filter add /Docker/ as seen below. Click “Update” and the “Save” button (top left).

Don’t forget to click the “Save” ;-)

Now if you want to add more types of datasources, that you could use the regex.

In this example, we want to add the Demo and Jenkins datasources since they are containerized.

So we will update the filter to /Docker|Demo|Jenkins/ as below:

You will now see in the Datasource list only the filtered datasources:

Ok, so now we have a list with only the datasources we wanted, but hey, if you switch datasources, you will notice that the data has not changed. In fact you always see only the local machine (well not in all cases, some dashboards you can download from the Grafana Lab are already set properly).

The trick here is that in each panel you should select the datasource you want to monitor.

However in our case we need to select “$Datasource” which will select the datasource according to the selected dropdown in the list we created above.

First click on the panel name and then click “Edit”:

And then select “$datasource” from the dropdown list.

In the case above we see the CPU load of the Docker1 server (datasource).

Now if we switch to the Demo server, we will see the Demo metrics:

Unfortunately this needs to be done with each panel on the dashboard.

Don’t for get to click “Save”!

After updating all the panels, you have one dashboard with multiple datasources:

Here is the Docker1 server metrics:

And here are the Demo server metrics:

Now you can switch between all your selected datasources from within one dashboard.

3. Present only the relevant data from each datasource according to the dashboard content.

You may have noticed in the above screenshots that the “Storage Load” panel has no data and present “N/A”:

This occurs on all of the datasources. The reason is that the panel arrives with the configuration of the fstype=”aufs”, while none of the monitored servers use this fstype. In this scenario the root “/” mount point would be monitored.

To see the query click the “Storage Load” panel name and then “Edit” (as above) to get to the metrics:

(node_filesystem_size_bytes{fstype=”aufs”} — node_filesystem_free_bytes{fstype=”aufs”}) / node_filesystem_size_bytes{fstype=”aufs”} * 100

The fstype=”aufs” will be replaced with mountpoint=”/”:

(node_filesystem_size_bytes{mountpoint=”/”} — node_filesystem_free_bytes{mountpoint=”/”}) / node_filesystem_size_bytes{mountpoint=”/”} * 100

Jenkins1 server — root mount point:

For the demonstration, one mount per server will be monitored. The monitored mount is the most critical one.

So we have a panel with the root mount point which is fine for the jenkins1 server, however this will apply to all datasources, which is not necessary the need.

When things get complicated :-/

Here is a scenario of a containerized Nexus server which was added to the Docker dashboard. The Nexus server has 26 mount points, out of which only three relate to Nexus.

Yet, the critical mount point to be monitored is the /usr/local/nexus-data mount point (which is not the root mount) where all the Nexus data is stored.

The query for the /usr/local/nexus-data mount point of the Nexus server after replacing the root mount point “/”:

(node_filesystem_size_bytes{mountpoint=”/usr/local/nexus-data”} — node_filesystem_free_bytes{mountpoint=”/usr/local/nexus-data”}) / node_filesystem_size_bytes{mountpoint=”/usr/local/nexus-data”} * 100

However changing the mount point in the query to fit the Nexus server impacts all other datasources, hence their data will not be available...

Here is the output of the Jenkins1 server after we updated the query to fit the Nexus needs :-/

We need to manipulate the query to fit all datasources. This may look complicated, but it really isn’t if you are well organized.

Querying “/” OR “/usr/local/nexus-data” is not enough. The issue is that Nexus also has a root mount point and which should not be monitored and should be filtered out.

Here is the output with the “OR” statement:

(node_filesystem_size_bytes{mountpoint=”/”} — node_filesystem_avail_bytes {mountpoint=”/”}) / node_filesystem_size_bytes{mountpoint=”/”} * 100 
or
(node_filesystem_size_bytes{mountpoint=”/usr/local/nexus-data”} — node_filesystem_avail_bytes {mountpoint=”/usr/local/nexus-data”}) / node_filesystem_size_bytes{mountpoint=”/usr/local/nexus-data”} * 100

To make it more complicated as you can see, The Nexus docker has also a virtual mount point which makes the mount appears twice.

So we have the following mounts:

1. nexus-data, the mount we want to monitor:
   - mountpoint=”/usr/local/nexus-data”
   - device=”/dev/mapper/vg_data-lv_nexus”
   - fstype=”xfs”
2. root mount, need to filter out:
   - mountpoint=”/”
   - device=”/dev/mapper/cl_nexus-root”
   - fstype=”xfs”
3. root (virtual), need to filter out:
   - mountpoint=”/”
   - device=”rootfs”
   - fstype=”rootfs”

We need to filter out the root mount point (2&3), yet we want to keep the Jenkins1 server root mount point monitored:
- mountpoint=”/”
- device=”/dev/mapper/cl-root”
- fstype=”xfs”

So we need to find the right combination:

(node_filesystem_size_bytes{mountpoint=”/”, fstype!=”rootfs”, device!=”/dev/mapper/cl_nexus-root”} — node_filesystem_avail_bytes {mountpoint=”/”, fstype!=”rootfs”, device!=”/dev/mapper/cl_nexus-root”}) / node_filesystem_size_bytes{mountpoint=”/”, fstype!=”rootfs”, device!=”/dev/mapper/cl_nexus-root”} * 100 
or
(node_filesystem_size_bytes{mountpoint=”/usr/local/nexus-data”} — node_filesystem_avail_bytes {mountpoint=”/usr/local/nexus-data”}) / node_filesystem_size_bytes{mountpoint=”/usr/local/nexus-data”} * 100

Here we monitor the following options:

1. The root mount point “/“ where:
   - device!=”/dev/mapper/cl_nexus-root” (root device of the nexus server)
   - fstype!=”rootfs” (the virtual device)
2. The /usr/local/nexus-data of the Nexus server.

So the first will provide the root mount but eliminate the nexus root mount but will show the jenkins1 server root mount:

The second will provide the nexus-data mount point only.

This was a demonstration of using one panel for multiple datasources.

Here, a specific scenario was represented. With more servers there may be a need to manipulate even more.

You can take the same idea for any object type needed.

4. Useful dashboards for your needs:

Host.json - Host / VM Resources ( CPU, RAM, Storage and I/O, Network).
Docker Containers.json - Docker Containers (Use of resources per container).

To import the dashboards:

1. Download the json files from the links above.

2. Go to the manage menu:

3. Click on import:

4. Click on the “Upload JSON file” icon

5. Enjoy :-)