Makura no Soshi

I still like Zabbix as a simple allround monitoring solution. With its agent and UserParameter configuration it is very flexible and can be used (and abused) in many interesting ways. Here I want to show and compare three different patterns of collecting metrics from a service:

1. simple item fetch
2. fetch and send
3. preprocessing

As an example application I will use Jitsi, because these days everybody is installing Jitsi Meet. After the installation I came across the useful Zabbix template at https://github.com/romantico88/jitsi_videobridge_zabbix_stats. In an early version it used the “simple” approach so I decided to improve it by using a preprocessing setup (cf. https://github.com/mschuett/jitsi_videobridge_zabbix_stats).

Scenario

We have a Jitsi Meet setup. The Jitsi Videobridge (jvb) publishes metrics in JSON format at an endpoint http://localhost:8080/colibri/stats. Now we want to collect these metrics with Zabbix. On the Jitsi server we have a local Zabbix Agent installed to communicate with our central Zabbix Server.

Simple Item Fetch

The simplest data collection mode uses the UserParameter function of the Zabbix Agent (more detailed description in the Zabbix blog).
A UserParameter defines new item keys on the Zabbix Agent with a shell command. Whenever the Zabbix Server requests that item the Agent will run the shell command and return its output.

UserParameter=jvb.stats[*],curl -s http://localhost:8080/colibri/stats | jq '.$1'

Our definition is a dynamic definition, in it the wildcard * and the $1 indicate that we expect a variable parameter. So the server can request item keys jvb.stats[threads] and jvb.stats[videochannels] (with type “Zabbix Agent”), and then the agent will run the shell commands curl -s http://localhost:8080/colibri/stats | jq '.threads' and curl -s http://localhost:8080/colibri/stats | jq '.videochannels' respectively.

In this case the item key jvb.stats is also valid and returns the full JSON text This is more or less accidental because an empty parameter still yields a correct command line (then the command is curl -s http://localhost:8080/colibri/stats | jq '.').

So on the server every metric gets defined as its own item with item type “Zabbix Agent” and its appropriate value type (unsigned, int, float, etc); e.g.

<item>
    <name>Jitsi Videobridge Threads</name>
    <key>jvb.stats[threads]</key>
    <status>0</status>
    <value_type>UNSIGNED</value_type>
    <!-- ... -->
</item>

The linked Zabbix template uses this to query 18 values, so for every measurement interval the server will request 18 values from the agent and the agent will run 18 shell command lines with curl and jq processes.
In this case this is (IMHO) ok, because the HTTP request is rather light-weight and we have only a few items. — But for some applications this introduces significant overhead, one example would be a MySQL server with a heavier client (and probably a login step) and possible several hundred item values.

The Zabbix Web UI shows this item view:

A simplified sequence diagram looks like this:

Fetch and Send

One early way to mitigate the overhead was to use the Zabbix sender utility and combine push and pull items.

In the template configuration all metrics get defined as Zabbix Trapper items, that means the server will not pull them on its own but will passively wait for other components to push them (like with SNMP traps).

<item>
    <name>Jitsi Videobridge Stats Fetch Script</name>
    <key>jvb.stats_collect</key>
    <value_type>TEXT</value_type>
    <description>Fetch and Send Script</description>
</item>
<item>
    <name>Jitsi Videobridge Threads</name>
    <type>TRAP</type>
    <key>jvb.threads</key>
</item>

Zabbix sender can push the values for us. To use it we “only” have to transform the metrics from the given JSON format to a suitable text format with Zabbix item keys. To fill this gap I use a small script of mine: flatten_json.py. This script reformats generic JSON metrics into a kind of dotted key value format as is often used for Zabbix item keys.

Example:

$ echo '{"some": {"nested": {"metric": 42}}}' | ./flatten_json.py --dash
- some.nested.metric 42
$ echo '{"some": {"nested": {"metric": 42}}}' | ./flatten_json.py --dash --prefix my_app
- my_app.some.nested.metric 42

So to recap: we now have a Zabbix server with defined Zabbix trapper items. And we wan build a shell command pipeline on the Jitsi server to fetch the metrics (curl ...), transform them (flatten_json.py ...), and send the to Zabbix (zabbix_sender ...).

Now one option is to put these commands into a cronjob, with central configuration management this is feasible but will decouple the metrics collection from the Zabbix server. The better option is to define one more auxillary item jvb.stats_collect of type Zabbix agent to run the shell command pipeline with this UserParameter definition. I added newlines for readability, Zabbix requires everything in one line; if there are more commands or parameters then it is better to move the shell commands into a script file:

UserParameter=jvb.stats_collect,curl -s http://localhost:8080/colibri/stats \
  | /usr/local/bin/flatten_json.py --dash --prefix jvb \
  | zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -

This way we get the performance benefits of a single curl request on the Jitsi server, but combine it with full control from the Zabbix server: like with other items we can enable/disable the collection and set the update interval.

The Zabbix Web UI shows this item view:

A simplified sequence diagram looks like this:

Preprocessing

Zabbix introduced the preprocessing feature in version 3.4 to improve this use case (more detailed description in the Zabbix blog). The idea is to collect all statistics in a single item value from the agent, and then extract the multiple other item values as dependent items on the server.

With the

UserParameter=jvb.stats_colibri,curl -s http://localhost:8080/colibri/stats

This is not dynamic but only a single static item, its value is the complete JSON output from the /colibri/stats API.

Now all other items are defined as dependend items, which are derived from the jvb.stats_all value and use different preprocessing steps to determine their values. In our case every item (like jvb.stats[threads] and jvb.stats[videochannels]) gets a JSONPath expression to access the right JSON attribute (like $.threads and $.videochannels).

<item>
    <name>Jitsi Videobridge Stats Colibri</name>
    <key>jvb.stats_colibri</key>
    <delay>30s</delay>
    <value_type>TEXT</value_type>
    <description>Full JSON status text</description>
</item>
<item>
    <name>Jitsi Videobridge Threads</name>
    <type>DEPENDENT</type>
    <key>jvb.stats[threads]</key>
    <value_type>UNSIGNED</value_type>
    <preprocessing>
        <step>
            <type>JSONPATH</type>
            <params>$.threads</params>
        </step>
    </preprocessing>
    <master_item>
        <key>jvb.stats_colibri</key>
    </master_item>
</item>

The Zabbix Web UI shows this item view:

A simplified sequence diagram looks like this:

Final Improvement: Remove curl

In all my examples I kept the UserParameter configuration line with a custom key. I found this useful because I often have services that require additional discovery, authentication, or filters in the metrics request.

But for simple HTTP APIs we do not need a UserParameter and shell commands at all.
To query the API from the Zabbix server we can define an item of type HTTP agent.
And to let a Zabbix agent do the HTTP query there is the built-in key web.page.get.

The Jitsi Videobridge stats are only available on the local interface, so for our use case we want the agent based approach using the key web.page.get[localhost,/colibri/stats,8080]. That is the current implementation in the romantico88/jitsi_videobridge_zabbix_stats template.

This entry was posted on Saturday, April 11th, 2020 at 16:11 and is filed under Admin, english, Tools, Verschiedenes. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.