Introduction

The monitoring and alerting feature allows you to receive notifications when monitoring measurements fall below or exceed a specified threshold. For instance, you can receive an email notification if a device’s available disk space runs out.

Monitoring data, also called measurements, are evaluated per client at the moment data arrives at the server. Rule processing is not a periodic job. If your clients send monitoring data once a minute, rules are evaluated once a minute too.

Rule evaluating is limited to the data sent by the client and the last 10 measurement of the past taken from an in-memory cache. That means a rule like “Fire an alert when both, client A and B have high CPU load” is currently not possible.

Please refer to the server sizing documentation regarding the memory required for the measurement cache.

To activate the alerting system, you need to configure three settings:

1. An SMTP server is required.
2. Create a template: Templates define who will receive notifications, the method of notification (e.g., email), and the content of the message (subject and body).
3. Create a rule or rule set: Rule sets determine which measurement is being evaluated, the threshold, and the action to be taken. If the action involves notification, the rule references a template.

To receive your first notification, follow these steps:

Ensure you have a valid SMTP configuration in your rportd.conf file. The config file contains all explanations needed. Also look at the recommendation for an optimal SMTP setup at the end of the document,

Create the template below using the UI or API

{
   "id": "t1",
   "transport": "smtp",
   "subject": "{{.Outcome}} {{.Rule.Severity}} for {{.Rule.ID}} on {{.Client.Name}}",
   "body": "The client {{.Client.Name}} ({{.Client.ID}}) <br>\nhas triggered rule ID: <b>{{.Rule.ID}} </b><br>\nwith severity: {{.Rule.Severity}} <br>\n<small>The client runs kernel {{.Client.OSKernel}}</small>",
   "html": true,
   "recipients": [
         "user@example.com"
   ]
}

The client {{.Client.Name}} ({{.Client.ID}}) <br>
has triggered rule ID: <b>{{.Rule.ID}} </b><br>
with severity: {{.Rule.Severity}} <br>
<small>The client runs kernel {{.Client.OSKernel}}</small>

Create the rule set below using the UI or by submitting it via HTTP PUT to the /api/v1/monitoring/rules endpoint:

{
  "rules": [
    {
      "id": "high-cpu-usage",
      "severity": "warning",
      "expr": "Avg(CPUUsagePercent(3)) > 80.0",
      "actions": [
        {
          "notify": [
            "t1"
          ]
        }
      ]
    }
  ]
}

This rule will trigger the notification template t1 if the average CPU usage of the last three measurements is greater than 80.

Test the rule by executing nohup stress -t 600 -c 1 -q >/dev/null 2>&1 & on a Linux system.

The following rules cover most common use cases:

1. CPU Monitoring: Avg(CPUUsagePercent(X)) > Y If the average of the last X CPU usage (percent) measurements is above Y, the expression becomes true, and actions are triggered.

2. Memory Monitoring: Avg(MemoryUsagePercent(5)) > N If the average of the last X memory usage (percent) measurements is above Y, the expression becomes true, and actions are triggered.

3. Disk Monitoring: Avg(MountPoints("*", "FreeBytes", X)) < (Y * GB) If the average of free bytes on any mount point or drive letter is below Y GB, the expression becomes true, and actions are triggered.

4. Process Monitoring: not Match("X", ProcessCmdLines(Y)) or not Match("X", ProcessNames(Y)) If the list of processes does not contain X during the Y last measurements, the expression fires. X supports wildcards. ProcessCmdLines() matches against the full command line (e.g., /usr/sbin/sshd -D), while ProcessNames() matches against the process name only (e.g., sshd).

The average and process match functions support the evaluation of the last 10 measurements at maximum.

As a built-in default notifications are also sent once a problem has recovered where recovery is defined as the rule expression evaluates to false.

Make sure to have the placeholder {{.Outcome}} in your template. It will be replaced either by ALERTING or RESOLVED.

Because messages, subject and body, are rendered by the go template engine you can further customize them by using an if-else-block.

{{ if eq .Outcome "ALERTING" }} 📣 Heads up {{ else }} 🧘‍♂️ Relax {{ end }}

Please note that only if eq .<VAR> "<COMPARISON VALUE>" works. Using if .<VAR> eq "<COMPARISON VALUE>" will not work resulting in messages not being sent.

Also, the comparison value must be in double quotation marks. Single quotation marks are not supported.

Multiple rule expression can be concatenated with the logical operators and and or (lowercase). Braces can be used inside complex expressions with multiple concatenations.

Example: client.os_kernel == "linux" and Avg(CPUUsagePercent(3)) > 80 This rule will trigger an action only if both conditions are met.

Example: any(client.Tags, # == "Berlin") and ... adds a condition to an expression to only trigger if Berlin is included in the list of tags.

When looking for an item in a list the hash sign # is the internal placeholder for the current item while looping over the list .

While client.connection_state is available for evaluation it’s not suitable to create rules to get a notification about disconnected clients.
On server stop all clients will be properly disconnected. Because it takes some time for clients to connect, on server start the majority of clients will have client.connection_state = disconnected resulting in firing many false notifications.

Monitoring the connection state of client is currently not supported. A proper rule expression for that will be integrated soon.

If you want to activate rules based on time or weekday you can use the following conditions

Date(measurement.Timestamp).Weekday in ["Monday", "Tuesday" ] or Date(measurement.Timestamp).Hour > 9

Example:

any(client.Tags, # == "Berlin") and 
any(measurement.Processes, .CmdLine contains "tetris") and 
Date(measurement.Timestamp).Weekday in ["Monday", "Tuesday", "Wednesday", "Thursday","Friday"] and
Date(measurement.Timestamp).Hour > 9 and
Date(measurement.Timestamp).Hour < 18

This expression will evaluate true if a process tetris is running, but only Monday to Friday between 9am and 6pm. The expression must be entered all on a single line. The example has line breaks just for better readability.

All time data refers to UTC.

When using client metadata in rule expressions, the field must be given camel case with a leading lowercase client as indicated in the table above. Example: client.OSKernel == "linux" and Avg(CPUUsagePercent(3)) > 80
Matching is case-sensitive.

When using client metadata in notification in templates, the field must be given camelcase with a leading uppercase .Client and prefixed by a dot inside curly braces.
Example: {{.Client.OSKernel}}

Besides triggering a notification via a template you can also use the ignore action.

Example: Rule expression: Date(measurement.Timestamp).Weekday in ["Saturday", "Sunday"]
Action: ignore: [*]

Make sure you put this rule as the first rule of the set. The order matters. Now you will not get any notifications during the weekend.
The ignore action specifies a list of rule IDs to be ignored. Because wildcards are supported you can disable all rules based on a condition.

Measurements can be accessed by average functions (see above) or directly using the measurement object.
Example: measurement.CPUUsagePercent > 95 or measurement.MemoryUsagePercent > 50.0

The rport server comes with a built-in SMTP client that dispatches email messages. The SMTP configuration is done through the rportd.conf configuration file. Slow SMTP servers can fill up internal queues leading to higher memory consumption.

It’s highly recommended to install a local SMTP server such as postfix on the rport server box. RPort should hand over messages to the local SMTP server which will then queue it and take care of the asynchronous sending, re-sending and network error handling.

Sending notification via any media like messanger apps, MS Teams or Slack is supported. The documentation will be published shortly.