Real Life Custom Agents – Linux Hardware Monitoring
(Part 1)
By Pontus Rydin, Sr. Solution Architect
Introduction
You have probably heard people from Quest mention Custom Agents as means of solving various data integration issues. While this all sounds really great in theory, how does it work in a real-life situation?
In this article, I’m using the Linux lm_sensors hardware monitoring framework as a candidate for integration with Foglight. The lm_sensors framework is a powerful Linux utility that lets you harvest data from the various sensors present on most modern motherboards. Typical metrics obtained from lm_sensors include voltages, fan speeds and temperatures.
It’s assumed that readers of this article have some familiarity with regular expressions and scripting in perl.
About lm_sensors
The lm_sensors framework is available for download from http://www.lm-sensors.org/ and works with all “modern” Linux kernels (including 2.4.x). The actual metrics available depend on what chipset is installed on your motherboard.
There are three major components to the lm_sensors framework:
1. Drivers: Low-level code for interfacing with the actual sensors and control chips.
2. API: A C-style API for reading and manipulating sensor data
3. Command line utilities: Simple utilities for reading sensor data
Since I didn’t feel like cranking up my old C-compiler, I opted for the command line version and some perl to parse and convert the result.
Installing lm_sensors
Since the way you install lm_sensors varies depending on what Linux distribution you’re running, the installation part is mostly left as an exercise to the reader. However, it typically consists of the following steps:
1. Get and install the lm_sensors 3.10 package by downloading the RPM from http://www.lm-sensors.org/ or using the package manager of you Linux distribution.
2. Auto-configuring lm_sensors by running the sensors-detect command as root.
3. Test you installation by running the sensors command. On my system, the output looks like below. Your output should look similar.
w83627ehf-isa-0290
Adapter: ISA adapter
Vcore: +1.07 V (min = +0.00 V, max = +1.74 V)
in1: +1.74 V (min = +1.49 V, max = +0.44 V) ALARM
AVCC: +3.34 V (min = +3.09 V, max = +1.44 V) ALARM
VCC: +3.31 V (min = +0.22 V, max = +1.09 V) ALARM
in4: +1.70 V (min = +0.85 V, max = +0.08 V) ALARM
in5: +2.04 V (min = +1.58 V, max = +0.77 V) ALARM
in6: +1.80 V (min = +0.39 V, max = +0.06 V) ALARM
3VSB: +3.38 V (min = +2.82 V, max = +0.45 V) ALARM
Vbat: +0.00 V (min = +3.12 V, max = +1.39 V) ALARM
in9: +0.00 V (min = +0.18 V, max = +0.67 V) ALARM
fan1: 0 RPM (min = 10546 RPM, div = 128) ALARM
fan2: 1004 RPM (min = 405 RPM, div = 16)
fan3: 0 RPM (min = 10546 RPM, div = 128) ALARM
fan4: 0 RPM (min = 1171 RPM, div = 128) ALARM
fan5: 0 RPM (min = 2636 RPM, div = 128) ALARM
temp1: +50.0°C (high = +97.0°C, hyst = +9.0°C)
temp2: +33.5°C (high = +80.0°C, hyst = +75.0°C)
temp3: -14.0°C (high = +80.0°C, hyst = +75.0°C)
About custom agents
This article isn’t intended to be a full tutorial or reference guide on custom agents, but here are some basic concepts:
A “scripted” custom agent is simply a script (or any kind of program, actually) that writes metrics to stdout in a predefined format. That’s really all you need! Foglight is then responsible for creating the appropriate tables, data types and do all the necessary data transformation. This all takes place behind the scenes and you, as the author of the agent, don’t have worry about it.
Custom agent data format
The output of a custom agent looks something like this:
TABLE Processor
START_SAMPLE_PERIOD
RowId.String.id = MainProcessor
utilization:percent = 42
END_SAMPLE_PERIOD
END_TABLE
The TABLE tag marks the start of a table, i.e. data container in Foglight that is very similar to a SQL-table. Processor is the name of the table.
The next tag, START_SAMPLE_PERIOD marks the beginning of a sample, i.e. a single metric or a set of metrics that represent the state at a certain point in time.
Now it’s time for the actual data. Since a system might have several processors, we need to name this particular instance of Processor (or in SQL-lingo: this particular row in the Processor table). This is very similar to a primary key in a SQL-table. For the same agent, there can only exist a single Processor with the same RowId.
Once we’ve established the identity, it’s time to send the actual metrics. A metric has a name (“utilization” in this case), a unit (“percent” in our example) and a value (42 in this case).
The next two tags just mark the end of the sample and of the table and we’re done!
Connecting lm_sensors to Foglight
Now we know enough about lm_sensors and Foglight Custom agents to go ahead and build the glue between the two. Our challenge is how to transform the output from the sensors command into the Foglight custom agent format described above. This looks like a job for perl!
We could just take the output of sensors I showed you above and parse it and transform it to the Foglight agent format, but if would be nice if we could get out data in a less chatty and easier to parse format. Luckily, the sensors command has a “-u” switch that causes it to print its output in a “raw” format (truncated to save space):
[root@vivaldi scripts]# sensors -u
w83627ehf-isa-0290
Adapter: ISA adapter
Vcore:
in0_input: 1.07
in0_min: 0.00
in0_max: 1.74
in0_alarm: 0.00
fan2:
fan2_input: 1004.00
fan2_min: 405.00
fan2_alarm: 0.00
fan2_div: 16.00
temp3:
temp3_input: 35.50
temp3_max: 80.00
temp3_max_hyst: 5.00
temp3_alarm: 0.00
This looks like something we can parse quite easily!
There seems to be a name of the sensor, followed by some readings. The sensors command is even nice enough to give us a hint as to what type of sensor we’re looking at: If the name of the value starts with “in” it’s a voltage, if it starts with “fan” its’a fan and if it starts with “temp” it’s a temperature.
So I guess the first thing our script needs to do is to match the “temp1:” header and let that indicate the start of a table. This is how we can match that:
/^([\w\d]+):$/
In other words: Match any string that’s the first and last item on a line and that’s a series of alpha numeric characters followed by a colon.
Next step is to match the metrics:
/^\s*in\d+_input:\s*(\d+.\d+)/
This one matches a voltage. It’s basically some white space, followed by the string “in”, followed by some numbers, a colon and a numeric value. We can also use this expression to isolate the value. As a bonus, since we know it matched “in”, we also know the table type, which we’ll be calling “Voltage” in this case. Temperatures and fan speeds are isolated the same way, but with slightly different matching rules.
We could actually stop ere if we wanted, since we’ve successfully found and isolated the metric. However, lm_sensors also keeps track of some threshold values and an alarm flag. The alarm flag is set if the metric crosses a threshold. We could, of course, implement the logic that checks the value against the threshold as a rule in Foglight, but since lm_sensors have some neat functionality for e.g. hysteresis, we grab the alarm flag as well. We’ll use that later to fire alarms. The min and max thresholds, as well as the alarm flags are captured using these regular expressions:
/^\s*[\w\d]*_min:\s*(\d+.\d+)/
/^\s*[\w\d]*_max:\s*(\d+.\d+)/
/^\s*[\w\d]*_alarm:\s*(\d+.\d+)/
And that’s it as far as matching goes. Now all that’s left to do is printing the output and some simple housekeeping. Here is the finished script:
#!/usr/bin/perl
use strict;
sub printTableHeader
{
my ($tableName, $sensor) = @_;
print "TABLE $tableName\nSTART_SAMPLE_PERIOD\nRowId.String.id = $sensor\n";
}
open(INPUT, "/usr/local/bin/sensors -u |") || die;
my $sensor;
my $value;
my $alarm_max = 9E99;
my $alarm_min = -9E99;
my $value = 0;
my $alarm = 0;
my $validRecord = 0;
while(<INPUT>)
{
chomp;
my $line = $_;
if($line =~ /^([\w\d]+):$/)
{
# Found new sensor
#
$sensor = $1;
}
elsif($line =~ /^\s*in\d+_input:\s*(\d+.\d+)/)
{
$value = $1;
printTableHeader("Voltage", $sensor);
$validRecord = 1;
}
elsif($line =~ /^\s*temp\d+_input:\s*(\d+.\d+)/)
{
$value = $1;
printTableHeader("Temperature", $sensor);
$validRecord = 1;
}
elsif($line =~ /^\s*fan\d+_input:\s*(\d+.\d+)/)
{
$value = $1;
printTableHeader("Fan", $sensor);
$validRecord = 1;
}
elsif($line =~ /^\s*[\w\d]*_min:\s*(\d+.\d+)/)
{
$alarm_min = $1;
}
elsif($line =~ /^\s*[\w\d]*_max:\s*(\d+.\d+)/)
{
$alarm_max = $1;
}
elsif($line =~ /^\s*[\w\d]*_alarm:\s*(\d+.\d+)/)
{
$alarm = $1;
# Alarm is also the last item we care about, so now it's time
# to dump the whole sample!
#
if($validRecord)
{
print("reading:count = $value\n");
print("alarm_min:count = $alarm_min\n");
print("alarm_max:count = $alarm_max\n");
print("alarm:count = $alarm\n");
print("END_SAMPLE_PERIOD\nEND_TABLE\n");
$validRecord = 0;
}
}
}
close(INPUT);
Installing the script
As you could see from the example above, writing the agent itself is pretty straightforward. All you have to do is some basic string matching and reformatting.
But how do we take this simple script and turn it into a Foglight agent?
All we have to do is to upload the script to Foglight and tell it to turn the script into an agent. Let’s asswume you’ve put the script in a file called “sensors.pl”. You can upload this by selecting “Administration->Tooling->Build script agent”. Simply browse to your script file (in this case the perl script).
Once you’ve done this, the agent is entered into Foglight’s repository and can be pushed out to the monitored hosts just like any other agent. You may do that by going to “Administration->Agents->Agent Status” and clicking “Deploy agent package” and selecting the “sensors” agent. Once the agent package is deployed, you can create an instance of it by clicking “Create Agent”. In other words: Just like you would be doing with any of the out-of-the-box agents. As a last step, activate the agent by clicking “Activate”.
Using the data from the agent
Once the agent is pushed out and activated, we can use the data for anything from custom dashboards, to rules and derived metrics. Just to make sure things work properly, let’s take some metrics and drag them onto a custom dashboard!
Caveats
· Not all systems will work with lm_sensors, as some chipsets use proprietary specifications that aren’t available to the developers of lm_sensors. If sensors-detect doesn’t find any supported hardware, your system may not be supported. Before giving up, make sure you search google for lm_sensors and the model number of your motherboard. There are some third party sources for drivers that are worth investigating.
· This script has been tested with lm_sensors 3.10 under CentOS 5.2 on a system based on an Asus P5Q Pro motherboard. Your sensor data will probably look different from what I got on my system.
· You may need to change the first line of your script to point to the location where your perl environment is installed.
Next steps
So far, we’ve built a very useful custom agent, just by parsing some output from the sensors command. I’m sure this gives you a ton of other ideas on Linux commands you want to wrap as custom agents. Feel free to experiment and post any questions you have on foglight.org.
In my next article, we will continue to refine the lm_sensors by adding some rules and more sophisticated dashboards.