How it works
The Cartridge_for_Integration_Guide.pdf describes all the necessary steps to enable integration. It is a fairly straightforward process, but there are a couple of tips I thought I would share, that can save you a lot of time. Before you start I highly recommend that you read the guide and get a basic understanding of the flow of data. The following diagram outlines this flow:
Tip 1: Preparation is crucial.
Making sure the environment is ready to go, before you attempt integration is crucial. I usually have a checklist which I go through with the customer prior to installation:
1) In the HP Operations manager system: The “Foglight Management server” must be set up as a ‘managed node’. i.e HP Operations Manager binaries will be deployed and installed on the Foglight Management Server. This includes the all important opcmsg binary which is used by the Foglight integration cartridge.
2) Ask the HP operations manager administrator to configure a “Node Group” where Foglight alerts will show up.
3) Create a Message Group in HP Operations Manager. The Message Group can signify different things to different organizations, but generally is a way for grouping or filtering alerts. The integration cartridge defaults to ‘foglight’ and is a good one for testing.
4) A good test to run before attempting integration is to run the following from a command line on the Foglight Management Server:
opcmsg.exe appl=Foglight msg_grp=Foglight severity=critical node=FMS obj=TEST msg_test=”This is a test"
5) This is what is ultimately run by the perl integration script, so make sure the alert makes it all the way through to HP Operations Manager.
6) Another good idea is to get access to HP Operations Manager alert browser. This way you can see the alerts and make sure the right data is getting across.
7) The integration cartridge uses perl to manipulate XML data sent from the Foglight management server and call the necessary API to then send data onto the 3rd party system. Perl is not installed on the FMS by default so you will need to make sure it is. On windows I install ActivePerl which work very nicely and is also a free download. On unix, perl can often be installed by default but often the XML libraries are not. If you plan to execute the integration script in a folder other than the one recommended in the setup guide, you will need to make sure that the XML libraries are installed. I ran into this on-site, so I now carry XML-Simple.tar (free download) in my kit which I know works well for Linux.
Tip 2: Setup a good testing environment.
It’s important that you set up an environment that will make it easy for you to test your integration. I always do something like:
1) Create an alert in Foglight that you can control. I usually deploy the Application Monitor cartridge to the Foglight Management Server and get it to monitor a process called “the_foglight_cannot_be_stoppped”. I also set the quick check property of the agent to 30 seconds. Now assuming there is no process called “the_foglight_cannot_be_stopped” a FATAL alert should fire in Foglight and when you clear it, it should come back fairly quickly.
2) tail the latest Foglight Management log file. There is often very useful information in here especially if something is not working.
3) Setup debug in the integration script. The integration cartridge comes with a set of example scripts which I generally use as the base script. The one for HP Operations Manager is called forward-alerts-to-ovo-nix.pl for unix or forward-alerts-to-ovo-win.pl for windows. To turn on debugging edit the variable $debugfile and give it the name and path of a debug file:
my $debugfile = "/opt/quest/Foglight/debug.txt";
4) I usually tail this file during testing. Now every time an alert fires you should get data appended to this file relating to the alert.
5) Another option is to uncomment the line print Dumper($xmlref); in the integration script. This will provide a dump of all the data in the XML payload in the Foglight Management server log.
6) Make sure you turn off all debugging when complete otherwise log file will grow fast!
Tip 3: Tune for environment.
By now, technically your integration is complete. To test, simply clear the test alarm from the Foglight alarm dashboard and wait for the rule to re-evaluate and re-generate an alert. You should only have to wait 30 seconds or so, if you have set up your test alert as instructed earlier. Once the alarm appears in Foglight, check the necessary log files that you are ‘tailing’ for errors and assuming you have asked and been granted access to the HP Operations Manager alert browser, you should see something like this:
My experience is that the out of box default configurations which come with the integration cartridge will almost always need to be customized. The good news is that by now you have a working end-to-end test that you can customize with confidence. Another thing going for you is that the integration cartridge provides plenty of flexibility to make this a fairly simple exercise. The critical factor - gather all the requirements as early as possible and to plan out your customization.
Here is a list of considerations to take into account:
1) Which alerts, what severities, should be sent up to 3rd party system?
By default every alert that gets raised in Foglight will be forwarded. Some sites may only want certain alerts or certain severities forwarded. The ‘Alarm Integration’ rule is a good place to do filtering. For example, changing the fire condition from true to @event_foglight_severity_level == ‘FATAL’ will only forward Foglight FATAL alarms.
2) Severity mapping.
Severity levels generally differ between different systems, so some form of mapping needs to occur. The perl integration script comes with default mapping, but it is always a good idea to understand the severity levels of the source and target systems. For Foglight to HP it looks something like this:
|
Foglight severities
|
HP Operations Manager Severities
|
|
NORMAL
WARNING
CRITICAL
FATAL
|
NORMAL
WARNING
MINOR
MAJOR
CRITICAL
|
The default mapping is
Foglight FATAL = HP Critical
Foglight Warning,Critical = HP Normal.
The perl integration script is the best place to alter this mapping to meet requirements.
3) Calling opcmsg:
By default the integration cartridge calls opcmsg using the following parameters:
opcmsg appl=Foglight msg_grp=Foglight severity=critical node=fms_hostname obj=Foglight_rule_name msg_test=Foglight_alert_text.
Make sure that all parameters are ok for the environment in question. For example I had to change the obj parameter from Foglight_rule_name to SQLServer_InstanceName/DBName for a recent integration.
4) Auto clearing.
Foglight will auto clear an alert when the condition that triggered the alert goes back to normal. The integration cartridge allows for clearing events. Therefore it is possible to automate the clearing of an alert in HP operations manager. This does add a level of complexity to your integration, so as always get the basics working before going down this path.
That is about all I can think of for now. Hope this helps your integration go smoothly.
Best of luck,
Jim Katsos.