Managing Resources That Have A CLI

Sometimes, you may find yourself wanting to manage a resource for which there is no current Jopr plugin or, perhaps, the resource doesn't even have a management API or interface.

I have developed a new "Script plugin" that may help manage these "unmanageable" resources. You can watch a demo that shows the Script plugin in action.

This plugin will assume there is a command line executable or script that can be used to interface with the managed resource. What you do is manually add your "Script" resource to the Jopr inventory - defining where the command line executable or script is, and how its results/output should be used to determine the state of the managed resource. To be clear, the "CLI" executable/script is not really the managed resource - it is only used to interface with the real managed resource. The typical example I give people is: "apachectl" is the script, Apache Web Server is the real managed resource.

There is the ability to show RED or GREEN availability for any Script resource. For example, I can configure the Script plugin to assume the resource is UP (aka GREEN) if:

• the executable merely exists on the file system, or...
• the executable was able to run or...
• the executable ran and returns a certain exit code and/or...
• the executable ran and output a certain text string

The idea is that I have a command line executable or script that needs to get executed with a certain set of arguments. A successful execution would be determined by a regular expression that needs to match the exit code or output from that CLI.

You can also use the same type of mechanism to define how to invoke the executable to obtain a certain set of metrics. You can define arguments and a regular expression and based on what the executable returns (and what the regex captures) would be the value of the metric data.

This plugin should be useful because it is very generic - you can use it as-is for your own generic resources that have command line interfaces, a developer could extend the plugin classes, or a developer could just write their own plugin descriptor and have it pick up the Script plugin's ability to execute any executable and get the results. I think we'll be able to extend this to build a Nagios plugin, for example.

If you have any thoughts or ideas on what features such a Script plugin should have, feel free to comment. I would be interested to know what kinds of CLIs people run that manage their resources.

If you are interested in helping out develop a Nagios plugin, let me know - I'm very desparate in getting help on that. :)

[note: this blog post was updated July 2009 to reflect the new name of the plugin. This plugin was previously called the "CLI" plugin, but in order to avoid confusion with Jopr's new remote client/CLI, it was renamed to "Script" plugin.]

Applying Patches, Updates and Other Content via Jopr

We like to say Jopr is a management platform, not just a monitoring tool. This is because Jopr does more than just monitor the health of managed resources. It can be used to configure resources and control resources, too. In addition, Jopr has a content subsystem that allows it to deploy content to any managed resource that supports the content facet.

What does this mean? It means that you can set up your Fedora boxes to be able to "yum install" packages directly from Jopr; it means you can install patches to your JBossAS Servers; it means just about anything you want it to mean (in the context of pushing and pulling content) because Jopr is extensible in such a way that you can write your own plugins to do what you need it to do with respect to pulling down content for deployment to your custom resources.

I've created a Jopr content demo that shows how you can aggregate content from multiple remote repositories into Jopr and then have Jopr serve that content to resources it is managing.

Here is an architectural diagram that discusses how content flows from remote repositories to the Jopr Server through to the managed resources via the Jopr Agent. This diagram, coupled with the demo, should provide some good insight into the basics of the Jopr content subsystem.

Jopr and Embedded Jopr - What Are They?

Jopr and Embedded Jopr. Aside from each having Jopr in the name, how are they related? How are they different? Why do these two projects exist? I'll try to explain that here.

Jopr is a management platform. It allows you to manage an entire network of systems and products hosted in your IT environment.

Embedded Jopr is a management console that is embedded in a JBoss Application Server to allow you to manage that particular JBossAS instance.

Jopr and Embedded Jopr use what are known as "plugins" to do the real management work - each plugin is specific to a particular product or software component that needs to be managed. For example, today there exists plugins to manage JBossAS, Tomcat, Hibernate, and PostgreSQL, among other things. These "plugins" live in what is called the "Jopr Plugin Container". This plugin container is responsible for controlling the lifecycle of the plugins.

Jopr and Embedded Jopr are related in one important aspect - they share alot of the same code! The beauty of the architectural design of Jopr is such that you can take the Jopr Plugin Container and its plugins and embed them in any Java virtual machine - including a virtual machine that is running a JBossAS instance.

Embedded Jopr uses the Jopr Plugin Container and embeds it directly inside a web application (aka .war) that is deployed inside JBossAS, allowing you to directly manage that JBossAS instance. In other words, Embedded Jopr is managing the very JBossAS instance in which it lives.

Jopr, on the other hand, has a standalone Jopr Agent and it is this Jopr Agent that embeds the Jopr Plugin Container. The Jopr Agent is able to do a few things Embedded Jopr cannot do but the main difference is that the Jopr Agent can communicate with a Jopr Server cloud, allowing it to participate in a full-fledged management enterprise environment.

The next couple diagrams illustrate these two different models. First is Embedded Jopr. Notice that everything here is directly hosted within a managed JBossAS instance. Inside JBossAS are, of course, its own internal services, one of which is Embedded Jopr. Embedded Jopr is used to manage this JBossAS instance. It is analogous to the jmx-console, only with more powerful features. You will notice that inside Embedded Jopr is code for the actual user interface as well as the Jopr Plugin Container. This Jopr Plugin Container code is identical to the plugin container used by Jopr - not a single line of code is different in this Jopr Plugin Container as compared to the plugin container deployed within the Jopr Agent (which we'll talk about later on). The plugins are, also, identical. In fact, this is designed in such a way that allows others to write custom Jopr plugins and deploy them, not only in Jopr Agents, but in Embedded Jopr as well. The hope is that Embedded Jopr will be able to support the management of other services deployed in its JBossAS instance, thus allowing you to enhance and extend Embedded Jopr with off-the-shelf plugins or your own custom plugins to manage your own custom services. There are many opportunities to extend Embedded Jopr in this way - think of all the different components that you could deploy in JBossAS that you'd want to manage (Portal, Seam, jBPM, Drools, JBossTS, JBossCache, etc.); if you had plugins for them, you'd just deploy them inside Embedded Jopr's plugin container and you could immediately begin to manage them. You could then take those plugins and later deploy them inside a Jopr Agent (without the need to touch or even rebuild your plugins) and have them be used within a full-fledged Jopr management enterprise. This concept has been proven to work by the mere fact that Embedded Jopr exists. Embedded Jopr already ships with some plugins common with the Jopr Agent (the JMX plugin is one of them).

Now let's look closer at the deployment model of Jopr. First, notice that the Jopr Agent, at its core, is the same Jopr Plugin Container that we saw in Embedded Jopr. It's the same code. Reusability is a wonderful thing. And remember, not only is the Jopr Plugin Container reused, but the plugins themselves are 100% reuseable, too. But now notice the difference. The Jopr Agent is standalone - its separate from any managed JBossAS instance. In fact, the Jopr Agent can manage multiple JBossAS instances! Not only that, but it can also manage any number of products or components - PostgreSQL databases, remoted Java Virtual Machines, operating system services, and anything else you want (even your own custom products or components), as long as you have a plugin that is capable of managing them (I won't get into plugin development topics, see the Plugin Development wiki to learn how you can write your own plugins).

The next major difference you see here is the Jopr Agent can communicate with the Jopr Server cloud (which consists of 1 or more Jopr Servers with persistence storage backing those servers). The Jopr Server provides additional capabilities not available to Embedded Jopr. These include the ability to: persist historical metric data from your managed resources, provide an alerting mechanism to notify IT administrators when something goes wrong within your managed environment, provide a persisted audit trail for security tracking, provide persistence storage of events occuring within your managed resources, and many other things.

Hopefully, I've answered the basic question, "What is Embedded Jopr and how is it different from Jopr?".

What I've also hoped to convey was a bit of an architectural overview of the Jopr Plugin Container and its management plugins and how their reusability is exploited to make development and use of both Embedded Jopr and Jopr much more easier.

SVN Statistics

I wanted to collect some SVN statistics for some codebases and found a very nice project that generates useful SVN reports and graphs. The project is called StatSVN.

It's very easy to use. I'll include both a UNIX and Windows script here so no matter what platform you are on, you can easily use this. Each script is only about 20 lines long.

First, download the statsvn.jar from the StatSVN download page.

Now, put the run-svnstats script (which I give below) in the same directory as the statsvn.jar file. If on UNIX, use the first script; Windows use the second script:

UNIX Script

#!/bin/sh

WORKING_COPY=/home/me/perf/src/rhq/trunk/modules
STATS_GEN_DIR=/home/me/svnstats-gen
START_END_REVISIONS=10:HEAD

SVN_LOGFILE=$STATS_GEN_DIR/svn.log
STATSSVN_JAR=$STATS_GEN_DIR/statsvn.jar

cd $WORKING_COPY
svn up
svn log -v --xml -r $START_END_REVISIONS > $SVN_LOGFILE
cd $STATS_GEN_DIR
if [ -d svnstats ]; then
 rm -rf svnstats
fi
mkdir -p svnstats
cd svnstats
java -jar $STATSSVN_JAR -concurrency-threshold 2000 -threads 50 $SVN_LOGFILE $WORKING_COPY

echo SVN Statistics are now located here: `pwd`

Windows Script

@echo off

set WORKING_COPY=C:\source\rhq\trunk\modules
set STATS_GEN_DIR=C:\svnstats-gen
set START_END_REVISIONS=10:HEAD

set SVN_LOGFILE=%STATS_GEN_DIR%\svn.log
set STATSSVN_JAR=%STATS_GEN_DIR%\statsvn.jar

cd %WORKING_COPY%
svn up
svn log -v --xml -r %START_END_REVISIONS% > %SVN_LOGFILE%
cd %STATS_GEN_DIR%
if exist svnstats rmdir /S /Q svnstats
mkdir svnstats
cd svnstats
java -jar %STATSSVN_JAR% -concurrency-threshold 2000 -threads 50 %SVN_LOGFILE% %WORKING_COPY%

echo SVN Statistics are now located here: %STATS_GEN_DIR%\svnstats

Lastly, you just have to edit the script to match your environment. Change the three variables defined at the top:

• WORKING_COPY: the full path to the SVN working copy as found on your local file system. This SVN working copy will be svn updated as part of the script, so make sure you don't mind having this working copy get updated to the latest revision.


• STATS_GEN_DIR: the full path of the location where you stored the statsvn.jar and the scripts. The generated SVN log and the final HTML pages, reports and images will get stored here as well.


• START_END_REVISIONS: the starting revision number and ending revision number of your SVN working copy. Only SVN statistics for that revision range will be reported on. This should be in SVN format (e.g. 1234:5678 or 1:HEAD)

That's it. Run the script and out comes an "svnstats" directory with a full set of HTML pages and images (this svnstats directory will be located under the STATS_GEN_DIR directory). Just copy the svnstats directory to a webserver (e.g. Apache's htdocs directory) and point your browser to the svnstats location and you can now peruse your SVN reports.

You can even run this script as a cron job to automatically update your stats periodically. Just have STATS_GEN_DIR point to a directory that your webserver can serve up and make sure your START_END_REVISION takes into account HEAD (so new checkins will get reported).

I thought this was a very simple yet cool way to get SVN stats for any SVN project; hope you can find this as helpful as it did me.