Byteman Plugin for RHQ

I read Andrew Dinn's post on Byteman 1.1.1 and was very intrigued. His post and the Byteman documentation was very quick and easy to read and I found out how easy it is to use Byteman. Just one (albeit long) command line option to your Java VM and you can easily inject code into existing Java classes using Byteman's byte-code instrumentation. I plan to use Byteman to do things like "fault-injection" testing - inject code that purposefully causes errors in my application to see how my application handles the errors (this is especially useful for forcing XA/2PC transaction errors to see how the application and transaction manager handles it).

I ended up doing some quick but useful enhancements to Byteman (thanks, Andrew, for the commit access :-) that allows any Java application to perform remote byte-code injection in a JVM where Byteman is running. There is now a Java API to the Byteman "submit" client (the new-and-improved Submit class).

I then took a few hours to write up a quick Byteman RHQ plugin prototype. I recorded a Flash demo to show how it works. Still a lot more that can be done with some additional enhancements to this plugin, but hopefully this gets across how easy it is to write RHQ plugins to manage anything. This plugin will allow you to view deployed rule definitions files and their rules and allow you to remove rules. With some enhancements, it could allow you to upload and deploy additional rules as well.

Interactive Shell / CLI for Jopr

Work is ongoing at a feverish pace to complete the new Jopr CLI. It is filling out quite nicely. What is this CLI, you ask? It's the long-awaited and often-requested command line interface utility for the Jopr Server.

Up until now, the only user interface into the Jopr system is via the web-based GUI. The new CLI now provides a scripting interface that allows users to tap into the Jopr system from their shell.

The CLI is actually a two-in-one utility - it can act as what people typically think of as a CLI but it is also an interactive shell (this is analogous to say, perl or python). The underlying scripting language is JavaScript - its the default Java scripting implementation found in the Java6 runtime.

Under the covers, the CLI utilizes the remote client API - this remote client API can be used by developers to write their own Jopr clients (anyone out there interested in writing a CLI GUI? :-) You can use the Java client or you can rely on the Web Services API that is exposed by the Jopr Server, allowing a client to consume the Jopr WSDLs and perform remote commands over Web Services.

I'll briefly show you just a few things you can do with the CLI. This should give you a taste for how you can interact with the CLI and some of its basic features.

First, here's the "long way" to query a list of resources and filter the results based on some custom criteria. Here I will ask the Jopr Server for all resources that have "mazzthink" in their name:

[mazz@mazzfedora bin]$ ./rhq-cli.sh -u rhqadmin -p rhqadmin -s mazzthink
RHQ - RHQ Enterprise Remote CLI 1.3.0-SNAPSHOT
Login successful
rhqadmin@mazzthink:7080$ criteria = new ResourceCriteria();
ResourceCriteria:

rhqadmin@mazzthink:7080$ criteria.addFilterName('mazzthink');

rhqadmin@mazzthink:7080$ ResourceManager.findResource

findResourceComposites    findResourceLineage       findResourcesByCriteria
rhqadmin@mazzthink:7080$ ResourceManager.findResourcesByCriteria(criteria);
id    name                                                            version        curre
ntAvailability resourceType
------------------------------------------------------------------------------------------
----------------------------------------------------------------------
10001 mazzthink                                                       Win32 5.1      UP
             Windows
10002 Trapd (mazzthink)                                                              DOWN
             SnmpTrapd
10004 mazzthink RHQ Agent                                             1.3.0-SNAPSHOT UP
             RHQ Agent
10005 mazzthink Jopr Server,  JBossAS 4.2.3.GA default (0.0.0.0:2099) 4.2.3.GA       UP
             JBossAS Server
10007 mazzthink Apache 2.2.9 (C:\mazz\apache-httpd\)                  2.2.9          DOWN
             Apache HTTP Server
10008 mazzthink Apache 2.2.9 (C:\apache\)                             2.2.9          DOWN
             Apache HTTP Server
10035 mazzthink  File System (local) C:\                                             UP
             File System
10059 mazzthink  Embedded JBossWeb Server 2.0.1.GA  (0.0.0.0)         2.0.1.GA       UP
             Embedded Tomcat Server
8 rows

A couple things to note from above. I created a ResourceCriteria that allows me to set filters, sorting and paging controls. Here I just filter on the name "mazzthink".

Second, I used the CLI interactive shell's auto-complete feature to help me figure out what methods are available on the ResourceManager. I simply typed "ResourceManager.findResource" then hit the Tab key and you see the suggestions printed out by the auto-completer. I then chose to use the findResourcesByCriteria method.

Since this is one of the most common things to want to do in the CLI (that is, query for resources), there is a shortcut method you can use for this - findResources. Below you will see me doing the same query as above, only using the "short way":

rhqadmin@mazzthink:7080$ findResources('mazzthink')
id    name                                                            version        curre
ntAvailability resourceType
------------------------------------------------------------------------------------------
----------------------------------------------------------------------
10001 mazzthink                                                       Win32 5.1      UP
             Windows
10002 Trapd (mazzthink)                                                              DOWN
             SnmpTrapd
10004 mazzthink RHQ Agent                                             1.3.0-SNAPSHOT UP
             RHQ Agent
10005 mazzthink Jopr Server,  JBossAS 4.2.3.GA default (0.0.0.0:2099) 4.2.3.GA       UP
             JBossAS Server
10007 mazzthink Apache 2.2.9 (C:\mazz\apache-httpd\)                  2.2.9          DOWN
             Apache HTTP Server
10008 mazzthink Apache 2.2.9 (C:\apache\)                             2.2.9          DOWN
             Apache HTTP Server
10035 mazzthink  File System (local) C:\                                             UP
             File System
10059 mazzthink  Embedded JBossWeb Server 2.0.1.GA  (0.0.0.0)         2.0.1.GA       UP
             Embedded Tomcat Server
8 rows

Now that I got my list of resources, I can use the results to help perform additional work. Suppose I want to know how much free memory I have on my "mazzthink" machine. I can see from my query above that the "mazzthink" Windows platform has an id of 10001. I'll pass that to the ProxyFactory.getResource method to retrieve a resource object that I can use to access information about that resource (such as its free memory metric):

rhqadmin@mazzthink:7080$ myResource = ProxyFactory.getResource(10001);
rhqadmin@mazzthink:7080$ myResource.

OSName                          OSVersion
architecture                    children
createdDate                     description
freeMemory                      freeSwapSpace
getChild                        getMeasurement
handler                         hostname
id                              idle
manualAutodiscovery             measurementMap
measurements                    modifiedDate
name                            operations
pluginConfiguration             pluginConfigurationDefinition
systemLoad                      toString
totalMemory                     totalSwapSpace
usedMemory                      usedSwapSpace
userLoad                        version
viewProcessList                 waitLoad
rhqadmin@mazzthink:7080$ myResource.freeMemory
Measurement:
              name: Free Memory
      displayValue: 872.8MB
       description: The total free system memory

Notice above I illustrate the usage of the auto-completer again. I hit Tab after typing in "myResource." and the CLI showed me all the valid measurements, operations and other methods that are available for that resource. I chose to get the "freeMemory" measurement, and you can see above that this resource has 872.8MB of free memory.

In addition to getting measurement data, another thing I can use the CLI for is to execute operations on managed resources. Continuing our example with the mazzthink platform resource, suppose I wanted to get a list of all the processes running on that box. The platform resource has a "viewProcessList" operation available to it (as you can see above from the auto-completer output). If you are unsure what operations are available on a resource, you can use the "operations" method on your resource proxy object to get the list:

rhqadmin@mazzthink:7080$ myResource.operations
name                description
------------------------------------------------------------------------------------------
----------------------------------------------------------------------
viewProcessList     View running processes on this system
manualAutodiscovery Run an immediate discovery to search for resources
2 rows

Using the CLI, we can invoke that viewProcessList operation on the mazzthink Windows platform and have the results shown on the CLI console:

rhqadmin@mazzthink:7080$ myResource.viewProcessList()
Invoking operation viewProcessList
Configuration [12186] - null
processList [99] {
  process [5] {
    kernelTime = 25140
    name = winlogon.exe
    pid = 1836
    userTime = 8156
    size = 95723520
  }
  process [5] {
    kernelTime = 31
    name = httpd.exe
    pid = 516
    userTime = 31
    size = 25780224
  }
  process [5] {
    kernelTime = 2296
    name = postgres.exe
    pid = 1572
    userTime = 937
    size = 75284480
  }
  process [5] {
    kernelTime = 159421
    name = javaw.exe
    pid = 2776
    userTime = 380347
    size = 1215180800
  }
  process [5] {
    kernelTime = 91190
    name = firefox.exe
    pid = 6088
    userTime = 251285
    size = 876466176
  }
  ...
}

And finally, to illustrate that the CLI is really a CLI and not just an interactive shell, you can pass a script command directly on the command line via the -c option:

[mazz@mazzfedora bin]$ ./rhq-cli.sh -u rhqadmin -p rhqadmin -s mazzthink -c "Pr
oxyFactory.getResource(10001).freeMemory;"
RHQ - RHQ Enterprise Remote CLI 1.3.0-SNAPSHOT
Login successful
Measurement:
              name: Free Memory
      displayValue: 884.5MB
       description: The total free system memory

The CLI also has a -f option that allows you to specify a script file to be executed - allowing you to write and prepackage scripts for execution later.

That's the CLI in a nutshell. It's still a work in progress - the remote API is still being flushed out and being enhanced to support the many use-cases we expect people are going to want to support.

The community documentation is also currently being worked on. But there is some documentation now, with more to come at the following pages:

CLI Install Documentation
Running the CLI documentation

Monitoring Generic Processes

A previous blog of mine talked about how you can monitor an application or generic operating system process by going through its Script/CLI interface (you can watch a demo of the Script plugin to see it in action).

There is now another way to do this in Jopr, thanks to the new process monitoring feature introduced in the platform plugin.

Even if you do not have a CLI interface for the operating system process you want to monitor, you can still ask Jopr to monitor it - you do so by pointing Jopr to a pidfile containing the process's pid, or you can give Jopr a PIQL process query to execute to find the process. (PIQL = process information query langage). Note that this is all cross-platform - you can monitor processes on Windows just as easily as you can on any UNIX flavored operating system, so long as there is native support for your platform and it is enabled (which is normally the case, so you typically don't have to worry about this - you'll get this stuff for free).

What this means is for any process running anywhere out on your network, you can monitor that process. This process monitoring mechanism can be used to inform you if your managed process is up or down, and it can collect some metrics on the managed process, such as CPU utilization, memory consumption, and file descriptor usage.

You can watch a demo of this process monitoring feature to get a feel for how it can be used and what it does.

Managed Resources Exposed Via WebDAV

Thanks to Greg Hinkle, Jopr has recently introduced an experimental feature that exposes your managed environment as a WebDAV repository.

This means that you can use any WebDAV client to browse your managed environment as if it were a simple file system and peek into your resource hierarchy, obtaining information such as a resource's availability, its configuration and measurement trait data (watch this WebDAV demo to see it in action; Greg's blog post and demo are here).

If you aren't familiar with WebDAV, all this means is that your managed environment will look like a simple file system (in the Microsoft Windows vernacular, it is called a "Web Folder"). Directories in this "file system" or "Web Folder" represent managed resources and files found in those directories represent data about those resources.

So, for example, if you had a machine called "comp.xyz.com", and on that machine you have installed a JBossAS server, a PostgreSQL server and a network adapter, your WebDAV file system paths to access those resources could look like this (notice how they look like simple file system paths):

Machine itself:	/webdav/resource/comp.xyz.com
Network adapter:	/webdav/resource/comp.xyz.com/eth0
JBossAS server:	/webdav/resource/comp.xyz.com/Banking%20App
PostgreSQL DB:	/webdav/resource/comp.xyz.com/My%20Postgres

This is currently in the "experimental" phase; however, by watching my WebDAV demo, it should be easy to see how this can prove to be a very powerful feature. Using any WebDAV client, you potentially could obtain alot of information about your managed resources (measurement data, alerts, events, logs, configuration and much more). It's pretty easy to add functionality to this WebDAV interface, so providing access to things like measurement data, alerts, events and the like should not be hard to accomplish. In about 30 minutes, I added the ability to view a resource's measurements traits (you will see that feature in the demo). 6/29/2009 note: spent another couple hours coding and was able to incorporate authentication/authorization into the WebDAV tier so it utilizes the normal Jopr authz layer; also added a new WebDAV resource "measurement_data.xml" which is an XML file of the numeric measurement data for a resource (min, max, avg values, etc).

Now we just need to find the time to flush out this functionality and make it production ready with additional features. If you are interested in working with the Jopr team and willing to get your hands dirty writing code, this might not be a bad place to start. You really don't need to know too much about the internals of the Jopr Server - you just have to interface with the internal Jopr stateless session EJB3 beans to obtain data about resources. The WebDAV API seems pretty easy to use - it's all based on the third party library called Milton. Don't hesitate to ask the team for help in getting started as a Jopr developer, we are usually around on freenode at #jopr or send an email to "jopr-dev at lists.jboss.org"

Jopr Is Now Easier To Install and Demo

Well, Joe did it again. I don't know how he does these things so fast but here you go:

http://josephmarques.wordpress.com/2009/06/03/jopr-has-embedded-database-support/

With this new H2 embedded database support, you can now install Jopr much easier now. You no longer have to install your own PostgreSQL or Oracle database separately. You now have the option to use the "embedded mode" which tells Jopr to use the embedded H2 database and the embedded agent.

This means to install and try out Jopr involves these simple steps (oh, and these are cross-platform instructions - it works the same on Windows as it does on UNIX or Linux):

1. Unzip the Jopr distribution
2. Run the Jopr server startup script (rhq-server.[sh, bat] start)
3. Point your browser to http://localhost:7080
4. Click the "Embedded Mode" button
5. Click the "Install!" button
   

That's it! Jopr will finish the installation for you, creating your H2 database and starting up both the server and embedded agent. The agent will automatically begin discovering and monitoring resources it finds on your box. The browser will provide you with a link to get started in the Jopr UI.

Use this quick and easy method to try out Jopr. You can try it out and demo it on anything from a laptop to a mainframe - so long as you have a Java virtual machine available.

(To Joe - thanks, job well done! Joe did the heavy lifting, I just enhanced the installer to be able to install Jopr using the embedded database.)

Supporting Events In Your Custom Plugin

This blog is in response to a question from user "earthling" about my Correlating Events with Jopr blog entry.

"How do i use this feature [Events] in a custom plugin?"

It's easiest if you just copy the code from one plugin to your custom plugin. It isn't that much code.

First, you need to declare that your plugin's custom resource type supports events. Look at how the JBossAS plugin descriptor does this - you can probably copy the <plugin-configuration>...<c:group name="event" displayName="Events"> section verbatim.

Then, your custom ResourceComponent subclass needs to add an event poller to the plugin container. You can do this within your start() method, as the JBossASServerComponent does, like this:

public void start(ResourceContext context) throws Exception {
 this.logFileEventDelegate = new LogFileEventResourceComponentHelper(this.resourceContext);
 this.logFileEventDelegate.startLogFileEventPollers();
 ...
}

Note that the "LogFileEventResourceComponentHelper" class is new that Ian added after 2.2. So, you'll have to build trunk/HEAD to be able to use it. If you have 2.2 or less, just grab that class and use it in your plugin (or at least copy-n-paste its code).

In your components stop() method, you need to remove the event poller, too:

public void stop() {
 this.logFileEventDelegate.stopLogFileEventPollers();
 ...
}

Once your resource has started, it will start emitting logs that the Jopr Server will show you! You get all the event functionality by adding that little bit of metadata and code. Of course, the example above assumes your events come from log4j log files. But events can come from anywhere. For example, the Windows platform plugin component can emit events from the Windows Events subsystem and Greg Hinkle has updated the Linux plugin component so that it emits syslog messages from his Linux box (he should probably check that in soon :).

Managing Resources That Have A CLI

Sometimes, you may find yourself wanting to manage a resource for which there is no current Jopr plugin or, perhaps, the resource doesn't even have a management API or interface.

I have developed a new "Script plugin" that may help manage these "unmanageable" resources. You can watch a demo that shows the Script plugin in action.

This plugin will assume there is a command line executable or script that can be used to interface with the managed resource. What you do is manually add your "Script" resource to the Jopr inventory - defining where the command line executable or script is, and how its results/output should be used to determine the state of the managed resource. To be clear, the "CLI" executable/script is not really the managed resource - it is only used to interface with the real managed resource. The typical example I give people is: "apachectl" is the script, Apache Web Server is the real managed resource.

There is the ability to show RED or GREEN availability for any Script resource. For example, I can configure the Script plugin to assume the resource is UP (aka GREEN) if:

• the executable merely exists on the file system, or...
  
• the executable was able to run or...
  
• the executable ran and returns a certain exit code and/or...
  
• the executable ran and output a certain text string

The idea is that I have a command line executable or script that needs to get executed with a certain set of arguments. A successful execution would be determined by a regular expression that needs to match the exit code or output from that CLI.

You can also use the same type of mechanism to define how to invoke the executable to obtain a certain set of metrics. You can define arguments and a regular expression and based on what the executable returns (and what the regex captures) would be the value of the metric data.

This plugin should be useful because it is very generic - you can use it as-is for your own generic resources that have command line interfaces, a developer could extend the plugin classes, or a developer could just write their own plugin descriptor and have it pick up the Script plugin's ability to execute any executable and get the results. I think we'll be able to extend this to build a Nagios plugin, for example.

If you have any thoughts or ideas on what features such a Script plugin should have, feel free to comment. I would be interested to know what kinds of CLIs people run that manage their resources.

If you are interested in helping out develop a Nagios plugin, let me know - I'm very desparate in getting help on that. :)

[note: this blog post was updated July 2009 to reflect the new name of the plugin. This plugin was previously called the "CLI" plugin, but in order to avoid confusion with Jopr's new remote client/CLI, it was renamed to "Script" plugin.]