Cross-Facet Correlation Provided by Jopr

I'm surprised you are still reading this, given the nebulous title of this blog :). But I'm glad you are here. I want to further explain what my last blog was really trying to convey.

Jopr is essentially an abstract management framework - at its core, Jopr does not understand about the concrete managed resources it actually is managing. Specific, concrete knowledge of the manage resource implementations is pushed out to the edges - in the agent plugins themselves.

But what Jopr knows, at its core, are the different management "facets" that those managed resources support. What is a "management facet"? Simple - a facet is a subset of functionality that all managed resources may or may not support. Facets are orthogonal - a managed resource and its plugin can support any number of facets in any combination.

Jopr supports several types of facets - for example, the measurement facet (for collecting metric data), the operation facet (for controlling managed resources), the configuration facet (for retrieving and setting resource configurations), the content facet (for pushing and pulling file content to/from managed resources), the event facet (to emit asynchronous event information such as a managed resource's log file messages) and others.

There are numerous advantages for having an abstract management platform that can handle all of these different management facets for many different kinds of managed resources. One of these advantages is the ability to correlate and process information across all the different facets.

Looking at Jopr's summary timeline, and what "cross-facet correlation" means soon becomes very apparent:This timeline (with time moving from left to right), in one view has correlated information such as:

• when the resource was up and down; notice the background color - light green means the resource was up at that time and a red background means the resource was down with grey meaning it is unknown what the state of the resource was (measurement facet)
• when events (e.g. log messages) happened, how many of them occurred and how severe these events were (severity is shown in the badging of the events icons). (event facet)
• when alerts were raised and how severe those alerts were (alerts are shown as flag icons with their severity indicated by their color). Alerting isn't an actual management facet per-se, the alert subsystem is provided to all managed resources with no effort needed on the part of the plugin - alerts are provided "for free".
• when a resource's configuration was changed and whether or not that change was successful or if it failed (configuration facet)
  
• when an operation was invoked on a resource and whether or not that attempt to control the resource was successful or not (operation facet)

This is what "cross-facet correlation" is all about. How is this helpful? Well, if I see a gap of events in my timeline, where that gap has a background of red, that tells me very quickly that I am not receiving any events because the resource is down! If I see that I changed the configuration of a resource, and soon after I see a flood of warning events and then perhaps followed by a red background, I can immediately begin to suspect that that configuration change caused an adverse affect on the behavior of my resource. If I executed an operation and soon after see one or more alerts trigger, I should start by investigating if that operation caused the resource to act strangely.

Cross-facet correlation occurs in other areas of the Jopr UI (and the sky's the limit to where we can take Jopr from here in the future). For example, here is the summary tab for my agent. Notice how all the different facets are combined into a single view so at a glance you can see what is currently going on with this agent resource.

I can see that I had one operation, out of the past three, that failed. I can see that I recently updated the configuration of this resource several times, one of which failed. I can see what the current measurements are for this resource, and what events and alerts have been logged, when they happened and their severity.

All of this data is linked to their respective pages in the UI - if I want to see more about the alerts, I click the alert links. More about the operations? Click the operation links. And so on.

And because Jopr has abstracted the facets so they are applicable across any number of managed resources, we can manage all different kinds of resources - JBoss Application Servers, Apache Web Servers, Tomcat Web Application Servers, hardware boxes, operating system services, even Jopr itself - and we reuse the same UI pages, the same code, and the same look & feel - no additional code needs to be introduced to the server to support additional types of managed resources!

For example, above you see the summary for the Jopr agent. What's the summary information look like for a JBoss Application Server that I am managing? You can see this here. Notice that the same look & feel, the same UI pages and code is used, the same SQL queries - everything is reused. No additional integration is needed on the server side. But notice the difference. The agent resource supports the "configuration facet" - which is why you saw the "Configure" tab and the configuration update information earlier. But the agent does not support the "content facet" (the agent does not push or pull content over the Jopr content subsystem). But look at the JBossAS resource - it does not support the configuration facet (so you don't see the Configure tab and there are no recent config updates) but it does support the content facet (you can see the Content tab along with some recent package history). The JBossAS resource component in the agent will send up to the Jopr server information about what packages (e.g. jar libraries) it has installed. If a resource supports it, you can even ship down updated packages to the resource (for example, to send down new jars that incorporate bug fixes).

The above just talks about correlating cross-facet information for a specific resource. What if I want to see information across my entire inventory? What if I want to see all of the alerts that were triggered by Jopr, regardless of the resource that triggered the alert. What if I want to see all the configuration changes made to my environment, regardless of which resource was reconfigured?

Again, because Jopr is abstract in nature, we can scan the inventory history and aggregate this kind of information. Below you see that we can view all the alerts and all configuration changes - you can even filter the results if you only care about a subset of the data.




I think this clearly shows how an abstract management platform that supports multiple management facets can provide tremendous value to anyone managing a network of hardware and software products.

Well, that's all I have to say on this subject (for now). I hope this makes it a bit more clear what Jopr brings to the table and its value-add that it can bring to your IT environment.

Correlating Events with Jopr

Today, I rediscovered how nice Jopr really is when I enhanced the agent plugin so it can track the agent log files.

In just an hour or two, I added a feature allowing you to enable log tracking for the agent itself. If you enable this feature, you can view the agent's log messages directly in the Jopr UI. This enables you to see what's going on inside an agent, and you can corrolate those log events with other types of changes happening to your agent (e.g. configuration changes, monitoring data, alerts, etc).

Let me discuss some of the nice things this allows you to do. Remember, even though this is a concrete example using the agent itself as the managed resource, everything I'm about to discuss can be done for your own managed resources because all of these features are abstract and can be utilized by any plugin, should the plugin developer choose to use them. This is what an abstract management framework provides you and is what Jopr is all about.

First, I took between one and two hours to add code to the agent plugin in order for it to utilize the event subsystem provided by Jopr. A little bit of Java code, a little bit of XML and I went from nothing to being able to fully integrate the agent log files into the Jopr events subsystem.

OK, so, what does this get you? First, and the most obvious, is you can now view the log message events from within the Jopr UI. You do not have to remotely log into the machine where the agent is running to view its log messages. See the image here on the left - this is the event history view. Because the agent emits its log messages as events, this event history view is essentially browsing the log file, with the added bonus of being able to filter the view based on the log message content and the severity of the messages (INFO, WARN, ERROR, etc).

Second, you can view the events corrolated with monitoring data - this allows you to see what your resource's measurements looked like at the time the resource was logging messages (which might help you in diagnosing a problem). Take a look at the bottom of the graphs and you can see the different colored icons to indicate the highest severity of events that occurred in any timeslice shown on the graph. Here you can see INFO (green), WARN (yellow) and DEBUG (blue) messages occurring in different times. And notice how you can corrolate those times with measurement data and event activity.

You can even drill down into the log file directly from this view so you can read the first several log messages that occurred within a narrow span of time. Again, this might help you to diagnose a problem if, using this agent resource as an example, you can see the actual ERROR log messages that occurred in or around the same time you saw the average execution time for sent commands starting to go up.



Jopr can be more proactive with these events/log messages, as well. You can define an alert definition such that you will get notified (via email for example) if the agent emits a log message at the ERROR severity level. You can even be alerted when a specific log message is emitted (e.g. have you ever wanted to be emailed when your application spits out an OutOfMemoryError log message? Now you can!). The alert definition UI page allows you to set this up.

And finally, you can use the summary timeline to further corrolate log message events with other things that have happened to this agent resource. For example, notice that I can see when my event messages occurred (and what their highest severities were) on this timeline correlated with other things that happened to this agent, such as when its configuration changed and when alerts were triggered. You would also see when operations were invoked on this timeline as well had any operations been executed during this timeframe that the timeline is showing. In effect, you get a wholistic view of what happened to this agent resource, across all the different management facets: configuration changes, control executions, alerts, events, etc!


Once again, I must emphasize the fact that you can get all of this functionality, too - and all you need is to write a plugin that talks to your managed resource and provides the raw data to Jopr. You get everything else for free - the corrolated timeline, the monitoring graphs, alerting and more.

So, even though the above screenshots show this functionality for the agent resource, the UI and all of these capabilities would be the same for any resource you want to manage, so long as that managed resource has a plugin that provides the same kind of raw information. In my case, I just had to spend a couple of hours to get the agent to report events from its log files, and Jopr took care of the rest. For example, I did not have to do anything to enable the alerting capabilities or the corrolating timeline. That is the value that Jopr brings to the table!

The Mighty Embeddable Plugin Container

Heiko has just demonstrated another way that the agent-side plugin container can be embedded in any Java VM.

We've already proven that this concept works because the plugin container has already been embedded in a few places: not only does the agent itself embed the plugin container, but our unit tests do it when the validity of plugins needs to be tested and also the Embedded Jopr project does it by embedding the plugin container directly in a JBossAS5 application server!

But what Heiko has done is go a step further by providing a very small, yet useful, wrapper around the plugin container to support plugin developers (it is called the "standalone plugin container"). It is "standalone" because you no longer need to install and run a full Jopr environment (a server and a database) in order to test your plugin's functionality.

If you are writing a custom plugin, just use the standalone plugin container and deploy and execute your plugin. This means you just take an existing agent distribution, and use a very simple script to start the standalone plugin container (there are Windows and UNIX versions of the script). The simplicity of these scripts border on trivial. Under the covers, all this does is run a new main class that embeds the plugin container, as opposed to the original AgentMain class (which does all the complex agent-to-server communications). This new standalone plugin container will accept commands on the prompt to help you exercise your plugin code - a great help for those writing plugins. See the README for some install instructions.

This type of capability sort of existed in the old JBoss ON 1.x code base - but its old (now obsolete) plugin model was never this modular and could never have been this easily embedded in so many different ways.

Jopr 2.2 Sneak Peek

Jopr 2.2 is almost ready for release. I was going to build some nice flash demos, but there is so much new to cover, I didn't know where to begin! I wanted to get the message out quickly though, so, rather than wait for my flash demos to be completed, I decided to simply take some screen snapshots and post them up on the Jopr wiki.

Take a sneak peek here at what's coming - I think you will find Jopr 2.2 to be a great leap forward. It is much better than anything JBoss ON 1.x ever could or did deliver, and it is a major milestone even from the latest releases of JBoss ON/Jopr.

I am very proud of what we have been able to accomplish as a team - this is definitely the most feature-rich release of any previous JBoss ON/Jopr release.

Quick Java Heap Analysis

I had a need to do some quick profiling of a Java application. I didn't want to spend alot of time (or money) setting up a profiler and my app to run within that profiler's environment. But it turns out that SUN's Java 6 distribution ships with some very handy tools that you can use to do what I wanted without setting up anything special - didn't even have to pass in any special set of -D system property definitions.

First, run your application and get it into a state that you want to analyze. Then perform the following steps:

1. Run "jps" to determine the process ID (aka pid) of your Java application. BTW: this is a cool nugget in and of itself. I've been so used to doing "ps -elf | grep java" or "pgrep java" to find my running Java applications that I never bothered looking for a simpler way. "jps" dumps the pid and the simple name of the Java main class - and most of the time this is all you are looking for. But you can also get the fully qualified main class names and the argument lists - use "jps -help" to see the usage syntax. This is a very helpful tool all by itself.


2. Run "jmap -dump:format=b,file=dump.dat <pid>" where <pid> is your Java applications's pid that you found in the previous step. This will dump information about your Java VM's memory in the file "dump.dat" and will be used by jhat in the next step.


3. Run "jhat -J-Xmx512m dump.dat" to start the Java heap analysis tool (jhat). This will start an HTTP listener on port 7000. The -J option lets me configure the jhat VM, which is needed if you have a large dump to analyze.


4. Point a browser to http://localhost:7000 and begin browsing around.

This jhat tool let's you examine the objects and classes currently found within your VM. It's nothing glorious and certainly not as well featured as many of the other commercial and open-source profilers out there. But, it comes with the JDK, and requires absolutely no setup. It can't get much easier to perform a quick heap analysis than this.

Credit goes to Frank Kieviet, whose blog brought my attention to these tools.

If Your Computer Is Going To Blow Up, Let Jopr Warn You

Well, OK, Jopr won't let you know if there is a cherry bomb strapped to your laptop, but, it can alert you if something is physically wrong with your hardware that may cause it to malfunction, assuming your hardware can provide Jopr with relevent data.

Take for example, the premature ending of Greg Hinkle's Jopr demo a few weeks ago. Greg's computer crashed for what seemed like an unknown reason - but because he was running Jopr on his laptop (which had his experimental hardware plugin deployed), he was actually able to use Jopr to figure out what the problem was. Turns out, his laptop was slowly overheating - which ended up crashing his box. Check out his blog to see what the graph looked like.

Too bad he didn't have any alerts set up on the temperature metric - he could have received an email from Jopr telling him his machine was about to blow up because his hardware reached 100 degrees :)

Classloaders Keeping Jar Files Open

If you write code that creates classloaders, you need to know about this bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5041014

It is very insidious and something I just came across myself in some code.

You normally only have to worry about this if you are writing code that creates and destroys classloaders (for example, if you have some kind of pluggable architecture where a pluggable component found in a jar file gets its own classloader, and you want that pluggable component to be hot-deployable - that is, you want to be able to overwrite or modify that jar file with updated code). In Jopr's case, this happens on the agent - each "product plugin" (e.g. the JBossAS plugin, or the Postgres plugin, etc) has its own classloader, managed separately and kept independent of other plugin classloaders (there is a dependency model in place, but ignore that for this discussion).

Well, this VM bug is so bad it seems that anytime a classloader loads in a jar file, that jar file's file descriptor remains open for the lifetime of that VM (in other words, the classloader never calls JarFile.close() for all the jar files it previously streamed content from). At least that's what the bug report infers and what I'm seeing when I was debugging this. There is a nifty tool from Timothy Quinn that he used to track issues in Glassfish, but this tool is useful to track this kind of problem for any application, not just Glassfish - in fact, I used it to debug the issue in the Jopr agent. This bug manifested itself in the Jopr agent when hot-deploying agent plugins on Windows (Windows has the "feature" of not being able to manipulate files that are locked by others). I suspect similar issues will occur on UNIX because, even though UNIX doesn't do the file locking that Windows does, the file descriptors are still open and copying a file with the same name over the opened file will probably just create a second file descriptor.

The worst part about this is - there is no real workaround. The Jopr agent has its own classloader implementation - it is very basic and extends java.net.URLClassLoader to reuse most of its functionality. But the Java classloader API has no public, protected or package-scoped method or data field that you can override or access within URLClassLoader to help workaround the problem.

To actually fix the problem, it is simple - when you know you are done with a classloader, you just need to have that classloader close all .jar files it previously had opened. Alas, there is no "close" type method on the classloader object - there is absolutely no way to tell a classloader "I am done with you, clean up any resources you have open".

Once a classloader opens a jar file, that jar file's file descriptor remains open by the operating system for the lifetime of the VM. I find this completely unacceptable - this is clearly a design flaw that slipped through the cracks when the Java API was conceived and implemented. In order to support hot-deployable Java code, one would need to destroy and recreate classloaders. The current Java implementation does not make it easy to do this (requiring people to write their own classloader implementations from scratch does not meet the definition of "easy-to-do" and doesn't that defeat the purpose of OO and code reuse anyway?).

So, how do you support hot-deployable code and not see this bug? There are two main ways to do this as I see it:

1) write your own classloader implementation that allows you to close the open file descriptors when the classloader is no longer needed
2) copy the jar files that a classloader needs to a temporary location and put the temporary jars in the classloader (NOT the original jar files). When you need to hot-deploy an updated jar file, simply copy that new jar to a new temporary location, throw away the old classloader (which still has the file descriptor open, but its the old temporary jar file) and create a new classloader that opens the new temporary jar file. This sucks because if you hot-deploy frequently, you may run into your limit of the number of allowed open file descriptors (along with the problem that Windows presents - that being you can't delete the old temporary jar files until your VM exits).

Anyway, here is some code you can use to "workaround" this issue. It is a major hack - it only works if you are running in a SUN VM and because it relies on the implementation of internal SUN classes and code, you may break in the future should SUN decide to change how these classes are implemented (however, the good thing about this code is it has no compile time dependencies on any SUN-specific classes). I tested this code on SUN's Java6 JRE.

This method needs to be placed in your classloader that extends URLClassLoader. It uses reflection to iterate over the set of currently opened jar files as found in a private data member (URLClassLoader.ucp.loaders) of the classloader you want to discard. After running this code, I verified that no more jar files are left open.

---

public void close() {
try {
   Class clazz = java.net.URLClassLoader.class;
   java.lang.reflect.Field ucp = clazz.getDeclaredField("ucp");
   ucp.setAccessible(true);
   Object sun_misc_URLClassPath = ucp.get(this);
   java.lang.reflect.Field loaders = 
      sun_misc_URLClassPath.getClass().getDeclaredField("loaders");
   loaders.setAccessible(true);
   Object java_util_Collection = loaders.get(sun_misc_URLClassPath);
   for (Object sun_misc_URLClassPath_JarLoader :
        ((java.util.Collection) java_util_Collection).toArray()) {
      try {
         java.lang.reflect.Field loader = 
            sun_misc_URLClassPath_JarLoader.getClass().getDeclaredField("jar");
         loader.setAccessible(true);
         Object java_util_jar_JarFile = 
            loader.get(sun_misc_URLClassPath_JarLoader);
         ((java.util.jar.JarFile) java_util_jar_JarFile).close();
      } catch (Throwable t) {
         // if we got this far, this is probably not a JAR loader so skip it
      }
   }
} catch (Throwable t) {
   // probably not a SUN VM
}
return;
}

---

If you happen to be using JNI (native libraries), you might also have to play games like the above to close the JNI jars too (same cavets as above apply regarding this needing to access the SUN implementation code). You can add this code to the close() method above:

---

// now do native libraries
clazz = ClassLoader.class;
java.lang.reflect.Field nativeLibraries = clazz.getDeclaredField("nativeLibraries");
nativeLibraries.setAccessible(true);
java.util.Vector java_lang_ClassLoader_NativeLibrary =
   (java.util.Vector) nativeLibraries.get(this);
for (Object lib : java_lang_ClassLoader_NativeLibrary) {
   java.lang.reflect.Method finalize =
      lib.getClass().getDeclaredMethod("finalize", new Class[0]);
   finalize.setAccessible(true);
   finalize.invoke(lib, new Object[0]);
}

---

But even if you do this, I'm still not sure everything will work due to yet more SUN VM bugs (well, I think these are all basically the same bug):

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4299094
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4642062
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4286309

In the end, the Jopr agent didn't really need to do the above. I found that in the Jopr agent code, it was creating temporary classloaders unnecessarily which was locking the plugin jars. Once I removed the unnecessary classloaders from being created, the agent hot-deployment worked just fine since the plugin jars no longer got locked. For the record, the Jopr agent uses method #2 as described above to do its hot-deployment.