Jopr 2.2 Sneak Peek

Jopr 2.2 is almost ready for release. I was going to build some nice flash demos, but there is so much new to cover, I didn't know where to begin! I wanted to get the message out quickly though, so, rather than wait for my flash demos to be completed, I decided to simply take some screen snapshots and post them up on the Jopr wiki.

Take a sneak peek here at what's coming - I think you will find Jopr 2.2 to be a great leap forward. It is much better than anything JBoss ON 1.x ever could or did deliver, and it is a major milestone even from the latest releases of JBoss ON/Jopr.

I am very proud of what we have been able to accomplish as a team - this is definitely the most feature-rich release of any previous JBoss ON/Jopr release.

Quick Java Heap Analysis

I had a need to do some quick profiling of a Java application. I didn't want to spend alot of time (or money) setting up a profiler and my app to run within that profiler's environment. But it turns out that SUN's Java 6 distribution ships with some very handy tools that you can use to do what I wanted without setting up anything special - didn't even have to pass in any special set of -D system property definitions.

First, run your application and get it into a state that you want to analyze. Then perform the following steps:

1. Run "jps" to determine the process ID (aka pid) of your Java application. BTW: this is a cool nugget in and of itself. I've been so used to doing "ps -elf | grep java" or "pgrep java" to find my running Java applications that I never bothered looking for a simpler way. "jps" dumps the pid and the simple name of the Java main class - and most of the time this is all you are looking for. But you can also get the fully qualified main class names and the argument lists - use "jps -help" to see the usage syntax. This is a very helpful tool all by itself.


2. Run "jmap -dump:format=b,file=dump.dat <pid>" where <pid> is your Java applications's pid that you found in the previous step. This will dump information about your Java VM's memory in the file "dump.dat" and will be used by jhat in the next step.


3. Run "jhat -J-Xmx512m dump.dat" to start the Java heap analysis tool (jhat). This will start an HTTP listener on port 7000. The -J option lets me configure the jhat VM, which is needed if you have a large dump to analyze.


4. Point a browser to http://localhost:7000 and begin browsing around.

This jhat tool let's you examine the objects and classes currently found within your VM. It's nothing glorious and certainly not as well featured as many of the other commercial and open-source profilers out there. But, it comes with the JDK, and requires absolutely no setup. It can't get much easier to perform a quick heap analysis than this.

Credit goes to Frank Kieviet, whose blog brought my attention to these tools.

If Your Computer Is Going To Blow Up, Let Jopr Warn You

Well, OK, Jopr won't let you know if there is a cherry bomb strapped to your laptop, but, it can alert you if something is physically wrong with your hardware that may cause it to malfunction, assuming your hardware can provide Jopr with relevent data.

Take for example, the premature ending of Greg Hinkle's Jopr demo a few weeks ago. Greg's computer crashed for what seemed like an unknown reason - but because he was running Jopr on his laptop (which had his experimental hardware plugin deployed), he was actually able to use Jopr to figure out what the problem was. Turns out, his laptop was slowly overheating - which ended up crashing his box. Check out his blog to see what the graph looked like.

Too bad he didn't have any alerts set up on the temperature metric - he could have received an email from Jopr telling him his machine was about to blow up because his hardware reached 100 degrees :)

Classloaders Keeping Jar Files Open

If you write code that creates classloaders, you need to know about this bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5041014

It is very insidious and something I just came across myself in some code.

You normally only have to worry about this if you are writing code that creates and destroys classloaders (for example, if you have some kind of pluggable architecture where a pluggable component found in a jar file gets its own classloader, and you want that pluggable component to be hot-deployable - that is, you want to be able to overwrite or modify that jar file with updated code). In Jopr's case, this happens on the agent - each "product plugin" (e.g. the JBossAS plugin, or the Postgres plugin, etc) has its own classloader, managed separately and kept independent of other plugin classloaders (there is a dependency model in place, but ignore that for this discussion).

Well, this VM bug is so bad it seems that anytime a classloader loads in a jar file, that jar file's file descriptor remains open for the lifetime of that VM (in other words, the classloader never calls JarFile.close() for all the jar files it previously streamed content from). At least that's what the bug report infers and what I'm seeing when I was debugging this. There is a nifty tool from Timothy Quinn that he used to track issues in Glassfish, but this tool is useful to track this kind of problem for any application, not just Glassfish - in fact, I used it to debug the issue in the Jopr agent. This bug manifested itself in the Jopr agent when hot-deploying agent plugins on Windows (Windows has the "feature" of not being able to manipulate files that are locked by others). I suspect similar issues will occur on UNIX because, even though UNIX doesn't do the file locking that Windows does, the file descriptors are still open and copying a file with the same name over the opened file will probably just create a second file descriptor.

The worst part about this is - there is no real workaround. The Jopr agent has its own classloader implementation - it is very basic and extends java.net.URLClassLoader to reuse most of its functionality. But the Java classloader API has no public, protected or package-scoped method or data field that you can override or access within URLClassLoader to help workaround the problem.

To actually fix the problem, it is simple - when you know you are done with a classloader, you just need to have that classloader close all .jar files it previously had opened. Alas, there is no "close" type method on the classloader object - there is absolutely no way to tell a classloader "I am done with you, clean up any resources you have open".

Once a classloader opens a jar file, that jar file's file descriptor remains open by the operating system for the lifetime of the VM. I find this completely unacceptable - this is clearly a design flaw that slipped through the cracks when the Java API was conceived and implemented. In order to support hot-deployable Java code, one would need to destroy and recreate classloaders. The current Java implementation does not make it easy to do this (requiring people to write their own classloader implementations from scratch does not meet the definition of "easy-to-do" and doesn't that defeat the purpose of OO and code reuse anyway?).

So, how do you support hot-deployable code and not see this bug? There are two main ways to do this as I see it:

1) write your own classloader implementation that allows you to close the open file descriptors when the classloader is no longer needed
2) copy the jar files that a classloader needs to a temporary location and put the temporary jars in the classloader (NOT the original jar files). When you need to hot-deploy an updated jar file, simply copy that new jar to a new temporary location, throw away the old classloader (which still has the file descriptor open, but its the old temporary jar file) and create a new classloader that opens the new temporary jar file. This sucks because if you hot-deploy frequently, you may run into your limit of the number of allowed open file descriptors (along with the problem that Windows presents - that being you can't delete the old temporary jar files until your VM exits).

Anyway, here is some code you can use to "workaround" this issue. It is a major hack - it only works if you are running in a SUN VM and because it relies on the implementation of internal SUN classes and code, you may break in the future should SUN decide to change how these classes are implemented (however, the good thing about this code is it has no compile time dependencies on any SUN-specific classes). I tested this code on SUN's Java6 JRE.

This method needs to be placed in your classloader that extends URLClassLoader. It uses reflection to iterate over the set of currently opened jar files as found in a private data member (URLClassLoader.ucp.loaders) of the classloader you want to discard. After running this code, I verified that no more jar files are left open.

---

public void close() {
try {
   Class clazz = java.net.URLClassLoader.class;
   java.lang.reflect.Field ucp = clazz.getDeclaredField("ucp");
   ucp.setAccessible(true);
   Object sun_misc_URLClassPath = ucp.get(this);
   java.lang.reflect.Field loaders = 
      sun_misc_URLClassPath.getClass().getDeclaredField("loaders");
   loaders.setAccessible(true);
   Object java_util_Collection = loaders.get(sun_misc_URLClassPath);
   for (Object sun_misc_URLClassPath_JarLoader :
        ((java.util.Collection) java_util_Collection).toArray()) {
      try {
         java.lang.reflect.Field loader = 
            sun_misc_URLClassPath_JarLoader.getClass().getDeclaredField("jar");
         loader.setAccessible(true);
         Object java_util_jar_JarFile = 
            loader.get(sun_misc_URLClassPath_JarLoader);
         ((java.util.jar.JarFile) java_util_jar_JarFile).close();
      } catch (Throwable t) {
         // if we got this far, this is probably not a JAR loader so skip it
      }
   }
} catch (Throwable t) {
   // probably not a SUN VM
}
return;
}

---

If you happen to be using JNI (native libraries), you might also have to play games like the above to close the JNI jars too (same cavets as above apply regarding this needing to access the SUN implementation code). You can add this code to the close() method above:

---

// now do native libraries
clazz = ClassLoader.class;
java.lang.reflect.Field nativeLibraries = clazz.getDeclaredField("nativeLibraries");
nativeLibraries.setAccessible(true);
java.util.Vector java_lang_ClassLoader_NativeLibrary =
   (java.util.Vector) nativeLibraries.get(this);
for (Object lib : java_lang_ClassLoader_NativeLibrary) {
   java.lang.reflect.Method finalize =
      lib.getClass().getDeclaredMethod("finalize", new Class[0]);
   finalize.setAccessible(true);
   finalize.invoke(lib, new Object[0]);
}

---

But even if you do this, I'm still not sure everything will work due to yet more SUN VM bugs (well, I think these are all basically the same bug):

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4299094
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4642062
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4286309

In the end, the Jopr agent didn't really need to do the above. I found that in the Jopr agent code, it was creating temporary classloaders unnecessarily which was locking the plugin jars. Once I removed the unnecessary classloaders from being created, the agent hot-deployment worked just fine since the plugin jars no longer got locked. For the record, the Jopr agent uses method #2 as described above to do its hot-deployment.

Jopr Agent Auto Update Complete

I have completed the agent auto-update functionality. This provides the ability for a Jopr agent running in an environment to automatically detect that it needs to be updated and does so without the need for manual intervention.

The cool thing about this is it is completely cross platform! I've testing on Windows and Linux and I see no reason why this wouldn't work on other UNIX flavors such as HP-UX, AIX and MacOS.

Here's the basics of how it works:

When a Jopr Agent tries to connect or register with a Jopr Server, that server verifies the version of that agent. If the agent is not a compatible version, the server will forbid that agent from connecting/registering and will tell the agent it needs to update itself.

At this point, the agent will shutdown all of its internals, download the latest agent update binary (either from the server or some other download location previously configured in the agent), fork another Java VM that will unpackage the new agent binary and update the old agent with the new binary. The old agent will shutdown its VM and the new agent VM will be started.

From an administrator's point of view, this all happens under the covers and automatically and the agent just looks like it goes offline for a minute or two before coming back online.

Tangential to this, is the addition to several features to the agent plugin. The agent resource metadata now includes several more child services that allow you to configure your agent without having to manually log onto the agent box (i.e. we are using Jopr to manage Jopr!). You can now even change agent JVM settings and restart the agent with those new settings (in case you need to change a -Xmx option, for example). You can read this wiki page to learn about these new plugin features.

All of this code will be forthcoming in our next RHQ/Jopr release.

Here's some additional documentation you can read if curious:

http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-PreparingYourAgentToBeAutoUpdatable
http://www.rhq-project.org/display/RHQ/Design-AgentAutoUpdate

Bundling the Jopr Agent For Deployment

One of the most requested enhancements for Jopr is for an easier way to perform agent installations. Because of this, efforts are underway to ease the pain of agent deployment.

For small Jopr environments, you can take the agent distributions as-is, install them and individually set up each agent by answering the setup questions at startup. This can be tedious the more machines you have. It would be nice if you could bundle your own "golden distro", push them out to all your machines and with no additional configuration or manual setup required, have the agents "just work".

With the current Jopr code, this is now possible. Following these steps, you can bundle your own agent into one "golden distro". We call it the "golden distro" because it's the one and only distro you will need in order to install all the agents in your environment. That distro will be able to install all your agents and have them start and configure themselves with no further setup required.

1. Unpackage the agent distribution that comes out-of-box. This is the starting point to build your own "golden distro".


2. Next, consider what, if any, customized environment variables you need to set for your agents. If there are any, edit rhq-agent-env.sh (rhq-agent-env.bat for agents that are to run on Windows). For example, if there are any -XX options you want to pass to your agent's JVM, set RHQ_AGENT_ADDITIONAL_JAVA_OPTS in the -env script appropriately.


3. The next several steps involve editing the conf/agent-configuration.xml file. So, load that file in an editor. Here is where you will preconfigure your agents in order for the agents to start up and successfully configure and initialize themselves, without requiring an admin to answer the setup quetions.


4. Set the configuration preference "rhq.agent.configuration-setup-flag" to "true". This tells the agent that, when it starts up, it should not ask any setup questions. Instead, it will immediately use what configuration preferences it has and attempt to initialize itself automatically. Of course, setting this to true infers that the rest of your agent's configuration in agent-configuration.xml is complete. But that's what we are going to make sure we do next.


5. Make sure that the "rhq.agent.name" configuration preference is left undefined (out-of-box, this setting is commented out in the agent-configuration.xml, make sure you keep it that way). Leaving this undefined will force the agent to attempt to auto-generate its own name. It does this by looking up the agent machine's fully qualified domain name and using that as the agent name. This should ensure that all agents will obtain a unique agent name (since by definition, a fully qualified domain name or IP address is unique within a network).


6. Next, determine which Jopr Server your agents will use as their "Registration Server". When new agents start up, they must communicate with a Jopr Server in order to register themselves into the Jopr environment. You must decide which of your Jopr Servers will be used to register newly installed agents (we'll call it the "Registration Server"). There is nothing special or different about a "Registration Server" compared to your other servers in your server cloud, i.e. you won't see any configuration settings or UI controls that turn on or off some "registration feature". Any Jopr Server can register any Jopr Agent. However, you must specify something in your golden distro's agent configuration so the agent knows where a server is so the agent can bootstrap itself into the Jopr environment. Once you determine which of your Jopr Servers will be the one to handle all new agent registrations, set that server's endpoint information in your golden distro's agent-configuration.xml settings:
   
   
   
   • rhq.agent.server.transport
   
   
   • rhq.agent.server.bind-port
   
   
   • rhq.agent.server.bind-address
   
   
   • rhq.agent.server.transport-params
   
   
   
   Note that this will not necessarily be the Jopr Server that will be assigned as the agent's primary server. Once the registration is complete, the agent will be assigned a server failover list, with the first server in the list to be designated as its "primary". This primary server may or may not be the same as the settings you provide here.
   


7. If you wish to assign multiple "Registration Servers" to your agent, you may do so by prepopulating a failover list and putting it in your golden distro. This allows you to have more than one Jopr Server assigned to all of your agents as Registration Servers. If the main registration server is down or is in maintenance mode, the agents will be able to failover to your secondary servers as defined in your failover list. Create a directory "data" in your distribution and place a file called "failover-list.dat" in it. Each line in that file must be of the form "address:port" where address is the IP or hostname of a Jopr Server and port is the port number the server is listening on (each server must require the same transport and transport parameters, so the "rhq.agent.server.transport[-params]" settings will be used for all servers). If you prepackage a failover list in your golden distro, you should place your main Registration Server (the one you configured in the previous step) as the top-most server in the list. Each server thereafter can be listed. If the servers at the top of the list are down, the agent will still be able to register because it just moves down the list until it finds a server it can talk to. Note that this prepopulated failover list is only temporary and is used only the first time the agent starts. Once the agent registers, it will be given a new failover list which will overwrite the list shipped in the golden distro. This is what you want because the server maintains an up-to-date failover list for each agent and you want the agent to refresh its list everytime it regsiters and starts up.


8. Make sure "rhq.communications.connector.bind-address" is left undefined (out-of-box, it is commented out in agent-configuration.xml, make sure you keep it that way). Leaving this undefined will tell the agent it needs to lookup its local IP address and use that as its bind address. It does so by using the Java API "InetAddress.getLocalHost().getCanonicalHostName()". Therefore, this uses whatever network adapters are installed on the box and chooses one from the list to determine which IP to use - usually it chooses the first network adapter that the operating system reports. (side note: this may choose an IP from the list of available IPs that is different from the one you actually want to use. You usually have to do some special configuration in your network adapters to get InetAddress.getLocalHost() to return the one you want).


9. Repackage the agent installation in a new jar - this is your "golden distro". Take this distro and push it out to all of your agent machines and they can all start up without any additional configuration or setup needed.

RHQ-496 now allows the agent to determine its name at runtime if one was not specifically given to it at setup time. This code is in trunk, but not in any current Jopr releases. So to get the ability to bundle the "golden distro" and deploy it to multiple agents, you must use a trunk build, until we release our next version.

After you have deployed your golden distro to machines in your network, you are then left with the question of how do I upgrade my agents? This then leads to the desire to perform automatic upgrades of agents already deployed and running in your environment. This feature is not fully complete, but most of the work is done and exists in trunk (see my earlier blog on this topic). To follow the development of this agent auto-update feature, watch the JIRA RHQ-110. The finished implementation will hopefully look like the design described on the RHQ wiki.

Configuration Change Detection in Jopr

A new feature has been added to trunk, a feature so interesting that it deserves its own blog.

I am sure most security-conscious administrators configure their IT infrastructure in a very specific way and they do not want anyone going onto any machine and re-configuring the machine or any of its software components willy-nilly. In fact, if something is reconfigured outside of a business' normal change-control processes, I would think administrators would want to be notified about it. It could be an innocent user mistakenly modifying something they should not be, or it could be an intruder trying to hack into the system. Being notified of configuration changes sounds like it could be a very useful thing.

Jopr now has this feature. If a plugin supports the configuration subsystem (i.e. it can retrieve configuration from its managed resource), the alert subsystem will have the ability to detect changes made in that remote managed resource and send notifications when that happens.

I've put together a demo that shows this feature in action. The scenario is quite simple - I have a Fedora box running sshd, and I do not want that sshd daemon process' configuration to change. If, for whatever reason, the configuration of sshd on the box does change, I want to be notified.

And because this config-change-notification feature is built into the core engine, any plugin that supports configuration gets this feature for free. So, if Jopr does not have a plugin that supports a particular resource whose configuration you want to monitor for changes, you can quite simply write your own plugin and deploy it into your Jopr environment and have this capability.

I can envision watching the following for configuration changes would be something people find helpful (and some of these you can already do today thanks to existing Jopr plugins):

• JBossAS's main jboss-service.xml configuration file


• JBossAS's authentication configuration (login-config.xml)


• JBossAS's datasource configuration


• /etc/hosts


• Jopr Agent's own configuration


• ...and many more...

And configuration does not have to be stored in a file on a filesystem. The Jopr configuration subsystem makes no distinction between configuration stored in a file, in a database, an LDAP server or whatever you can think of. It's the plugin's job to translate the resource's configuration into configuration data that conforms to the plugin's metadata. Once the configuration data makes it into the core engine, it is treated the same.

And finally, if a configuration change is detected, and that change was unauthorized, the Jopr user has the ability to immediately rollback that change by reverting to an earlier configuration set. This configuration-rollback feature is orthogonal to the change-notification feature, but you can see how both can be used hand-in-hand to keep a tight grip on your IT infrastructure's configuration.