Using Byteman To Analyze the RHQ Agent

I recently had a need to run some quick and dirty performance testing over a few method calls in one of the RHQ plugins that was deployed in my agent. I didn't have a profiler installed and ready to go so I used Byteman. I quickly wrote up a couple of Byteman rules (thanks to Jay S. for correcting some problems I had with those rules), ran the RHQ Agent with Byteman installed and got some really good data.

It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.

Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.

I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.

I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).

Read the wiki page on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.

Remotely Installing An Agent

One new feature coming soon to RHQ is the ability to remotely install an RHQ Agent on to any machine in your network (with the caveat that the remote machine must be accessible via SSH).

What this now means is you do not have to manually log onto a remote machine and do the tasks of downloading the agent update binary distribution, installing it and running it. RHQ can now do this for you all remotely - all you need to provide to the RHQ GUI is the machine name, your SSH credentials and the location where you want to install the agent.

I think this could still use some enhancements - there is no way to customize each agent's configuration (i.e. providing custom answers to the startup setup questions). But that's for another day. If you want to customize the new agent, you can do so by just importing the RHQ Agent resource into inventory and going to its Configuration tab and adjusting its configuration (don't forget to restart the agent so it can pick up the changed configuration- you can restart the agent using this new RHQ GUI page! I'll talk about that next).

This mechanism will also allow you to stop and start an existing agent that is already installed. This is very useful if you have not started the agent yet but wish to bring it back online, or for those cases where you have not commited the RHQ Agent resource in inventory yet but still need a way to shutdown or restart an agent.

I put together a flash demo that shows this stuff in action.

RHQ 3.0.0 Has Been Released

The RHQ team is proud to announce the immediate availability of RHQ 3.0.0.

This release features several months of hard work by the development team and external contributors. Many of the changes have been already made available in the past through seven community releases.

You can browse the full release notes on the RHQ wiki.

The release can be downloaded via the RHQ web site or directly from SourceForge.

Download it and try it out. The documentation wiki has full install instructions.

Provisioning Content via RHQ

Work has been steadily progressing in RHQ to introduce a new "provisioning" feature. Our beta will soon be released, and as part of our getting ready of that release, I put together a quick flash demo that illustrates our current state of this provisioning feature.

There is still much more work to get done - the new GWT GUI is continually improving and we need more provisioning support for things like starting, stopping and installing services being provisioning, orchestration between machines in a cluster (to support provisioning software within a cluster) and other things. But what we have is a good base framework to build upon - watch the demo if you want to see what we have now.

For more information, we have documentation on our wiki:

http://rhq-project.org/display/JOPR2/Provisioning

Hibernate Performance Monitoring

Joseph Marques has a very good blog (good in both substance and technical details) on how you can analyze the performance of your Hibernate application through the use of a simple set of utility classes. He even goes into how you can do this inside your EJB3 app. We do this when analyzing performance in RHQ (in addition to using RHQ itself to do some analysis). Joe's code examples show that it's a solution that is easily transferable to any Java application that uses Hibernate as its ORM layer.

Monitoring JBossAS Clusters with RHQ

CirrAS is an effort to automatically deploy clustered JBoss AS servers in the Cloud. With so much focus on cloud computing these days, keep an eye on this CirrAS project.

The CirrAS guys are currently using RHQ to monitor every JBossAS instance in the cluster and recently utilized the RHQ CLI to perform some automation steps to get some RHQ things setup. For a description of what they did, see http://cloudpress.org/2010/01/29/rhq-cli-configuring-and-importing-resources/. Eventually, the CirrAS team wants to use RHQ to manage every node in the JBossAS cluster (deploy app, restart, etc).

Idiot Lesson On The Types Of SQL JOINs

I'm going to take a departure from my usual topics and talk briefly about a very basic concept - SQL JOINs. I'm doing this mainly for my benefit - I wanted to write this little nugget down so I don't forget it, and what better place to do so than my blog? :)

What's the difference between INNER, LEFT and RIGHT JOINs?

It seems every so often, I forget the answer to that question - I usually get confused ("what happens if there is a null - do I get a row in my results or not?"). Now, I'm sure it is easy for most people to remember how these JOINs work, but I always seem to forget. However, recently I got a good idiot lesson from Joseph Marques on this; he gave me a very short and easy way to remember the rules of these three types of JOINs. It was such an elegant way to explain it that I promised myself I would write it down so I don't forget it, so here it goes (again, I'm sure everyone knows this, and I'm sure I was even told this myself at some point in the past, but of course, I don't remember):

JOINs can be thought of in the context of a Venn diagram. You have two circles A and B. These circles partially overlap. (circles A and B represent your table data - when you join the two circles/tables together, the overlap represents relationships between the data.)

• The overlap is the INNER join - it contains only the elements in A that also have associated B elements.


• The A circle is the LEFT join - it contains all A elements regardless of whether they have associated B elements.


• The B circle is the RIGHT join - it contains all B elements regardless of whether they have associated A elements.

Note: A LEFT or RIGHT JOIN are OUTER JOINs - sometimes you see these specified as "LEFT OUTER JOIN" or "RIGHT OUTER JOIN". Looking at the Venn diagram you can see why - they include the data outside of the inner overlap area (but also including the inner data). If you just see the word "JOIN" without a specifier, it typically is referring to an "INNER JOIN".

Below is an example complete with this Venn diagram visualization:

I have a Person table containing three people: John, Mary and Bob. I have a Phone table containing three phones that have extensions of 987, 555 and 123. John and Bob have been assigned phones (I can call John on extension 987, Bob on extension 123). Mary is not assigned a phone. The phone with the extension 555 is not assigned to any person.

The INNER join:

select name,dial_number
from person
inner join phone on person.phone_id=phone.phone_id

returns results as found in the overlapping area (John:987 and Bob:123).

The LEFT join:

select name,dial_number
from person
left join phone on person.phone_id=phone.phone_id

returns results as found in the entire left circle (John:987, Mary:null, Bob:123). Because the left table is the Person table, I have one row for each person, regardless of whether they have a phone assigned or not.

The RIGHT join:

select name,dial_number
from person
right join phone on person.phone_id=phone.phone_id

returns results as found in the entire right circle (John:987, null:555, Bob:123). Because the right table is the Phone table, I have one row for each phone, regardless of whether a person is assigned a phone or not.

I don't know why it is so hard for me to remember this - it's so obvious when you see it written down like that. But, nevertheless, I find it helpful to refresh my memory with that Venn diagram explanation whenever I experience a brain freeze while developing queries.

Java Decompiler For Java5 and 6 - A Jad Replacement!

A few years ago, JAD was the Java Decompiler everyone used. However, it's essentially a dead project now - if you still have it, you'll know it can only decompile Java 1.4 class files.

However, the good news is that a new Java decompiler has come along to take its place - JD. I used this on Windows before, but I'm now on Fedora 11 and I never installed this new Java Decompiler there.

Thanks to Ian's blog, it took me less than 5 minutes to install the JD Java decompiler and its GUI front end (JD-GUI) on my F11 64-bit box. I had to issue this yum command, as per Ian's instructions, to get some libraries installed needed by JD-GUI:

yum install libcanberra-gtk2.i586 PackageKit-gtk-module.i586 gtk2-engines.i586

I then just un-tar'ed the JD-GUI download and it ran fine.

I also took the time to install the JD-Eclipse plugin for my Eclipse 3.5. That worked flawlessly on my first attempt.

So now I have a nice Java decompiler on my F11 box, both as a standalone GUI and as an Eclipse plugin.

Byteman Plugin Revisited

A couple of months ago, I came across Byteman and liked it so much, I wanted to see how easy it would be to manage it, so I wrote a quick RHQ agent plugin prototype for it and blogged about it.

Since I had some time over Christmas, I decided to revisit that prototype and build it out further. The finished product can probably be considered a good 1.0 version of a fully featured RHQ Byteman plugin.

With this RHQ-Byteman integration, you can now do remote byte-code injection into any Java virtual machine that has a Byteman agent running inside of it. Simply write your Byteman rules, store them in one or more scripts and file-upload those scripts from your browser to the RHQ Server. The RHQ Server will then stream down those script files to the remote Byteman agent, which then injects the code directly in its Java virtual machine. It is extremely easy to run a VM with Byteman inside of it - all you need is to pass a -javaagent VM argument to Java when you start it; something like:

-javaagent:/opt/byteman/lib/byteman.jar=listener:true

The uses for this kind of thing are boundless - from tracing how long it takes for a Java method to execute to testing an application's fault tolerance by forcing test exceptions to be thrown. In the future, I plan on attempting to write an extension to the Byteman plugin to allow me to trace JDBC calls and create reports based on the data I get back. In fact, work on that has already begun; the bulk of the RHQ plugin code is already written, I just have to figure out how to write the proper Byteman rules to get it to do what I want. Once I complete this "JDBC Trace" RHQ agent plugin, I'll blog/demo it.

If you are interested in this new RHQ integration with Byteman, watch the flash demo.

Note that in order to finish this Byteman plugin, I had to add some additional capabilities within Byteman itself. The bulk of the changes manifested themselves as new API methods to the Byteman client - the "Submit" class. These new features allow Byteman to be easily managed, not just by RHQ, but by any management tool that wants to integrate with Byteman. I committed this new code to Byteman's SVN repository which should make these new features available in its next release. Because the new Byteman is not released yet, nor available in the JBoss maven repository, I placed the RHQ Byteman plugin code in its own git branch. Once the next Byteman version is released, I'll merge my code into RHQ's master branch.