Using Byteman To Analyze the RHQ Agent

I recently had a need to run some quick and dirty performance testing over a few method calls in one of the RHQ plugins that was deployed in my agent. I didn't have a profiler installed and ready to go so I used Byteman. I quickly wrote up a couple of Byteman rules (thanks to Jay S. for correcting some problems I had with those rules), ran the RHQ Agent with Byteman installed and got some really good data.

It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.

Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.

I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.

I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).

Read the wiki page on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.