Simple Tools To Analyze Thread Dumps

When debugging problems in, or analyzing the performance of, your Java applications, you sometimes have to sift through one or more Java thread dumps. Depending on the size and complexity of your Java applications, these thread dumps can be lengthy. It is hard to use a text editor and scan it for things like long running threads, blocked threads and deadlocked threads.

Clearly, you can use sophisticated Java analysis tools (such as JProfiler) for such a job. However, there are instances where you simply don't have access to such tools. For example, if you are supporting a remote customer and all you have is the ability to ask the customer to use "jstack" or to send a SIGQUIT signal to the Java app process. You usually cannot ask a customer to stop an app and restart it with the appropriate debugging system properties and options and then connect to it with a tool like JProfiler (which also assumes they even have such a tool installed and available).

So, what you normally have to fall back on is obtaining a thread dump (which is nothing more than normal text). Once that thread dump is captured, you can save it as a .txt file, send it around via email, post it via pastebin or even send it over IM (if its small enough).

A simple way you can get a thread dump is via the "jstack" utility that ships with the JDK. Once you know the pid of your Java JVM process (to get this information, you can use the standard "ps" utility or you can use "jps", which is another JDK utility), you simply tell "jstack" to output a thread dump associated with that process which you redirect to a file:

---

# find the process IDs of all of my running Java applications
$ jps
21147 Jps
16640 Main

# take an initial thread dump snapshot of my "Main" Java application
$ jstack 16640 > /tmp/my-thread-dump.txt

# take a second thread dump snapshot and append it to the original
$ jstack 16640 >> /tmp/my-thread-dump.txt

---

The question then is - how do you analyze it? I recently came across a couple of small, free, easy-to-install, easy-to-run GUI tools that help assist in analyzing thread dumps. The first is Samurai and the second is TDA (Thread Dump Analyzer). They are easy to download and install:

• Samurai is downloaded as a simple samurai.jar file that you start via "java -jar samurai.jar"
• TDA is downloaded as a zip file that you unzip which again gives you a simple tda.jar that you start via "java -jar tda.jar"

I like the extremely simple way to install and run these.

Each tool has basically the same way to start - you just tell it to open up a text file that contains one or more thread dumps in them (such as the "my-thread-dump.txt" file that my example above captured).

Each tool has its own way of displaying the threads:

Samurai Main Screen That Shows Threads and Their States

TDA Main Screen That Shows Threads and Their States

Its fairly obvious how to use these tools, so I won't bore you with details. Just click around, and you'll get it fairly quickly. They both have relatively intuitive user interfaces. There isn't much too them - don't expect any artificial intelligence to analyze your threads - but you can use them to navigate through the stack traces of all the threads easier than scrolling up and down the base text file.

Notice also that both tools can analyze multiple thread dumps if it detects more than one in your thread dump text file (in my snapshots above, I had three thread dumps). It is excruciating trying to do this by hand (scrolling through multiple thread dumps in a single file) so this is when these tools are really of great help. I also like how Samurai, in one view, shows the states of the threads that are common in the multiple thread dumps. So you can see which threads were running, blocked, etc and which ones changed states between the different thread dumps. However, Samurai loses points because, if thread names are really long, you are forced to scroll horizontally to see the names of the shorter-named threads and to see the states.

One final thing I wanted to show is how they can quickly point out the threads that are deadlocked. This is where manually scanning thread dumps with your eyes can be slow, so these tools can definitely save some time. I took some deadlock code I found online and ran it. Using "jstack", I captured a thread dump and used both tools to see how they report the deadlock. Here's what to expect if you analyze thread dumps that have deadlocks in them:

Samurai Table of Threads Showing The Two That Are Deadlocked

Samurai Showing In Red The Stacktraces of the Deadlocked Threads

TDA Showing The Deadlocked Threads and Their Stacktraces

That's all I wanted to show. It looks like either of these tools can be useful to do simple analysis of thread dumps and to help quickly determine if a thread dump has one or more deadlocks.

EJB Calltime Monitoring

I wanted to give a brief illustration of RHQ's calltime monitoring feature by showing how it is used to collect EJB calltime statistics. The idea here is that I have an EJB application (in my example, I am using the RHQ Server itself as the monitored application!) and I want to see calltime statistics for the EJB method calls being made. For example, in my EJB application being monitored, I have many EJB Stateless Session beans (SLSBs). I want to see how my EJB SLSBs are behaving by looking at how many times my EJB SLSB methods are called and how efficient they are (that is, I want to see the maximum, minimum and average time each EJB SLSB method took).

So, first what I do is go to my EJB SLSB resources that are in inventory and I enable the "Method Invocation Time" calltime metric. You can do this on an individual EJB resource or you can do a bulk change of all your EJB's schedules by navigating to your EJB autogroup and changing the schedule in the autogroup's Schedules subtab:

At this point, the RHQ Agent will collect the information and send it up to the RHQ Server just like any other metric collection. Over time, I can now examine my EJB SLSB calltime statistics by traversing to my EJB resource's Calltime subtab:

I can even look at an aggregate view of all my EJBs via the EJB autogroup. In the RHQ GUI, you will see in the left hand tree all of my EJB SLSBs are children of the autogroup node "EJB3 Session Beans". If I select that autogroup node and navigate to its Calltime subtab, I can see all of the measurements for all of my EJB SLSBs:

Note that this calltime measurement collection feature is not specific to EJBs, or the JBossAS 4 plugin. This is a generic subsystem supported by any plugin that wants to enable this feature. If you want to write your own custom plugin that wants to monitor calltime-like statistics (say, for HTTP URL endpoints or any other type of infrastructure that has a "duration" type metric that can be collected) you can utilize the calltime subsystem to collect your information and let the RHQ Server store it and display it in this Calltime subtab for your resource.

Monitoring Custom Data from DB Queries

There is an interesting little feature in RHQ that I thought I would quickly mention here. Specifically, it's a feature of the Postgres plugin that let's you track metrics from any generic query you specify.

Suppose you have data in your database and you want to expose that data as a metric. For example, suppose you want to track the total number of users that are currently logged into your application and that information is tucked away in some database table that you can query.

Import your Postgres database into RHQ and manually add a "Query" resource under your Postgres Database Server resource (see the image below where the "Import" menu provides you with the names of the resource types you can manually add as a child to the database server resource - in this case, the only option is the Query resource type).



When you "import" this Query resource through the manual add feature, you will be asked for, among other things, the query that you want to execute that extracts your metric data.



Once you do, you'll have a new Query resource in your RHQ inventory that is now tracking your metric value like any other metric (e.g. you will be able to see the historical values of your data in the graph on the Monitoring tab; you'll be able to alert on those values; etc.)



The one quirky thing about this is the query needs to return a single row of two columns - the first column must have a value of "metricColumn" (literally) and the second column must be a numeric value. To follow the earlier example (tracking the number of users currently logged in), it could be something like:

SELECT 'metricColumn', count(id) FROM my_application_user WHERE is_logged_in = true

That's it. A pretty simple feature, but it seems like this could have a wide range of uses. Hopefully, this little tidbit can spark an idea in your head about how you can use this feature while monitoring your systems.

Sending RHQ Alerts over XMPP

Rafael has created a very cool server plugin to allow RHQ to send alerts to his Google account over XMPP. Not only that, but he was able to use that same XMPP channel to send commands to the RHQ Server, like another kind of CLI.

Watch his demo to see how it works. Very awesome!

This is exactly the kind of innovation we envisioned the community being able to do via the server plugins, as I mentioned in my earlier blog entry titled "RHQ Server Plugins - Innovation Made Easy"

Nice job, Rafael!

Managing Compliance Across Multiple Resources

In the latest RHQ project release, and Red Hat's JBoss Operations Network product, a new feature called "drift monitoring" has been introduced.

If you've seen my previous blog, it (along with its demo) described how you can monitor changes in files belonging to a single resource. If a resource's files deviate from a trusted snapshot, that resource will be considered out of compliance and the UI will indicate when this happens.

This pinning/compliance feature within the drift subsystem can be combined with drift templating to allow you to pin a single snapshot of content to multiple resources. This allows you to have a single snapshot which all resources can share. In other words, if you have a cluster of app servers and they all have the same web application deployed, you can pin a snapshot of the in-compliant web application to a drift template that all of your app servers use when they scan for drift. Thus, for an entire cluster, if one of the servers in that cluster drifts away from that shared, in-compliant snapshot, that server will be flagged as "out of compliance".

To see how this works, see my "Managing Compliance Across Multiple Resources" demo.

Managing Compliance with Drift Pinning

In the latest RHQ project release, and Red Hat's JBoss Operations Network product, a new feature called "drift monitoring" has been introduced.

Drift monitoring provides the ability to monitor changes in files and to determine if those files are in or out of compliance with a desired state. In other words, if I installed an application and someone changes files in that installation, I can be told when those changes occurred and I can analyze those changes.

I put together another demo for my "drift demo series" that illustrates this concept:

• Managing Compliance with Drift Pinning

How it works is this - suppose you have a set of files on your machine and you don't want those files changed. In other words, the files you have now are "in compliance" with what you want and this set of in compliance files should not be touched. In RHQ/JON, you would create a new drift definition and pin that definition to your current snapshot of files. This pinning effectively marks that snapshot of files as the "in compliant" version. Any changes now made to those files being tracked will be considered drift and out of compliance. In the graphical user interface, you can see what has gone out of compliance and you can drill down to see what files drifted and even what parts of those files drifted.

This pinning/compliance feature within the drift subsystem can be combined with drift templating to allow you to pin a single snapshot of content to multiple resources allowing you to have a single snapshot which all resources can share. In other words, if I have a cluster of app servers and they all have the same web application deployed, I can pin a snapshot of the in-compliant web application to a drift template that all my app servers use when they scan for drift. Thus, for my entire cluster, if one of my servers in that cluster drifts away from that shared, in-compliant snapshot, that server will be flagged as "out of compliance". I will be posting another demo to illustrate this concept in the near future.

JBoss ON 3.0 Released and a Drift Demo

Red Hat has released the next version of JBoss Operations Network. This is version 3.0 and incorporates the new look and feel of the GWT user interface that RHQ introduced recently. The bundle UI has been enhanced a bit as well (it was the only GWT-based portion of the UI that was in JBoss ON 2.4.1).

A new feature introduced in this JBoss ON 3.0 release is drift monitoring. If you have been following the progress of the RHQ upstream project, you'll know that this drift monitoring feature provides administrators with the ability to keep on the look out for unintended changes (either accidental or malicious) to your installed software and other file content.

I posted a demo showing the basics of the new drift feature:

• "The Basics of Drift Monitoring" Demo

I plan on posting some more demos on drift in the near future.