tag:blogger.com,1999:blog-69440501875682410592012-01-27T07:57:16.340-08:00John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.comBlogger511100tag:blogger.com,1999:blog-6944050187568241059.post-51285677536472444052012-01-27T07:49:00.000-08:002012-01-27T07:57:16.475-08:00Rafael has created a very cool server plugin to allow RHQ to send alerts to his Google account over XMPP. Not only that, but he was able to use that same XMPP channel to send commands to the RHQ Server, like another kind of CLI.<br /><a href="http://vimeo.com/35730049"><br />Watch his demo</a> to see how it works. Very awesome!<br /><br />This is exactly the kind of innovation we envisioned the community being able to do via the server plugins, as I mentioned in my earlier blog entry titled "<a href="/2009/12/rhq-server-plugins-innovation-made-easy.html">RHQ Server Plugins - Innovation Made Easy</a>"<br /><br />Nice job, Rafael!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5128567753647244405?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-87252274229839132242011-12-16T10:48:00.000-08:002011-12-16T12:53:54.970-08:00In the latest <a href="http://www.rhq-project.org/">RHQ</a> project release, and Red Hat's <a href="http://www.jboss.com/products/jbosson/">JBoss Operations Network</a> product, a new feature called "<a href="http://rhq-project.org/display/JOPR2/Drift">drift monitoring</a>" has been introduced.<br /><br />If you've seen my <a href="http://www.blogger.com/2011/12/managing-compliance-with-drift-pinning.html">previous blog</a>, it (along with its <a href="http://mazz.fedorapeople.org/demos/drift/drift-pinning.ogv">demo</a>) described how you can monitor changes in files belonging to a single resource. If a resource's files deviate from a trusted snapshot, that resource will be considered out of compliance and the UI will indicate when this happens.<br /><br />This pinning/compliance feature within the drift subsystem can be combined with drift templating to allow you to pin a single snapshot of content to multiple resources. This allows you to have a single snapshot which all resources can share. In other words, if you have a cluster of app servers and they all have the same web application deployed, you can pin a snapshot of the in-compliant web application to a drift template that all of your app servers use when they scan for drift. Thus, for an entire cluster, if one of the servers in that cluster drifts away from that shared, in-compliant snapshot, that server will be flagged as "out of compliance".<br /><br />To see how this works, see my "<a href="http://mazz.fedorapeople.org/demos/drift/drift-pinned-template.ogv">Managing Compliance Across Multiple Resources</a>" demo.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-8725227422983913224?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-78340153075770505322011-12-15T08:16:00.000-08:002011-12-15T08:37:19.752-08:00In the latest <a href="http://www.rhq-project.org/">RHQ</a> project release, and Red Hat's <a href="http://www.jboss.com/products/jbosson/">JBoss Operations Network</a> product, a new feature called "<a href="http://rhq-project.org/display/JOPR2/Drift">drift monitoring</a>" has been introduced.<br /><br />Drift monitoring provides the ability to monitor changes in files and to determine if those files are in or out of compliance with a desired state. In other words, if I installed an application and someone changes files in that installation, I can be told when those changes occurred and I can analyze those changes.<br /><br />I put together another demo for my "drift demo series" that illustrates this concept:<br /><ul><li><a href="http://mazz.fedorapeople.org/demos/drift/drift-pinning.ogv">Managing Compliance with Drift Pinning</a></li></ul><br />How it works is this - suppose you have a set of files on your machine and you don't want those files changed. In other words, the files you have now are "in compliance" with what you want and this set of in compliance files should not be touched. In RHQ/JON, you would create a new drift definition and <span style="font-style:italic;">pin</span> that definition to your current snapshot of files. This pinning effectively marks that snapshot of files as the "in compliant" version. Any changes now made to those files being tracked will be considered drift and out of compliance. In the graphical user interface, you can see what has gone out of compliance and you can drill down to see what files drifted and even what parts of those files drifted.<br /><br />This pinning/compliance feature within the drift subsystem can be combined with drift templating to allow you to pin a single snapshot of content to multiple resources allowing you to have a single snapshot which all resources can share. In other words, if I have a cluster of app servers and they all have the same web application deployed, I can pin a snapshot of the in-compliant web application to a drift template that all my app servers use when they scan for drift. Thus, for my entire cluster, if one of my servers in that cluster drifts away from that shared, in-compliant snapshot, that server will be flagged as "out of compliance". I will be posting another demo to illustrate this concept in the near future.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-7834015307577050532?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-37547993221949001612011-12-08T10:41:00.000-08:002011-12-14T07:10:48.939-08:00Red Hat <a href="http://www.redhat.com/about/news/prarchive/2011/red-hat-middleware-product-updates-enhance-manageability-and-usability-for-enterprise-IT-professionals">has released</a> the next version of <a href="http://www.jboss.com/products/jbosson/">JBoss Operations Network</a>. This is version 3.0 and incorporates the new look and feel of the GWT user interface that <a href="http://www.rhq-project.org/">RHQ</a> introduced recently. The bundle UI has been enhanced a bit as well (it was the only GWT-based portion of the UI that was in JBoss ON 2.4.1).<br /><br />A new feature introduced in this JBoss ON 3.0 release is <a href="http://rhq-project.org/display/JOPR2/Drift">drift monitoring</a>. If you have been following the progress of the RHQ upstream project, you'll know that this drift monitoring feature provides administrators with the ability to keep on the look out for unintended changes (either accidental or malicious) to your installed software and other file content.<br /><br />I posted a demo showing the basics of the new drift feature:<br /><ul><li><a href="http://mazz.fedorapeople.org/demos/drift/drift-basics.ogv">"The Basics of Drift Monitoring" Demo</a></li></ul>I plan on posting some more demos on drift in the near future.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3754799322194900161?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-34627868980743614412011-10-06T05:23:00.000-07:002011-10-06T06:13:32.062-07:00I just came across an insidious problem in some code that went undetected for a while because it was hard to detect. I only found it through a unit test I ended up writing and I happen to come across it (I guess unit tests <span style="font-style:italic;">are</span> a good thing!).<br /><br />The problem has to do with paging through database data and the query used to do the paging.<br /><br />The code in question was using a JPA query that looked like this:<br /><blockquote><pre>SELECT r.id, r.foo<br /> FROM Table t<br /> JOIN t.relation r<br /> WHERE t.id = :tid</pre></blockquote><br />This query has the potential to return many hundreds of rows. The code using this query, therefore, used paging to retrieve chunks of the data, one page at a time. This paging code did so by setting the appropriate start/max rows via the JPA API:<br /><blockquote><pre>query.setFirstResult(startRow);<br />query.setMaxResults(pageSize);</pre></blockquote><br />In my unit test, I set up test data that contained over 1,000 rows to be returned by this query. Much to my surprise, my unit test was failing because it wasn't getting the full amount of rows expected. Worse yet, after multiple runs of my test (which paged through what was supposed to be the entire data set), it was getting what seemed to be a random number of rows back for each run - it wasn't even consistent! Sometimes the unit test would get back 800 rows, other times 900. Sometimes it would work with 2 or 3 pages of data, but would be broken with more pages. The reason why this was hard to detect (without a unit test) was because it only showed up when there was many multiple pages of data to be retrieved (thus it required first to have a lot of data and second it required you to actually scan the data and notice you had less rows than expected).<br /><br />It turns out that some database implementations do not guarantee the order in which a query returns back results without a defined ORDER BY clause in the query. In this case here, because the query didn't have a ORDER BY clause, each time the paging code ran a query to obtain a page of data, the query was returning the data in a different order. Code that ran the query multiple times to obtain different pages (that is, with different firstRow/maxResults settings) wasn't guaranteed that it wouldn't get back duplicate rows from a previous query execution. <br /><br />I found this behavior in both Postgres 8 and Oracle 10.<br /><br />The good news is once I added an ORDER BY clause to that query, all worked well and my unit test started passing:<br /><blockquote><pre>SELECT r.id, r.foo<br /> FROM Table t<br /> JOIN t.relation r<br /> WHERE t.id = :tid<br /> <b>ORDER BY r.id</b></pre></blockquote><br />So the moral of the story is: if you ever use paging queries, always double check your results by testing with <span style="font-style:italic;">many multiple</span> pages of data and if you see problems, put an ORDER BY clause in your query to see if it fixes the issue.<br /><br />NOTE: I found the following two resources that talk about this issue (I'm sure there are more). See:<br /><ul><li><a href="http://www.jroller.com/sjivan/entry/hibernate_and_oracle_pagination_gotcha">Hibernate and Oracle pagination gotcha</a></li><li><a href="http://community.jboss.org/wiki/Pagination">Hibernate Pagination</a></li></ul><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3462786898074361441?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-138666446769724392011-07-06T06:46:00.000-07:002011-07-06T10:07:08.794-07:00We recently hit a problem in <a href="http://rhq-project.org/display/JOPR2/Home">RHQ</a> that required us to learn about a feature in <a href="http://code.google.com/webtoolkit/">GWT</a> (specifically the GWT compiler) that was very useful and I thought I would blog about it.<br /><br />First, some background. In RHQ, we have split up the source code into several "modules". Each module is built by <a href="http://maven.apache.org/">Maven</a> and artifacts are produced (for example, some modules will output a jar file after the build completes). Dependent artifacts are built first, then modules that depend on other modules' artifacts are built afterwards - Maven knows how to maintain the proper dependent hierarchy so it can build modules in the proper order. After our entire suite of modules are built, the build system assembles all the module artifacts into the RHQ distribution. As you can see, there is nothing special here, any complex application needs a build system that does the same thing.<br /><br />One of RHQ's modules is what we call the <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/core/domain;hb=master">"domain" module</a>. This is simply the module that contains the source code for all of our domain objects that map to our data model (in other words, the domain objects simply represent the set of all of our database entities, such as Users, or Roles, or Resources, or Alerts, etc.)<br /><br />These domain objects are essentially the basic building blocks on which all of RHQ is built. These domain objects are used all throughout the RHQ codebase - in the agent, in the server, in the remote CLI and in the GWT client. All of their modules depend on the domain module - the domain module is one of the first ones built.<br /><br />Since our GWT client needs these domain objects, the domain module needs to not only be passed through the Java compiler, it needs a second pass through the GWT compiler.<br /><br />Here is where the problem comes in. Some of our domain objects require that they import certain Java classes that <a href="http://code.google.com/webtoolkit/doc/latest/DevGuideCodingBasicsCompatibility.html#differences">GWT doesn't support</a>. Because the domain objects are used everywhere, they end up needing to provide functionality that is required by the server and agent, not just the GWT client. But this functionality sometimes requires the use of certain JRE features that are not emulated, and thus not available, in the GWT runtime (things such as java.io or java.sql classes).<br /><br />But how can we do this? If we import these GWT-unsupported JRE classes into some the domain classes, the GWT client cannot load the domain jar and the GWT client becomes dead. But without those JRE classes, the functionality that the domain module needs to provide to the other modules (like the server or agent) becomes broken. It is a <a href="http://en.wikipedia.org/wiki/Catch-22_%28logic%29">catch-22</a>. So, the question becomes, how can we provide all of the domain module functionality to all dependent modules, but remove only portions of it that are not GWT compatible from the GWT client module?<br /><br />One idea was to create another module - a second domain module - that was compatible with the GWT client (essentially it would be the original domain module, minus the GWT-incompatible code). We could possibly do this by refactoring the original domain objects and creatively using inheritance. But it would give us two domain jars. Adding yet another module to the build isn't something we wanted to do.<br /><br />As it turns out, after being perplexed at how best to design around this problem, we found out that GWT provides a very, very easy solution to this. You can tell the GWT compiler to <a href="http://code.google.com/webtoolkit/doc/latest/DevGuideOrganizingProjects.html#DevGuidePathFiltering">filter out and exclude certain classes</a> when it compiles the Java source into GWT Javascript. Since those classes don't get compiled into GWT Javascript, the GWT client never sees them or tries to load them. Without having to split our domain module into two separate modules, we get the effect of having two different domain libraries - one for the GWT client and one for the rest of the RHQ system.<br /><br />Now we can use normal Java inheritance to share common code while at the same time we ensure that only certain subclasses will use the non-GWT-supported features of the JRE. We can then exclude those subclasses from the GWT compilation thus allowing the GWT client to load the domain module, minus those classes it couldn't use anyway.<br /><br />To use this feature, we had to add the &lt;exclude&gt; XML element to our GWT module's XML file in order to tell the GWT compiler which Java classes to ignore:<br /><br /><pre><blockquote><br />&lt;entry-point class="org.rhq.core.client.RHQDomain" /&gt;<br />&lt;source path="domain"&gt;<br /> &lt;exclude name="**/DriftFileBits.*" /&gt;<br />&lt;/source&gt;<br /></blockquote></pre><br /><br />This essentially tells the GWT compiler to compile all of our domain module's classes <span style="font-style: italic;">except for</span> the DriftFileBits class. If, in the future, we have to introduce other classes that are not supported by GWT, we can add more &lt;exclude&gt; elements to filter out those additional Java packages or individual classes.<br /><br />One negative aspect of this is that we have to be careful to not "leak" references to those excluded classes into our GWT API or GWT implementation classes. But even if we do, it is caught rather quickly when the GWT client attempts to load the application.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-13866644676972439?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-21487869786914832272011-06-20T19:51:00.000-07:002011-06-20T20:11:58.506-07:00I recently completed the initial implementation of <a href="https://bugzilla.redhat.com/show_bug.cgi?id=644328">a feature</a> that folks have been asking for lately, so I decided to put together a <a href="http://mazz.fedorapeople.org/demos/bundles/bundles-nonplatforms.ogg">quick demo</a> to show it.<br /><br />First, some background. The initial <a href="http://rhq-project.org/display/JOPR2/Provisioning">RHQ Bundle Provisioning feature</a> (of which I have <a href="http://management-platform.blogspot.com/2011/01/bundle-provisioning-via-rhq.html">blogged about in the past</a>) supported the ability to deploy bundle distributions (which are, essentially, just files of generic content) to a group of platforms. That is, given a set of files bundled together, RHQ can ship out those files to a group of platform resources where the agents on those platforms will unbundle the files to the root file system.<br /><br />This new feature (submitted as enhancement request <a href="https://bugzilla.redhat.com/show_bug.cgi?id=644328">BZ 644328</a>) now allows you to target your bundle deployments to a group of <span style="font-style: italic;">non-platform</span> resources if you wish. For example (as the demo shows), we can now target bundle deployments to JBossAS Server resources. This means you can bundle up a WAR or an EAR, and push that bundle to a group of JBoss AS Servers such that the WAR or EAR gets deployed directly to the deploy/ directory (and hence gets deployed into your application servers).<br /><br />The demo is 10 minutes long and shows what a simple workflow looks like to deploy a WAR bundle to a group of JBoss EAP application servers.<br /><br /><a href="http://mazz.fedorapeople.org/demos/bundles/bundles-nonplatforms.ogg">View the demo here</a>.<br /><br />[Note: unlike my previous demos which were built on a Windows laptop using Wink and formatted as Flash, I decided to try Istanbul on my Fedora desktop. So the demo is formatted in .ogg format, as opposed to Flash. Hopefully, this doesn't limit the audience that is able to view the demo.]<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-2148786978691483227?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-46170728618930781502011-05-23T12:48:00.000-07:002011-09-21T13:59:00.088-07:00One thing we encountered pretty quickly when we started our GWT work was the fact that you can't serialize objects over the wire from server to GWT client if those objects were obtained via a Hibernate/JPA entity manager.<br /><br />If you've ever worked with Hibernate/JPA, you'll know that when you get back entity POJOs whose fields are not loaded (i.e. marked for lazy loading and you didn't ask for the data to be loaded), your entity POJO instance will have Hibernate proxies where you would expect a "null" object to be (this is to allow you to load the data later, if your object is still attached to the entity manager session).<br /><br />Having these proxies even after leaving a JPA entity manager session is a problem in the GWT world because the GWT client sitting in your browser doesn't have Hibernate classes available to it! Trying to send these entity POJO instances that have references to Hibernate proxies causes serialization errors and your GWT client will fail to operate properly.<br /><br />This is a known issue and <a href="http://code.google.com/webtoolkit/articles/using_gwt_with_hibernate.html">is discussed here</a>.<br /><br />We pretty quickly decided against using DTOs. As that page above mentioned, "if you have many Hibernate objects that need to be translated, the DTO / copy method creation process can be quite a hassle". We have a lot of domain objects that are used server side in RHQ. There was no reason why we shouldn't be able to reuse our domain objects both server side and client side - introducing DTOs just so we could workaround this serialization issue seemed ill-advised. It would have just added bloat and unnecessary complexity.<br /><br />I can't remember how mature the <a href="http://noon.gilead.free.fr/gilead/index.php?page=gwt">Gilead</a> project was at the time we started our GWT work, or maybe we just didn't realize it existed. Gilead does require you to have your domain objects and server side impl classes extend certain Java classes (LightEntity for example), so it has a slight downside that it requires you to modify all your domain objects. In any event, we do not use Gilead to do this detaching of hibernate proxies.<br /><br />RHQ's solution was to write our own "Hibernate Detach Utility". This is a single static utility that you use to process your objects just prior to sending them over the wire to your GWT client. Essentially it scrubs your object of all Hibernate proxies, cleaning it such that it can be serialized over the wire successfully.<br /><br />We also used this when we originally developed a web services interface to the RHQ remote API.<br /><br />Here is the<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterprise/server/safe-invoker/src/main/java/org/rhq/enterprise/server/safeinvoker/HibernateDetachUtility.java;hb=refs/heads/master"> HibernateDetachUtility source code</a> in case you are interested in seeing how we do it - maybe you could use this in your own GWT/Hibernate application. I think it is reuseable - not much custom RHQ stuff is going on in here.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4617072861893078150?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-61244312950866047172011-05-23T11:40:00.001-07:002011-05-23T11:50:26.291-07:00It has been a very long road for this one, but we managed to release <a href="http://sourceforge.net/projects/rhq/files/rhq/">RHQ 4</a>.<br /><br />We managed to standardize on GWT as our user interface framework. Here's an example of what the new GWT based UI looks like:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/-tpm2lvmfpLs/TdqrGW-vYvI/AAAAAAAAAC0/xMDwCi8Tpo8/s1600/resource-summary.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 206px;" src="http://1.bp.blogspot.com/-tpm2lvmfpLs/TdqrGW-vYvI/AAAAAAAAAC0/xMDwCi8Tpo8/s320/resource-summary.png" alt="" id="BLOGGER_PHOTO_ID_5609984411579671282" border="0" /></a><br />There are still a few JSP pages around, but for the most part, the RHQ GUI is now a GWT application with SmartGWT components. One of the fun parts of this job is to learn new and exciting technologies (though I guess "new" is relative - but I still consider GWT a "new" GUI technology, compared to things like Struts and JSP).<br /><br />Take a look, give it a test drive and see what you think.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-6124431295086604717?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-83486545686870437172011-02-05T11:46:00.000-08:002011-02-05T11:58:09.770-08:00RHQ has the ability to invoke operations on any resource when it triggers alerts. When you combine this with the ability of the <a href="http://management-platform.blogspot.com/2009/04/managing-resources-that-have-cli.html">Script plugin</a> to run remote scripts via its "Execute" operation, you have a very powerful mechanism to integrate your own processes and rules to help correct or workaround abnormal conditions that occur in your managed environment.<br /><br />Because I've heard several people ask if RHQ has this ability, I put together a <a href="http://www.rhq-project.org/display/JOPR2/Demo-AlertsRemoteScriptExecution">flash demo</a> that shows how you do it. The demo shows how you can execute any script on a remote machine when an alert is triggered by RHQ. For example, if you set up an alert that detects when your app server is using an abnormally large number of threads (a possible indication of heavy load), you can have RHQ execute a custom script on your app server machine to help alleviate problems that might occur due to that condition (such a script could be one that reconfigures a load balancer to help redirect load away from your app server).<br /><br />The use-cases for this feature are virtually endless. Any set of alert conditions on any managed resource can trigger the execution of any script you have. And as the <a href="http://www.rhq-project.org/display/JOPR2/Demo-AlertsRemoteScriptExecution">demo </a>illustrates, it is really easy to set this up within RHQ.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-8348654568687043717?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-40178042320428591712011-01-26T08:51:00.001-08:002011-01-27T07:12:10.621-08:00A certain amount of enhancements and UI clean up were added to the RHQ Bundle/Provisioning feature. So, I figure now would be a good time to re-introduce it in a new blog entry. I also put together <a href="http://rhq-project.org/display/JOPR2/Demo-Bundles">a flash demo</a> if you would like to see the UI in action.<br /><br />Let me recap what this RHQ Bundle/Provisioning feature is all about. RHQ allows you to bundle up a set of files and push them out to remote machines. You can install and upgrade these sets of files as well as revert back to a previous version of the files or purge the bundle files completely. You sometimes see this mentioned as the "Provisioning" feature, and other times you will see it referred to as the "Bundle" subsystem. (I prefer the term "Bundle" since that is the term that the RHQ user interface uses).<br /><br />There are a few concepts you must know in order to understand how RHQ does its thing. This is covered on the <a href="http://rhq-project.org/display/JOPR2/Provisioning#Provisioning-Introduction">wiki</a>, but I'll try to explain it briefly here, too.<br /><br />First is the concept of "bundle". A "bundle" is a logical concept and basically refers to an application ("Pet Store Application" or "My Wiki Server"). A bundle has one or more "versions". A "bundle version" refers to an actual set of files that you want to push out to a set of remote machines. Think of it as your application distribution. Each bundle version has its own "recipe" which tells RHQ what files exist in the bundle, and how to configure and provision those files to the remote machines. Developers or application packagers are responsible for writing the recipe and 'bundling up' the application's files (hence the name 'bundle') with the recipe into an RHQ "bundle version".<br /><br />What you do with a bundle version brings us to the next set of concepts. A bundle "destination" is associated with a specific bundle and is simply a place (or places) where you want to deploy your bundle. A destination specifies two things - a group (which contains one or more remote machines) and a destination directory (which specifies where on the remote machines' filesystems the bundle files should go). Once you have a "bundle destination" in place, you can begin to deploy one or more of its bundle's versions to that destination. A bundle "deployment" represents one deployment of a bundle version to a destination.<br /><br />It may make more sense if I give an example. Suppose I have a web application (call it "My Application") that runs inside a JBoss application server. This is my "bundle". I actually have two versions of my application, 1.0 and 2.0. These are my "bundle versions".<br /><br />Now suppose that I have a QA environment that consists of two machines - a Windows machine and a Linux machine. I want to test my application on my two QA machines. So I need to install my application on both of them. I want to install my application on each machine's "/home/mazz/opt/myapp" directory. This group of two QA machines, along with the destination directory, is a "bundle destination" for my application bundle (call it the "QA destination"). I also have a group of three Linux machines that make up my production environment. After my application passes all tests, I want to deploy my application to that production environment in the "/opt" directory. That group of three production machines, along with the "/opt" directory specification, is another bundle destination associated with my application bundle - call it the "production destination".<br /><br />Once I tell RHQ to deploy the "1.0" bundle version of my application to my "QA destination", I will have a "bundle deployment". This bundle deployment will be considered the "live" deployment because its the last one I pushed out. I can then test that version while it is on my QA machines. Suppose I find that I want to upgrade my QA environment with the newer "2.0" version of my application. I simply deploy that bundle version to the "QA destination" and now I have a second "bundle deployment". This second deployment is now considered live. If I find that I do not like this newer "2.0" version of my application, I can ask RHQ to revert back to the last live deployment (which was my "1.0" bundle version) - this revert becomes yet another "bundle deployment" (the third) but it reverts back to the "1.0" bundle version content. Once I pass all of my QA tests, I can then deploy whatever bundle version I deem appropriate to my "production destination".<br /><br />Most of what I describe above is <a href="http://rhq-project.org/display/JOPR2/Demo-Bundles">actually demonstrated in my flash demo</a>. The only thing I do not show in the demo is the use of a second "production destination", but it is the same effort to deploy to a second destination as it is to deploy to the first destination.<br /><br />One new feature that has been introduced to RHQ is the ability to "purge" a destination of all bundle content. If, for example, you want to remove all bundle files completely from the QA destination, you can ask RHQ to purge that destination. What RHQ will do is remove all bundle content from the remote machines that were associated with that destination.<br /><br />Another new feature that has been added is the ability for RHQ to deploy a bundle into an already existing deployment directory that may have other non-managed content that should be left alone. Such would be the case if you want to deploy an EAR or WAR to a JBossAS deploy/ directory (which obviously has other files inside of it). This deserves some additional explanation.<br /><br />Typically, you will want to deploy a set of application files into its own directory on some file system. For example, if you have a JBoss application server, you want to install it in something like "/opt/my-jboss". All of your application server files are in that directory, but no other files are in there. If you want to remove your JBoss application server installation, it is as simple as "rm -rf /opt/my-jboss".<br /><br />However, what if you deployed a bundle version in that directory already, but you then upgrade that bundle deployment with a new bundle version? In this case, you will already have files in /opt/my-jboss (the original bundle version content). RHQ will actually overwrite, backup or ignore conflicting files that it finds following strict <a href="http://rhq-project.org/display/JOPR2/Ant+Bundles#AntBundles-UpgradeRules">upgrade rules</a>. If, for example, RHQ finds files in that /opt/my-jboss directory that don't belong to the new bundle version, they will be removed. RHQ calls this "managing the root directory".<br /><br />This is usually what you want if you are deploying a standalone software product. If there are any unknown files in the deployment directory, RHQ has to remove them to make sure the bundle deployment directory is exactly in the state the new bundle version recipe wants it to be. However, this is not what you want if you desire to deploy an EAR or WAR to an already existing JBossAS's deploy/ directory. That's because we already know there will be unrelated files in this deploy/ directory that must remain intact and in place. RHQ must leave any files it finds in that destination directory alone - even though they aren't part of our bundle deployment. In other words, we do not want RHQ to manage this root deployment directory.<br /><br />RHQ now supports this by allowing the Ant recipe author to specify the <span style="font-family:courier new;">manageRootDir="false"</span> attribute in the <span style="font-family: courier new;">rhq:deployment-unit</span> task. This new feature is documented in <a href="https://bugzilla.redhat.com/show_bug.cgi?id=659142">Bugzilla 659142</a> and this new attribute is <a href="http://rhq-project.org/display/JOPR2/Ant+Bundles#AntBundles-rhq%3Adeploymentunit">documented on the RHQ wiki</a>.<br /><br />You can read more about the <a href="http://rhq-project.org/display/JOPR2/Provisioning">Provisioning/Bundle feature on the RHQ wiki</a>.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4017804232042859171?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com1tag:blogger.com,1999:blog-6944050187568241059.post-39031079241505802732010-10-26T18:52:00.000-07:002010-10-26T19:22:47.803-07:00I recently had a need to run some quick and dirty performance testing over a few method calls in one of the RHQ plugins that was deployed in my agent. I didn't have a profiler installed and ready to go so I used <a href="http://www.jboss.org/byteman">Byteman</a>. I quickly wrote up a couple of Byteman rules (thanks to <a href="http://jayshaughnessy.blogspot.com/">Jay S.</a> for correcting some problems I had with those rules), ran the RHQ Agent with Byteman installed and got some really good data.<br /><br />It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.<br /><br />Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.<br /><br />I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.<br /><br />I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).<br /><br /><a href="http://rhq-project.org/display/RHQ/Running+The+Agent+With+Byteman">Read the wiki page</a> on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3903107924150580273?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-38348697540366528492010-08-05T23:31:00.001-07:002010-08-05T23:45:51.950-07:00One new feature coming soon to RHQ is the ability to remotely install an RHQ Agent on to any machine in your network (with the caveat that the remote machine must be accessible via SSH).<br /><br />What this now means is you do not have to manually log onto a remote machine and do the tasks of downloading the agent update binary distribution, installing it and running it. RHQ can now do this for you all remotely - all you need to provide to the RHQ GUI is the machine name, your SSH credentials and the location where you want to install the agent.<br /><br />I think this could still use some enhancements - there is no way to customize each agent's configuration (i.e. providing custom answers to the startup setup questions). But that's for another day. If you want to customize the new agent, you can do so by just importing the RHQ Agent resource into inventory and going to its Configuration tab and adjusting its configuration (don't forget to restart the agent so it can pick up the changed configuration- you can restart the agent using this new RHQ GUI page! I'll talk about that next).<br /><br />This mechanism will also allow you to stop and start an existing agent that is already installed. This is very useful if you have not started the agent yet but wish to bring it back online, or for those cases where you have not commited the RHQ Agent resource in inventory yet but still need a way to shutdown or restart an agent.<br /><br />I put together a <a href="http://mazz.fedorapeople.org/demos/remote-agent-install/remote-agent-install.htm">flash demo</a> that shows this stuff in action.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3834869754036652849?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-11084147189260894382010-07-08T05:49:00.000-07:002010-07-08T05:55:51.941-07:00The <a href="http://rhq-project.org/display/RHQ/Home">RHQ</a> team is proud to announce the immediate availability of <strong>RHQ 3.0.0</strong>.<br /><br />This release features several months of hard work by the development team and external contributors. Many of the changes have been already made available in the past through seven community releases.<br /><br />You can browse the full <a href="http://rhq-project.org/display/RHQ/Release+Notes+3.0.0">release notes on the RHQ wiki</a>.<br /><br />The release can be downloaded via the <a href="http://rhq-project.org/display/RHQ/Home">RHQ</a> web site or directly from <a href="http://sourceforge.net/projects/rhq/files/">SourceForge</a>.<br /><br />Download it and try it out. The <a href="http://rhq-project.org/display/JOPR2/Home">documentation wiki</a> has full <a href="http://rhq-project.org/display/JOPR2/Installation">install instructions</a>.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-1108414718926089438?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-88664528463274301762010-06-02T06:41:00.000-07:002010-06-02T06:48:31.048-07:00Work has been steadily progressing in RHQ to introduce a new "provisioning" feature. Our beta will soon be released, and as part of our getting ready of that release, I put together a <a href="http://mazz.fedorapeople.org/demos/provisioning_beta/prov-beta.htm">quick flash demo</a> that illustrates our current state of this provisioning feature.<br /><br />There is still much more work to get done - the new GWT GUI is continually improving and we need more provisioning support for things like starting, stopping and installing services being provisioning, orchestration between machines in a cluster (to support provisioning software within a cluster) and other things. But what we have is a good base framework to build upon - watch the demo if you want to see what we have now.<br /><br />For more information, we have documentation on our wiki:<br /><br /><a href="http://rhq-project.org/display/JOPR2/Provisioning">http://rhq-project.org/display/JOPR2/Provisioning</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-8866452846327430176?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com3tag:blogger.com,1999:blog-6944050187568241059.post-15432616511573181552010-03-25T06:00:00.000-07:002010-05-13T04:58:36.468-07:00Joseph Marques has a <a href="http://josephmarques.wordpress.com/2010/03/24/performance-monitoring-using-hibernate/">very good blog</a> (good in both substance and technical details) on how you can analyze the performance of your Hibernate application through the use of a simple set of utility classes. He even goes into how you can do this inside your EJB3 app. We do this when analyzing performance in RHQ (in addition to <a href="http://management-platform.blogspot.com/2008/11/monitoring-hibernate-with-jopr.html">using RHQ itself to do some analysis</a>). Joe's code examples show that it's a solution that is easily transferable to any Java application that uses Hibernate as its ORM layer.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-1543261651157318155?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-44482742796180770092010-02-01T08:31:00.000-08:002010-02-01T08:49:01.745-08:00<a href="http://www.jboss.org/stormgrind/projects/cirras.html">CirrAS</a> is an effort to automatically deploy clustered JBoss AS servers in the Cloud. With so much focus on cloud computing these days, keep an eye on this CirrAS project.<br /><br />The CirrAS guys are currently using RHQ to monitor every JBossAS instance in the cluster and recently utilized the RHQ CLI to perform some automation steps to get some RHQ things setup. For a description of what they did, see <a href="http://cloudpress.org/2010/01/29/rhq-cli-configuring-and-importing-resources/">http://cloudpress.org/2010/01/29/rhq-cli-configuring-and-importing-resources/</a>. Eventually, the CirrAS team wants to use RHQ to manage every node in the JBossAS cluster (deploy app, restart, etc).<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4448274279618077009?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-71468729196327858112010-01-28T21:02:00.000-08:002010-01-28T22:00:54.041-08:00I'm going to take a departure from my usual topics and talk briefly about a very basic concept - SQL JOINs. I'm doing this mainly for my benefit - I wanted to write this little nugget down so I don't forget it, and what better place to do so than my blog? :)<br /><br />What's the difference between INNER, LEFT and RIGHT JOINs?<br /><br />It seems every so often, I forget the answer to that question - I usually get confused ("what happens if there is a null - do I get a row in my results or not?"). Now, I'm sure it is easy for most people to remember how these JOINs work, but I always seem to forget. However, recently I got a good idiot lesson from <a href="http://josephmarques.wordpress.com/">Joseph Marques</a> on this; he gave me a very short and easy way to remember the rules of these three types of JOINs. It was such an elegant way to explain it that I promised myself I would write it down so I don't forget it, so here it goes (again, I'm sure everyone knows this, and I'm sure I was even told this myself at some point in the past, but of course, I don't remember):<br /><br /><blockquote>JOINs can be thought of in the context of a Venn diagram. You have two circles A and B. These circles partially overlap. (circles A and B represent your table data - when you join the two circles/tables together, the overlap represents relationships between the data.)<br /><ul><li>The overlap is the INNER join - it contains only the elements in A that also have associated B elements.</li><br /><li>The A circle is the LEFT join - it contains all A elements regardless of whether they have associated B elements.</li><br /><li>The B circle is the RIGHT join - it contains all B elements regardless of whether they have associated A elements.</li></ul></blockquote>Note: A LEFT or RIGHT JOIN are OUTER JOINs - sometimes you see these specified as "LEFT OUTER JOIN" or "RIGHT OUTER JOIN". Looking at the Venn diagram you can see why - they include the data outside of the inner overlap area (but also including the inner data). If you just see the word "JOIN" without a specifier, it typically is referring to an "INNER JOIN".<br /><br />Below is an example complete with this Venn diagram visualization:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_I05vMaF_3Ec/S2Jy_7WEAoI/AAAAAAAAACc/D__HH5TopKI/s1600-h/join-venn.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 266px; height: 320px;" src="http://4.bp.blogspot.com/_I05vMaF_3Ec/S2Jy_7WEAoI/AAAAAAAAACc/D__HH5TopKI/s320/join-venn.png" alt="" id="BLOGGER_PHOTO_ID_5432030543149793922" border="0" /></a>I have a Person table containing three people: John, Mary and Bob. I have a Phone table containing three phones that have extensions of 987, 555 and 123. John and Bob have been assigned phones (I can call John on extension 987, Bob on extension 123). Mary is not assigned a phone. The phone with the extension 555 is not assigned to any person.<br /><br />The INNER join:<br /><pre>select name,dial_number<br />from person<br />inner join phone on person.phone_id=phone.phone_id</pre><br />returns results as found in the overlapping area (John:987 and Bob:123).<br /><br />The LEFT join:<br /><pre>select name,dial_number<br />from person<br />left join phone on person.phone_id=phone.phone_id</pre><br />returns results as found in the entire left circle (John:987, Mary:null, Bob:123). Because the left table is the Person table, I have one row for each person, regardless of whether they have a phone assigned or not.<br /><br />The RIGHT join:<br /><pre>select name,dial_number<br />from person<br />right join phone on person.phone_id=phone.phone_id</pre><br />returns results as found in the entire right circle (John:987, null:555, Bob:123). Because the right table is the Phone table, I have one row for each phone, regardless of whether a person is assigned a phone or not.<br /><br />I don't know why it is so hard for me to remember this - it's so obvious when you see it written down like that. But, nevertheless, I find it helpful to refresh my memory with that Venn diagram explanation whenever I experience a brain freeze while developing queries.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-7146872919632785811?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2tag:blogger.com,1999:blog-6944050187568241059.post-13756296602048595482010-01-27T08:18:00.000-08:002010-01-27T08:40:31.368-08:00A few years ago, <a href="http://en.wikipedia.org/wiki/JAD_%28JAva_Decompiler%29">JAD</a> was the Java Decompiler everyone used. However, it's essentially a dead project now - if you still have it, you'll know it can only decompile Java 1.4 class files.<br /><br />However, the good news is that a new Java decompiler has come along to take its place - <a href="http://java.decompiler.free.fr/">JD</a>. I used this on Windows before, but I'm now on Fedora 11 and I never installed this new Java Decompiler there.<br /><br />Thanks to <a href="http://freshjava.blogspot.com/2010/01/jd-gui-on-64-bit-fedora-12.html">Ian's blog</a>, it took me less than 5 minutes to install the JD Java decompiler and its GUI front end (JD-GUI) on my F11 64-bit box. I had to issue this yum command, as per Ian's instructions, to get some libraries installed needed by JD-GUI:<br /><br /><blockquote><pre>yum install libcanberra-gtk2.i586 PackageKit-gtk-module.i586 gtk2-engines.i586</pre></blockquote><br /><br />I then just un-tar'ed the JD-GUI download and it ran fine.<br /><br />I also took the time to <a href="http://java.decompiler.free.fr/?q=jdeclipse#install">install the JD-Eclipse plugin</a> for my Eclipse 3.5. That worked flawlessly on my first attempt.<br /><br />So now I have a nice Java decompiler on my F11 box, both as a standalone GUI and as an Eclipse plugin.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-1375629660204859548?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-64359992859589750492010-01-03T23:44:00.000-08:002010-06-17T05:43:33.668-07:00A couple of months ago, I came across <a href="http://jboss.org/byteman">Byteman</a> and liked it so much, I wanted to see how easy it would be to manage it, so I wrote a quick RHQ agent plugin prototype for it and <a href="http://management-platform.blogspot.com/2009/10/byteman-plugin-for-rhq.html">blogged about it</a>.<br /><br />Since I had some time over Christmas, I decided to revisit that prototype and build it out further. The finished product can probably be considered a good 1.0 version of a fully featured RHQ Byteman plugin.<br /><br />With this RHQ-Byteman integration, you can now do remote byte-code injection into any Java virtual machine that has a Byteman agent running inside of it. Simply write your Byteman rules, store them in one or more scripts and file-upload those scripts from your browser to the RHQ Server. The RHQ Server will then stream down those script files to the remote Byteman agent, which then injects the code directly in its Java virtual machine. It is extremely easy to run a VM with Byteman inside of it - all you need is to pass a -javaagent VM argument to Java when you start it; something like:<br /><br /><pre>-javaagent:/opt/byteman/lib/byteman.jar=listener:true</pre><br /><br />The uses for this kind of thing are boundless - from tracing how long it takes for a Java method to execute to testing an application's fault tolerance by forcing test exceptions to be thrown. In the future, I plan on attempting to write an extension to the Byteman plugin to allow me to trace JDBC calls and create reports based on the data I get back. In fact, work on that has <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/plugins/jdbctrace;hb=refs/heads/byteman-lsof-plugins">already begun</a>; the bulk of the RHQ plugin code is already written, I just have to figure out how to write the proper Byteman rules to get it to do what I want. Once I complete this "JDBC Trace" RHQ agent plugin, I'll blog/demo it.<br /><br />If you are interested in this new RHQ integration with Byteman, watch the <a href="http://rhq-project.org/display/JOPR2/Demo-Byteman">flash demo</a>.<br /><br />Note that in order to finish this Byteman plugin, I had to add some additional capabilities within Byteman itself. The bulk of the changes manifested themselves as new API methods to the Byteman client - the "Submit" class. These new features allow Byteman to be easily managed, not just by RHQ, but by any management tool that wants to integrate with Byteman. I committed this new code to Byteman's SVN repository which should make these new features available in its next release. Because the new Byteman is not released yet, nor available in the JBoss maven repository, I placed the RHQ Byteman plugin code in its <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/plugins/byteman;hb=refs/heads/byteman-lsof-plugins">own git branch</a>. Once the next Byteman version is released, I'll merge my code into RHQ's <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=shortlog;h=refs/heads/master">master branch</a>.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-6435999285958975049?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-25797587452252041322009-12-11T07:54:00.000-08:002010-06-17T05:34:56.586-07:00The RHQ project is pleased to announce the first developer release of the RHQ 1.4.0 management platform. This developer release, <em>1.4.0-B01</em>, provides an early look at the new features which are being planned for the 1.4.0 general release.<br /><br />Note that we've pulled in the Jopr plugins into this RHQ release. So, consider this new RHQ release a merging of both the <a href="http://www.rhq-project.org/">RHQ project</a> and the <a href="http://www.jboss.org/jopr/">Jopr project</a>.<br /><br /><strong>Note:</strong> do not upgrade any existing Jopr or RHQ installs with this build! Specifically, the Jopr plugins in this new RHQ build are versioned differently and will not upgrade your existing Jopr plugins. We'll fix this in a later release.<br /><br />Now, here are the particulars if you are interested in learning more:<br /><br /><ul><br /><li>To find out the new features that are being introduced, read the <a href="http://rhq-project.org/display/RHQ/Release+Notes+1.4.0.B01">Release Notes</a></li><br /><li>You can <a href="https://sourceforge.net/projects/rhq/files/rhq/rhq-1.4.0.beta1/rhq-server-1.4.0.B01.zip/download">download the software</a></li><br /><li>You can get the source code from the new <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=summary">git repository on fedorahosted</a></li><br /><li>Feel free to ask any questions you have in our chat room at <a href="irc://irc.freenode.net/#rhq">#rhq on irc.freenode.net</a></li><br /><li>If you have questions on how to use RHQ, you can ask them on our <a href="https://fedorahosted.org/mailman/listinfo/rhq-users">rhq-users mailing list</a></li><br /><li>If you have questions on how to develop new enhancements/fixes for RHQ or how to write RHQ plugins, you may ask them on our <a href="https://fedorahosted.org/mailman/listinfo/rhq-devel">rhq-devel mailing list</a></li><br /><li>RHQ bugs/enhancements are now found in <a href="https://bugzilla.redhat.com/browse.cgi?product=RHQ%20Project">BugZilla</a></li><br /><li>Individual Maven artifacts have been published to the <a href="http://repository.jboss.com/maven2/">JBoss maven repository</a> mostly under <code><a href="http://repository.jboss.com/maven2/org/rhq/">org/rhq/</a></code> (but some JBoss plugin related things are under <code><a href="http://repository.jboss.com/maven2/org/jboss/on/">org/jboss/on/</a></code>)</li><br /><li>As usual, the user documentation can be found on the <a href="http://rhq-project.org/display/JOPR2/Home">RHQ user documentation wiki space</a></li><br /><li>Design documentation can be found on the <a href="http://rhq-project.org/display/RHQ/Home">RHQ design doc wiki space</a></li><br /></ul><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-2579758745225204132?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-37533100655782808522009-12-05T10:52:00.000-08:002010-06-17T05:41:24.303-07:00RHQ is introducing a major new feature - <span style="font-style: italic;">server plugins</span>. I know what you are saying... "ho-hum, another 'plugin mechanism'". It is true that this is analogous to the RHQ Agent plugin functionality - just as you can write your own agent plugins to extend the types of resources an agent can manage, you now have similar capabilities in the RHQ Server.<br /><br />But when you examine the capabilities of this new server plugin subsystem, you will notice that you now have access to the full breadth and depth of functionality that the core RHQ Server code has, <span style="font-style: italic;">without having to modify core code</span>. Using server plugins, you can plugin your own management code to the RHQ Server, allowing you to add to the existing core management feature set. With a minimum of a single Java POJO class bundled with an XML descriptor, you can now extend RHQ server functionality in areas you couldn't with the agent plugins. The ability to innovate new features and capabilities will be much easier.<br /><br />I put together a <a href="http://rhq-project.org/display/JOPR2/Demo-ServerPlugin-SimpleReports">demo</a> for an illustration of the potential that server plugins open up. If you are interested, the demo's plugin code can be found <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=etc/samples/simplereport-serverplugin;hb=master">here</a>. This sample plugin has a <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=etc/samples/simplereport-serverplugin/src/main/java/org/custom/SimpleReportsPluginComponent.java;hb=master">single Java POJO</a> and an <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=etc/samples/simplereport-serverplugin/src/main/resources/META-INF/rhq-serverplugin.xml;hb=master">XML plugin descriptor</a>.<br /><br />The core of the server plugin container infrastructure is checked in. More work will be done to enhance it further, but it works today. What the RHQ team is currently working on is adding support for different types of server plugins.<br /><br />For example, the demo mentioned above uses a "generic plugin" - that is just one type of server plugin. This is a "catch-all" type of plugin - all other plugin types will essentially build on top of the same capabilities of a "generic plugin". This generic functionality allows for the server plugin developer to write Java code that gets embedded in the server and runs. Each plugin can define a plugin component (much like an agent resource component in an agent plugin). The plugin component is notified when lifecycle events occur (such as when the plugin is started and stopped). The generic plugin can also utilize the server's scheduler to run jobs on a periodic basis. For example, if I want to generate management reports, I can write a server plugin that has a job that runs every day, generating reports when it runs (in fact, this is what the demo above illustrates).<br /><br />Other types of plugins that are actively being developed are:<br /><ul><br /><li>Content Plugins - these will allow you to define remote content repositories so you can pull in software from those remote locations to be pushed out to remote machines.</li><br /><li>Alert Plugins - these will allow you to plug in remote alert destinations. Heiko has some great ideas on this and has already demoed a few of them. The one I like in particular is the <a href="http://pilhuhn.blogspot.com/2009/11/rhq-and-future-of-alerts.html">alert plugin that integrates with Mobicents</a> so when an RHQ alert is triggered, a voice message is sent to an administrator's phone. Very cool :) </li><br /><li>Perspective Plugin - this is an extremely powerful type of plugin that will enable a developer to add additional graphical user interfaces to the core RHQ Server. There is a entire <a href="http://rhq-project.org/display/RHQ/Design-Perspectives">perspectives design</a>, which will build within the server plugin infrastructure.<br /></li><br /></ul><br /><br />A lot more will be communicated about these server plugins as they are built out. But you can still do server plugin development today. <a href="http://rhq-project.org/display/RHQ/Server+Plugin+Development">Documentation</a> is available, and we'll be building out more documentation as we get further along.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3753310065578280852?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-44666491870432253662009-12-03T06:31:00.001-08:002010-06-17T05:42:29.838-07:00Recently some changes have been made to the infrastructure used by the RHQ and Jopr projects.<br /><br />We have consolidated the work into a <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=summary">single fedorahosted.org project</a>.<br /><br />We have also changed from the use of JIRA to Bugzilla. All JIRA issues have been ported over to the <a href="https://bugzilla.redhat.com/browse.cgi?product=RHQ%20Project">RHQ Bugzilla</a> project.<br /><br />You may see changes to my previous blog entries - I'll go through them all to edit the links so they point to the new places, as opposed to the old, outdated SVN and JIRA locations.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4466649187043225366?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2tag:blogger.com,1999:blog-6944050187568241059.post-52632082659357705952009-12-01T14:39:00.000-08:002009-12-01T14:57:32.204-08:00There was previously some requests for RHQ to support "configuration remediation". In short, it would be nice to "freeze" a resource configuration such that if that configuration ever changes for whatever reason, we want that configuration to immediately revert back to its "frozen" configuration state.<br /><br />I spent a couple of hours coding up a prototype to show this feature in RHQ. See the <a href="http://www.rhq-project.org/display/RHQ/Configuration+Freeze">flash demo here</a>.<br /><br />There are a few things to say about this.<br /><br />First, specifically about configuration remediation - this demonstrates how easily it was for RHQ to support such a feature.<br /><br />Second, and more generally, this effort illustrates the benefits of the RHQ architecture and the concept of "management at the edges". The code was added to the core configuration subsystem - it did not involve any one specific type of managed resource. Because of that, the couple hours that I spent adding this feature provided a wide array of functionality - I now have configuration remediation support for <span style="font-weight: bold;">any</span> managed resource that supports configuration. The storage of the configuration on the managed resources is handled at the edges by each manage resource's plugin. But the remediation logic is in the core server, independent of any managed resource that is out in the environment. So, not only was I able to quickly add a new configuration feature to RHQ, but that feature was immediately available across all types of resources without any additional effort.<br /><br />What does this mean? It means, for example, that I can now freeze configuration of a JBossAS data source just as easily as I can freeze the configuration of my OpenSSH daemon (or any other managed resource that I can configure). On top of that, my RHQ users won't have to learn many different user interfaces or different mechanisms depending on what resource they want to configure - learn one user interface and you will know how to do this for JBossAS data sources, or OpenSSH daemons, or anything else.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5263208265935770595?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-22400092372127632602009-10-28T07:04:00.001-07:002009-10-28T09:38:11.722-07:00I read <a href="http://jbossts.blogspot.com/2009/10/monitoring-your-jvm-using-byteman-111.html">Andrew Dinn's post on Byteman 1.1.1</a> and was very intrigued. His post and the <a href="http://www.jboss.org/byteman/documentation.html">Byteman documentation</a> was very quick and easy to read and I found out how easy it is to use Byteman. Just one (albeit long) command line option to your Java VM and you can easily inject code into existing Java classes using Byteman's byte-code instrumentation. I plan to use Byteman to do things like "fault-injection" testing - inject code that purposefully causes errors in my application to see how my application handles the errors (this is especially useful for forcing XA/2PC transaction errors to see how the application and transaction manager handles it).<br /><br />I ended up doing some quick but useful enhancements to Byteman (thanks, Andrew, for the commit access :-) that allows any Java application to perform remote byte-code injection in a JVM where Byteman is running. There is now a Java API to the Byteman "submit" client (the new-and-improved Submit class).<br /><br />I then took a few hours to write up a quick Byteman RHQ plugin prototype. I recorded a <a href="http://mazz.fedorapeople.org/demos/byteman-plugin/byteman-plugin.htm">Flash demo</a> to show how it works. Still a lot more that can be done with some additional enhancements to this plugin, but hopefully this gets across how easy it is to write RHQ plugins to manage anything. This plugin will allow you to view deployed rule definitions files and their rules and allow you to remove rules. With some enhancements, it could allow you to upload and deploy additional rules as well.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-2240009237212763260?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-78244505500636665652009-08-04T21:21:00.000-07:002009-12-03T06:59:09.634-08:00Work is ongoing at a feverish pace to complete the new Jopr CLI. It is filling out quite nicely. What is this CLI, you ask? It's the long-awaited and often-requested command line interface utility for the Jopr Server.<br /><br />Up until now, the only user interface into the Jopr system is via the web-based GUI. The new CLI now provides a scripting interface that allows users to tap into the Jopr system from their shell.<br /><br />The CLI is actually a two-in-one utility - it can act as what people typically think of as a CLI but it is also an interactive shell (this is analogous to say, perl or python). The underlying scripting language is JavaScript - its the default Java scripting implementation found in the Java6 runtime.<br /><br />Under the covers, the CLI utilizes the remote client API - this remote client API can be used by developers to write their own Jopr clients (anyone out there interested in writing a CLI GUI? :-) You can use the Java client or you can rely on the Web Services API that is exposed by the Jopr Server, allowing a client to consume the Jopr WSDLs and perform remote commands over Web Services.<br /><br />I'll briefly show you just a few things you can do with the CLI. This should give you a taste for how you can interact with the CLI and some of its basic features.<br /><br />First, here's the "long way" to query a list of resources and filter the results based on some custom criteria. Here I will ask the Jopr Server for all resources that have "mazzthink" in their name:<br /><br /><pre><blockquote><br />[mazz@mazzfedora bin]$ ./rhq-cli.sh -u rhqadmin -p rhqadmin -s mazzthink<br />RHQ - RHQ Enterprise Remote CLI 1.3.0-SNAPSHOT<br />Login successful<br />rhqadmin@mazzthink:7080$ criteria = new ResourceCriteria();<br />ResourceCriteria:<br /><br />rhqadmin@mazzthink:7080$ criteria.addFilterName('mazzthink');<br /><br />rhqadmin@mazzthink:7080$ ResourceManager.findResource<br /><br />findResourceComposites findResourceLineage findResourcesByCriteria<br />rhqadmin@mazzthink:7080$ ResourceManager.findResourcesByCriteria(criteria);<br />id name version curre<br />ntAvailability resourceType<br />------------------------------------------------------------------------------------------<br />----------------------------------------------------------------------<br />10001 mazzthink Win32 5.1 UP<br /> Windows<br />10002 Trapd (mazzthink) DOWN<br /> SnmpTrapd<br />10004 mazzthink RHQ Agent 1.3.0-SNAPSHOT UP<br /> RHQ Agent<br />10005 mazzthink Jopr Server, JBossAS 4.2.3.GA default (0.0.0.0:2099) 4.2.3.GA UP<br /> JBossAS Server<br />10007 mazzthink Apache 2.2.9 (C:\mazz\apache-httpd\) 2.2.9 DOWN<br /> Apache HTTP Server<br />10008 mazzthink Apache 2.2.9 (C:\apache\) 2.2.9 DOWN<br /> Apache HTTP Server<br />10035 mazzthink File System (local) C:\ UP<br /> File System<br />10059 mazzthink Embedded JBossWeb Server 2.0.1.GA (0.0.0.0) 2.0.1.GA UP<br /> Embedded Tomcat Server<br />8 rows<br /></blockquote></pre><br /><br />A couple things to note from above. I created a ResourceCriteria that allows me to set filters, sorting and paging controls. Here I just filter on the name "mazzthink".<br /><br />Second, I used the CLI interactive shell's auto-complete feature to help me figure out what methods are available on the ResourceManager. I simply typed "ResourceManager.findResource" then hit the Tab key and you see the suggestions printed out by the auto-completer. I then chose to use the findResourcesByCriteria method.<br /><br />Since this is one of the most common things to want to do in the CLI (that is, query for resources), there is a shortcut method you can use for this - findResources. Below you will see me doing the same query as above, only using the "short way":<br /><br /><pre><blockquote><br />rhqadmin@mazzthink:7080$ findResources('mazzthink')<br />id name version curre<br />ntAvailability resourceType<br />------------------------------------------------------------------------------------------<br />----------------------------------------------------------------------<br />10001 mazzthink Win32 5.1 UP<br /> Windows<br />10002 Trapd (mazzthink) DOWN<br /> SnmpTrapd<br />10004 mazzthink RHQ Agent 1.3.0-SNAPSHOT UP<br /> RHQ Agent<br />10005 mazzthink Jopr Server, JBossAS 4.2.3.GA default (0.0.0.0:2099) 4.2.3.GA UP<br /> JBossAS Server<br />10007 mazzthink Apache 2.2.9 (C:\mazz\apache-httpd\) 2.2.9 DOWN<br /> Apache HTTP Server<br />10008 mazzthink Apache 2.2.9 (C:\apache\) 2.2.9 DOWN<br /> Apache HTTP Server<br />10035 mazzthink File System (local) C:\ UP<br /> File System<br />10059 mazzthink Embedded JBossWeb Server 2.0.1.GA (0.0.0.0) 2.0.1.GA UP<br /> Embedded Tomcat Server<br />8 rows<br /></blockquote></pre><br /><br />Now that I got my list of resources, I can use the results to help perform additional work. Suppose I want to know how much free memory I have on my "mazzthink" machine. I can see from my query above that the "mazzthink" Windows platform has an id of 10001. I'll pass that to the ProxyFactory.getResource method to retrieve a resource object that I can use to access information about that resource (such as its free memory metric):<br /><br /><pre><blockquote><br />rhqadmin@mazzthink:7080$ myResource = ProxyFactory.getResource(10001);<br />rhqadmin@mazzthink:7080$ myResource.<tab><br /><br />OSName OSVersion<br />architecture children<br />createdDate description<br />freeMemory freeSwapSpace<br />getChild getMeasurement<br />handler hostname<br />id idle<br />manualAutodiscovery measurementMap<br />measurements modifiedDate<br />name operations<br />pluginConfiguration pluginConfigurationDefinition<br />systemLoad toString<br />totalMemory totalSwapSpace<br />usedMemory usedSwapSpace<br />userLoad version<br />viewProcessList waitLoad<br />rhqadmin@mazzthink:7080$ myResource.freeMemory<br />Measurement:<br /> name: Free Memory<br /> displayValue: 872.8MB<br /> description: The total free system memory<br /></tab></blockquote></pre><br /><br />Notice above I illustrate the usage of the auto-completer again. I hit Tab after typing in "myResource." and the CLI showed me all the valid measurements, operations and other methods that are available for that resource. I chose to get the "freeMemory" measurement, and you can see above that this resource has 872.8MB of free memory.<br /><br />In addition to getting measurement data, another thing I can use the CLI for is to execute operations on managed resources. Continuing our example with the mazzthink platform resource, suppose I wanted to get a list of all the processes running on that box. The platform resource has a "viewProcessList" operation available to it (as you can see above from the auto-completer output). If you are unsure what operations are available on a resource, you can use the "operations" method on your resource proxy object to get the list:<br /><br /><pre><blockquote><br />rhqadmin@mazzthink:7080$ myResource.operations<br />name description<br />------------------------------------------------------------------------------------------<br />----------------------------------------------------------------------<br />viewProcessList View running processes on this system<br />manualAutodiscovery Run an immediate discovery to search for resources<br />2 rows<br /></blockquote></pre><br /><br />Using the CLI, we can invoke that viewProcessList operation on the mazzthink Windows platform and have the results shown on the CLI console:<br /><br /><pre><blockquote><br />rhqadmin@mazzthink:7080$ myResource.viewProcessList()<br />Invoking operation viewProcessList<br />Configuration [12186] - null<br />processList [99] {<br /> process [5] {<br /> kernelTime = 25140<br /> name = winlogon.exe<br /> pid = 1836<br /> userTime = 8156<br /> size = 95723520<br /> }<br /> process [5] {<br /> kernelTime = 31<br /> name = httpd.exe<br /> pid = 516<br /> userTime = 31<br /> size = 25780224<br /> }<br /> process [5] {<br /> kernelTime = 2296<br /> name = postgres.exe<br /> pid = 1572<br /> userTime = 937<br /> size = 75284480<br /> }<br /> process [5] {<br /> kernelTime = 159421<br /> name = javaw.exe<br /> pid = 2776<br /> userTime = 380347<br /> size = 1215180800<br /> }<br /> process [5] {<br /> kernelTime = 91190<br /> name = firefox.exe<br /> pid = 6088<br /> userTime = 251285<br /> size = 876466176<br /> }<br /> ...<br />}<br /></blockquote></pre><br /><br />And finally, to illustrate that the CLI is really a CLI and not just an interactive shell, you can pass a script command directly on the command line via the -c option:<br /><br /><pre><blockquote><br />[mazz@mazzfedora bin]$ ./rhq-cli.sh -u rhqadmin -p rhqadmin -s mazzthink -c "Pr<br />oxyFactory.getResource(10001).freeMemory;"<br />RHQ - RHQ Enterprise Remote CLI 1.3.0-SNAPSHOT<br />Login successful<br />Measurement:<br /> name: Free Memory<br /> displayValue: 884.5MB<br /> description: The total free system memory<br /></blockquote></pre><br /><br />The CLI also has a -f option that allows you to specify a script file to be executed - allowing you to write and prepackage scripts for execution later.<br /><br />That's the CLI in a nutshell. It's still a work in progress - the remote API is still being flushed out and being enhanced to support the many use-cases we expect people are going to want to support.<br /><br />The community documentation is also currently being worked on. But there is some documentation now, with more to come at the following pages:<br /><br /><a href="http://www.rhq-project.org/display/JOPR2/RHQ+CLI+Installation">CLI Install Documentation</a><br /><a href="http://www.rhq-project.org/display/JOPR2/Running+the+RHQ+CLI">Running the CLI documentation</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-7824450550063666565?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2tag:blogger.com,1999:blog-6944050187568241059.post-70328584761239500822009-07-10T07:23:00.000-07:002010-06-17T05:32:35.904-07:00A previous blog of mine talked about how you can <a href="http://management-platform.blogspot.com/2009/04/managing-resources-that-have-cli.html">monitor an application or generic operating system process by going through its Script/CLI interface</a> (you can watch a <a href="http://www.rhq-project.org/display/JOPR2/Demo-ScriptPlugin">demo of the Script plugin</a> to see it in action).<br /><br />There is now another way to do this in Jopr, thanks to the new process monitoring feature introduced in the platform plugin.<br /><br />Even if you do not have a CLI interface for the operating system process you want to monitor, you can still ask Jopr to monitor it - you do so by pointing Jopr to a pidfile containing the process's pid, or you can give Jopr a PIQL process query to execute to find the process. (<a href="http://git.fedorahosted.org/git/?hb=master;p=rhq/rhq.git;a=blob;f=modules/core/native-system/src/main/java/org/rhq/core/system/pquery/ProcessInfoQuery.java">PIQL</a> = process information query langage). Note that this is all cross-platform - you can monitor processes on Windows just as easily as you can on any UNIX flavored operating system, so long as there is native support for your platform and it is enabled (which is normally the case, so you typically don't have to worry about this - you'll get this stuff for free).<br /><br />What this means is for any process running anywhere out on your network, you can monitor that process. This process monitoring mechanism can be used to inform you if your managed process is up or down, and it can collect some metrics on the managed process, such as CPU utilization, memory consumption, and file descriptor usage.<br /><br />You can <a href="http://www.rhq-project.org/display/JOPR2/Demo-ProcessMonitoring">watch a demo of this process monitoring feature</a> to get a feel for how it can be used and what it does.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-7032858476123950082?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2tag:blogger.com,1999:blog-6944050187568241059.post-56228597045476159512009-06-27T23:45:00.000-07:002009-12-03T06:55:34.947-08:00Thanks to Greg Hinkle, Jopr has recently introduced an experimental feature that exposes your managed environment as a WebDAV repository.<br /><br />This means that you can use any WebDAV client to browse your managed environment as if it were a simple file system and peek into your resource hierarchy, obtaining information such as a resource's availability, its configuration and measurement trait data (watch this <a href="http://www.rhq-project.org/display/JOPR2/Demo-WebDAV">WebDAV demo</a> to see it in action; Greg's blog post and demo are <a href="http://www.jroller.com/ghinkle/entry/browsing_your_environment_as_a">here</a>).<br /><br />If you aren't familiar with <a href="http://www.webdav.org/">WebDAV</a>, all this means is that your managed environment will look like a simple file system (in the Microsoft Windows vernacular, it is called a "Web Folder"). Directories in this "file system" or "Web Folder" represent managed resources and files found in those directories represent data about those resources.<br /><br />So, for example, if you had a machine called "comp.xyz.com", and on that machine you have installed a JBossAS server, a PostgreSQL server and a network adapter, your WebDAV file system paths to access those resources could look like this (notice how they look like simple file system paths):<br /><br /><table border="1"><br /><tr><td>Machine itself:</td><td>/webdav/resource/comp.xyz.com</td></tr><br /><tr><td>Network adapter:</td><td>/webdav/resource/comp.xyz.com/eth0</td></tr><br /><tr><td>JBossAS server:</td><td>/webdav/resource/comp.xyz.com/Banking%20App</td></tr><br /><tr><td>PostgreSQL DB:</td><td>/webdav/resource/comp.xyz.com/My%20Postgres</td></tr><br /></table><br /><br />This is currently in the "experimental" phase; however, by watching my <a href="http://www.rhq-project.org/display/JOPR2/Demo-WebDAV">WebDAV demo</a>, it should be easy to see how this can prove to be a very powerful feature. Using any WebDAV client, you potentially could obtain alot of information about your managed resources (measurement data, alerts, events, logs, configuration and much more). It's pretty easy to add functionality to this WebDAV interface, so providing access to things like measurement data, alerts, events and the like should not be hard to accomplish. In about 30 minutes, I added the ability to view a resource's measurements traits (you will see that feature in the demo). <i>6/29/2009 note: spent another couple hours coding and was able to incorporate authentication/authorization into the WebDAV tier so it utilizes the normal Jopr authz layer; also added a new WebDAV resource "measurement_data.xml" which is an XML file of the numeric measurement data for a resource (min, max, avg values, etc).</i><br /><br />Now we just need to find the time to flush out this functionality and make it production ready with additional features. If you are interested in working with the Jopr team and willing to get your hands dirty writing code, this might not be a bad place to start. You really don't need to know too much about the internals of the Jopr Server - you just have to interface with the internal Jopr stateless session EJB3 beans to obtain data about resources. The WebDAV API seems pretty easy to use - it's all based on the third party library called <a href="http://milton.ettrema.com">Milton</a>. Don't hesitate to ask the team for help in getting started as a Jopr developer, we are usually around on freenode at #jopr or send an email to "jopr-dev at lists.jboss.org"<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5622859704547615951?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-50319927838926570932009-06-05T09:43:00.001-07:002009-06-05T09:56:05.598-07:00Well, Joe did it again. I don't know how he does these things so fast but here you go:<br /><br /><a href="http://josephmarques.wordpress.com/2009/06/03/jopr-has-embedded-database-support/">http://josephmarques.wordpress.com/2009/06/03/jopr-has-embedded-database-support/</a><br /><br />With this new <a href="http://www.h2database.com/">H2</a> embedded database support, you can now install Jopr much easier now. You no longer have to install your own PostgreSQL or Oracle database separately. You now have the option to use the "embedded mode" which tells Jopr to use the embedded H2 database and the embedded agent.<br /><br />This means to install and try out Jopr involves these simple steps (oh, and these are cross-platform instructions - it works the same on Windows as it does on UNIX or Linux):<br /><br /><ol><li>Unzip the Jopr distribution</li><li>Run the Jopr server startup script (rhq-server.[sh, bat] start)</li><li>Point your browser to http://localhost:7080</li><li>Click the "Embedded Mode" button</li><li>Click the "Install!" button<br /></li></ol>That's it! Jopr will finish the installation for you, creating your H2 database and starting up both the server and embedded agent. The agent will automatically begin discovering and monitoring resources it finds on your box. The browser will provide you with a link to get started in the Jopr UI.<br /><br />Use this quick and easy method to try out Jopr. You can try it out and demo it on anything from a laptop to a mainframe - so long as you have a Java virtual machine available.<br /><br /><span style="font-style: italic;">(To Joe - thanks, job well done! Joe did the heavy lifting, I just enhanced the installer to be able to install Jopr using the embedded database.)</span><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5031992783892657093?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-87695287594656295562009-05-18T11:31:00.000-07:002010-06-17T05:30:58.975-07:00This blog is in response to a question from user "earthling" about my <a href="http://management-platform.blogspot.com/2009/03/correlating-events-with-jopr.html">Correlating Events with Jopr</a> blog entry.<br /><br />"How do i use this feature [Events] in a custom plugin?"<br /><br />It's easiest if you just copy the code from one plugin to your custom plugin. It isn't that much code.<br /><br />First, you need to declare that your plugin's custom resource type supports events. Look at how the <a href="http://git.fedorahosted.org/git/?hb=master;p=rhq/rhq.git;a=blob;f=modules/plugins/jboss-as/src/main/resources/META-INF/rhq-plugin.xml">JBossAS plugin descriptor</a> does this - you can probably copy the &lt;plugin-configuration>...&lt;c:group name="event" displayName="Events"> section verbatim.<br /><br />Then, your custom ResourceComponent subclass needs to add an event poller to the plugin container. You can do this within your start() method, as the <a href="http://git.fedorahosted.org/git/?hb=master;p=rhq/rhq.git;a=blob;f=modules/plugins/jboss-as/src/main/java/org/rhq/plugins/jbossas/JBossASServerComponent.java">JBossASServerComponent</a> does, like this:<br /><br /><pre>public void start(ResourceContext context) throws Exception {<br />this.logFileEventDelegate = new LogFileEventResourceComponentHelper(this.resourceContext);<br />this.logFileEventDelegate.startLogFileEventPollers();<br />...<br />}</pre><br /><br />Note that the "<a href="http://git.fedorahosted.org/git/?hb=master;p=rhq/rhq.git;a=blob;f=modules/core/plugin-api/src/main/java/org/rhq/core/pluginapi/event/log/LogFileEventResourceComponentHelper.java">LogFileEventResourceComponentHelper</a>" class is new that Ian added after 2.2. So, you'll have to build trunk/HEAD to be able to use it. If you have 2.2 or less, just grab that class and use it in your plugin (or at least copy-n-paste its code).<br /><br />In your components stop() method, you need to remove the event poller, too:<br /><br /><pre>public void stop() {<br />this.logFileEventDelegate.stopLogFileEventPollers();<br />...<br />}</pre><br /><br />Once your resource has started, it will start emitting logs that the Jopr Server will show you! You get all the event functionality by adding that little bit of metadata and code. Of course, the example above assumes your events come from log4j log files. But events can come from anywhere. For example, the Windows platform plugin component can emit events from the Windows Events subsystem and Greg Hinkle has updated the Linux plugin component so that it emits syslog messages from his Linux box (he should probably check that in soon :).<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-8769528759465629556?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com1tag:blogger.com,1999:blog-6944050187568241059.post-20627306050463643102009-04-28T19:35:00.000-07:002010-06-17T05:29:17.262-07:00Sometimes, you may find yourself wanting to manage a resource for which there is no current Jopr plugin or, perhaps, the resource doesn't even have a management API or interface.<br /><br />I have developed <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/plugins/script;hb=master">a new "Script plugin"</a> that may help manage these "unmanageable" resources. You can watch a <a href="http://www.rhq-project.org/display/JOPR2/Demo-ScriptPlugin">demo</a> that shows the Script plugin in action.<br /><br />This plugin will assume there is a command line executable or script that can be used to interface with the managed resource. What you do is manually add your "Script" resource to the Jopr inventory - defining where the command line executable or script is, and how its results/output should be used to determine the state of the managed resource. To be clear, the "CLI" executable/script is not really the managed resource - it is only used to interface with the real managed resource. The typical example I give people is: "apachectl" is the script, Apache Web Server is the real managed resource.<br /><br />There is the ability to show RED or GREEN availability for any Script resource. For example, I can configure the Script plugin to assume the resource is UP (aka GREEN) if:<br /><ul><li>the executable merely exists on the file system, or...<br /></li><li>the executable was able to run or...<br /></li><li>the executable ran and returns a certain exit code and/or...<br /></li><li>the executable ran and output a certain text string</li></ul><br />The idea is that I have a command line executable or script that needs to get executed with a certain set of arguments. A successful execution would be determined by a regular expression that needs to match the exit code or output from that CLI.<br /><br />You can also use the same type of mechanism to define how to invoke the executable to obtain a certain set of metrics. You can define arguments and a regular expression and based on what the executable returns (and what the regex captures) would be the value of the metric data.<br /><br />This plugin should be useful because it is very generic - you can use it as-is for your own generic resources that have command line interfaces, a developer could extend the plugin classes, or a developer could just write their own plugin descriptor and have it pick up the Script plugin's ability to execute any executable and get the results. I think we'll be able to extend this to build a Nagios plugin, for example.<br /><br />If you have any thoughts or ideas on what features such a Script plugin should have, feel free to comment. I would be interested to know what kinds of CLIs people run that manage their resources.<br /><br />If you are interested in helping out develop a Nagios plugin, let me know - I'm very desparate in getting help on that. :)<br /><br /><span style="font-style:italic;">[note: this blog post was updated July 2009 to reflect the new name of the plugin. This plugin was previously called the "CLI" plugin, but in order to avoid confusion with Jopr's new remote client/CLI, it was renamed to "Script" plugin.]</span><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-2062730605046364310?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com6tag:blogger.com,1999:blog-6944050187568241059.post-52143832135754606522009-04-26T23:28:00.001-07:002009-12-03T06:42:12.553-08:00We like to say Jopr is a management platform, not just a monitoring tool. This is because Jopr does more than just monitor the health of managed resources. It can be used to configure resources and control resources, too. In addition, Jopr has a content subsystem that allows it to deploy content to any managed resource that supports the content facet.<br /><br />What does this mean? It means that you can set up your Fedora boxes to be able to "yum install" packages directly from Jopr; it means you can install patches to your JBossAS Servers; it means just about anything you want it to mean (in the context of pushing and pulling content) because Jopr is extensible in such a way that you can write your own plugins to do what you need it to do with respect to pulling down content for deployment to your custom resources.<br /><br />I've created a <a href="http://www.rhq-project.org/display/JOPR2/Demo-Content">Jopr content demo</a> that shows how you can aggregate content from multiple remote repositories into Jopr and then have Jopr serve that content to resources it is managing.<br /><br />Here is an architectural diagram that discusses how content flows from remote repositories to the Jopr Server through to the managed resources via the Jopr Agent. This diagram, coupled with the demo, should provide some good insight into the basics of the Jopr content subsystem.<br /><br /><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_I05vMaF_3Ec/SfVW9g5kbQI/AAAAAAAAAB0/spzBkKccUIk/s1600-h/jopr-content.png"><img style="cursor: pointer; width: 320px; height: 234px;" src="http://2.bp.blogspot.com/_I05vMaF_3Ec/SfVW9g5kbQI/AAAAAAAAAB0/spzBkKccUIk/s320/jopr-content.png" alt="" id="BLOGGER_PHOTO_ID_5329261348866256130" border="0" /></a><br /></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5214383213575460652?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com5tag:blogger.com,1999:blog-6944050187568241059.post-73131614057987878482009-04-21T20:02:00.000-07:002009-12-03T06:37:54.382-08:00<a href="http://www.jboss.org/jopr/">Jopr</a> and <a href="http://www.jboss.org/embjopr/">Embedded Jopr</a>. Aside from each having Jopr in the name, how are they related? How are they different? Why do these two projects exist? I'll try to explain that here.<br /><br />Jopr is a management platform. It allows you to manage an entire network of systems and products hosted in your IT environment.<br /><br />Embedded Jopr is a management console that is embedded in a JBoss Application Server to allow you to manage that particular JBossAS instance.<br /><br />Jopr and Embedded Jopr use what are known as "plugins" to do the real management work - each plugin is specific to a particular product or software component that needs to be managed. For example, today there exists plugins to manage JBossAS, Tomcat, Hibernate, and PostgreSQL, among other things. These "plugins" live in what is called the "Jopr Plugin Container". This plugin container is responsible for controlling the lifecycle of the plugins.<br /><br />Jopr and Embedded Jopr are related in one important aspect - they share alot of the same code! The beauty of the architectural design of Jopr is such that you can take the Jopr Plugin Container and its plugins and embed them in any Java virtual machine - including a virtual machine that is running a JBossAS instance.<br /><br />Embedded Jopr uses the Jopr Plugin Container and embeds it directly inside a web application (aka .war) that is deployed inside JBossAS, allowing you to directly manage that JBossAS instance. In other words, Embedded Jopr is managing the very JBossAS instance in which it lives.<br /><br />Jopr, on the other hand, has a standalone Jopr Agent and it is this Jopr Agent that embeds the Jopr Plugin Container. The Jopr Agent is able to do a few things Embedded Jopr cannot do but the main difference is that the Jopr Agent can communicate with a Jopr Server cloud, allowing it to participate in a full-fledged management enterprise environment.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_I05vMaF_3Ec/Se6MRo7oq6I/AAAAAAAAABk/UBMAQGMzSF8/s1600-h/embedded-jopr.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 213px; height: 320px;" src="http://3.bp.blogspot.com/_I05vMaF_3Ec/Se6MRo7oq6I/AAAAAAAAABk/UBMAQGMzSF8/s320/embedded-jopr.png" alt="" id="BLOGGER_PHOTO_ID_5327349643899087778" border="0" /></a>The next couple diagrams illustrate these two different models. First is Embedded Jopr. Notice that everything here is directly hosted within a managed JBossAS instance. Inside JBossAS are, of course, its own internal services, one of which is Embedded Jopr. Embedded Jopr is used to manage this JBossAS instance. It is analogous to the jmx-console, only with more powerful features. You will notice that inside Embedded Jopr is code for the actual user interface as well as the Jopr Plugin Container. This Jopr Plugin Container code is identical to the plugin container used by Jopr - not a single line of code is different in this Jopr Plugin Container as compared to the plugin container deployed within the Jopr Agent (which we'll talk about later on). The plugins are, also, identical. In fact, this is designed in such a way that allows others to write custom Jopr plugins and deploy them, not only in Jopr Agents, but in Embedded Jopr as well. The hope is that Embedded Jopr will be able to support the management of other services deployed in its JBossAS instance, thus allowing you to enhance and extend Embedded Jopr with off-the-shelf plugins or your own custom plugins to manage your own custom services. There are many opportunities to extend Embedded Jopr in this way - think of all the different components that you could deploy in JBossAS that you'd want to manage (Portal, Seam, jBPM, Drools, JBossTS, JBossCache, etc.); if you had plugins for them, you'd just deploy them inside Embedded Jopr's plugin container and you could immediately begin to manage them. You could then take those plugins and later deploy them inside a Jopr Agent (without the need to touch or even rebuild your plugins) and have them be used within a full-fledged Jopr management enterprise. This concept has been proven to work by the mere fact that Embedded Jopr exists. Embedded Jopr already ships with some plugins common with the Jopr Agent (the JMX plugin is one of them).<br /><br />Now let's look closer at the deployment model of Jopr.<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_I05vMaF_3Ec/Se6TMiGLg3I/AAAAAAAAABs/Hzpv-dtIIrc/s1600-h/jopr.png"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 252px; height: 320px;" src="http://2.bp.blogspot.com/_I05vMaF_3Ec/Se6TMiGLg3I/AAAAAAAAABs/Hzpv-dtIIrc/s320/jopr.png" alt="" id="BLOGGER_PHOTO_ID_5327357252746314610" border="0" /></a> First, notice that the Jopr Agent, at its core, is the same Jopr Plugin Container that we saw in Embedded Jopr. It's the same code. Reusability is a wonderful thing. And remember, not only is the Jopr Plugin Container reused, but the plugins themselves are 100% reuseable, too. But now notice the difference. The Jopr Agent is standalone - its separate from any managed JBossAS instance. In fact, the Jopr Agent can manage multiple JBossAS instances! Not only that, but it can also manage any number of products or components - PostgreSQL databases, remoted Java Virtual Machines, operating system services, and anything else you want (even your own custom products or components), as long as you have a plugin that is capable of managing them (I won't get into plugin development topics, see the <a href="http://rhq-project.org/display/RHQ/Plugin+Community">Plugin Development wiki</a> to learn how you can write your own plugins).<br /><br />The next major difference you see here is the Jopr Agent can communicate with the Jopr Server cloud (which consists of 1 or more Jopr Servers with persistence storage backing those servers). The Jopr Server provides additional capabilities not available to Embedded Jopr. These include the ability to: persist historical metric data from your managed resources, provide an alerting mechanism to notify IT administrators when something goes wrong within your managed environment, provide a persisted audit trail for security tracking, provide persistence storage of events occuring within your managed resources, and many other things.<br /><br />Hopefully, I've answered the basic question, "What is Embedded Jopr and how is it different from Jopr?".<br /><br />What I've also hoped to convey was a bit of an architectural overview of the Jopr Plugin Container and its management plugins and how their reusability is exploited to make development and use of both Embedded Jopr and Jopr much more easier.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-7313161405798787848?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com3tag:blogger.com,1999:blog-6944050187568241059.post-12909163545112357772009-04-04T21:06:00.000-07:002009-04-04T21:32:00.754-07:00I wanted to collect some SVN statistics for some codebases and found a very nice project that generates useful SVN reports and graphs. The project is called <a href="http://www.statsvn.org/">StatSVN</a>.<br /><br />It's very easy to use. I'll include both a UNIX and Windows script here so no matter what platform you are on, you can easily use this. Each script is only about 20 lines long.<br /><br />First, download the statsvn.jar from the <a href="http://www.statsvn.org/downloads.html">StatSVN download page</a>.<br /><br />Now, put the run-svnstats script (which I give below) in the same directory as the statsvn.jar file. If on UNIX, use the first script; Windows use the second script:<br /><br /><h4>UNIX Script</h4><br /><pre><blockquote><br />#!/bin/sh<br /><br />WORKING_COPY=/home/me/perf/src/rhq/trunk/modules<br />STATS_GEN_DIR=/home/me/svnstats-gen<br />START_END_REVISIONS=10:HEAD<br /><br />SVN_LOGFILE=$STATS_GEN_DIR/svn.log<br />STATSSVN_JAR=$STATS_GEN_DIR/statsvn.jar<br /><br />cd $WORKING_COPY<br />svn up<br />svn log -v --xml -r $START_END_REVISIONS > $SVN_LOGFILE<br />cd $STATS_GEN_DIR<br />if [ -d svnstats ]; then<br /> rm -rf svnstats<br />fi<br />mkdir -p svnstats<br />cd svnstats<br />java -jar $STATSSVN_JAR -concurrency-threshold 2000 -threads 50 $SVN_LOGFILE $WORKING_COPY<br /><br />echo SVN Statistics are now located here: `pwd`<br /></blockquote></pre><br /><br /><h4>Windows Script</h4><br /><pre><blockquote><br />@echo off<br /><br />set WORKING_COPY=C:\source\rhq\trunk\modules<br />set STATS_GEN_DIR=C:\svnstats-gen<br />set START_END_REVISIONS=10:HEAD<br /><br />set SVN_LOGFILE=%STATS_GEN_DIR%\svn.log<br />set STATSSVN_JAR=%STATS_GEN_DIR%\statsvn.jar<br /><br />cd %WORKING_COPY%<br />svn up<br />svn log -v --xml -r %START_END_REVISIONS% > %SVN_LOGFILE%<br />cd %STATS_GEN_DIR%<br />if exist svnstats rmdir /S /Q svnstats<br />mkdir svnstats<br />cd svnstats<br />java -jar %STATSSVN_JAR% -concurrency-threshold 2000 -threads 50 %SVN_LOGFILE% %WORKING_COPY%<br /><br />echo SVN Statistics are now located here: %STATS_GEN_DIR%\svnstats<br /></blockquote></pre><br /><br />Lastly, you just have to edit the script to match your environment. Change the three variables defined at the top:<br /><br /><ul><br /><li>WORKING_COPY: the full path to the SVN working copy as found on your local file system. This SVN working copy <b>will be svn updated</b> as part of the script, so make sure you don't mind having this working copy get updated to the latest revision.</li><br /><li>STATS_GEN_DIR: the full path of the location where you stored the statsvn.jar and the scripts. The generated SVN log and the final HTML pages, reports and images will get stored here as well.</li><br /><li>START_END_REVISIONS: the starting revision number and ending revision number of your SVN working copy. Only SVN statistics for that revision range will be reported on. This should be in SVN format (e.g. 1234:5678 or 1:HEAD)</li><br /></ul><br /><br />That's it. Run the script and out comes an "svnstats" directory with a full set of HTML pages and images (this svnstats directory will be located under the STATS_GEN_DIR directory). Just copy the svnstats directory to a webserver (e.g. Apache's htdocs directory) and point your browser to the svnstats location and you can now peruse your SVN reports.<br /><br />You can even run this script as a cron job to automatically update your stats periodically. Just have STATS_GEN_DIR point to a directory that your webserver can serve up and make sure your START_END_REVISION takes into account HEAD (so new checkins will get reported).<br /><br />I thought this was a very simple yet cool way to get SVN stats for any SVN project; hope you can find this as helpful as it did me.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-1290916354511235777?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-31737532937950218482009-03-26T07:29:00.000-07:002009-03-26T09:39:23.305-07:00I'm surprised you are still reading this, given the nebulous title of this blog :). But I'm glad you are here. I want to further explain what my <a href="http://management-platform.blogspot.com/2009/03/correlating-events-with-jopr.html">last blog</a> was really trying to convey.<br /><br />Jopr is essentially an abstract management framework - at its core, Jopr does not understand about the concrete managed resources it actually is managing. Specific, concrete knowledge of the manage resource implementations is pushed out to the edges - in the agent plugins themselves.<br /><br />But what Jopr knows, at its core, are the different management "facets" that those managed resources support. What is a "management facet"? Simple - a facet is a subset of functionality that all managed resources may or may not support. Facets are orthogonal - a managed resource and its plugin can support any number of facets in any combination.<br /><br />Jopr supports several types of facets - for example, the <span style="font-style: italic;">measurement facet</span> (for collecting metric data), the <span style="font-style: italic;">operation facet</span> (for controlling managed resources), the <span style="font-style: italic;">configuration facet</span> (for retrieving and setting resource configurations), the <span style="font-style: italic;">content facet</span> (for pushing and pulling file content to/from managed resources), the <span style="font-style: italic;">event facet</span> (to emit asynchronous event information such as a managed resource's log file messages) and others.<br /><br />There are numerous advantages for having an abstract management platform that can handle all of these different management facets for many different kinds of managed resources. One of these advantages is the ability to correlate and process information across all the different facets.<br /><br />Looking at Jopr's summary timeline, and what "cross-facet correlation" means soon becomes very apparent:<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_I05vMaF_3Ec/Scujj9zi_-I/AAAAAAAAAA8/Z0DcU_9hR7w/s1600-h/timeline.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 226px;" src="http://3.bp.blogspot.com/_I05vMaF_3Ec/Scujj9zi_-I/AAAAAAAAAA8/Z0DcU_9hR7w/s320/timeline.png" alt="" id="BLOGGER_PHOTO_ID_5317523623322517474" border="0" /></a>This timeline (with time moving from left to right), in one view has correlated information such as:<br /><ul><li>when the resource was up and down; notice the background color - light green means the resource was up at that time and a red background means the resource was down with grey meaning it is unknown what the state of the resource was <span style="font-style: italic;">(measurement facet) </span></li><li>when events (e.g. log messages) happened, how many of them occurred and how severe these events were (severity is shown in the badging of the events icons). <span style="font-style: italic;">(event facet)</span></li><li>when alerts were raised and how severe those alerts were (alerts are shown as flag icons with their severity indicated by their color). Alerting isn't an actual management facet per-se, the alert subsystem is provided to all managed resources with no effort needed on the part of the plugin - alerts are provided "for free".</li><li>when a resource's configuration was changed and whether or not that change was successful or if it failed <span style="font-style: italic;">(configuration facet)<br /></span></li><li>when an operation was invoked on a resource and whether or not that attempt to control the resource was successful or not <span style="font-style: italic;">(operation facet)</span></li></ul>This is what "cross-facet correlation" is all about. How is this helpful? Well, if I see a gap of events in my timeline, where that gap has a background of red, that tells me very quickly that I am not receiving any events because the resource is down! If I see that I changed the configuration of a resource, and soon after I see a flood of warning events and then perhaps followed by a red background, I can immediately begin to suspect that that configuration change caused an adverse affect on the behavior of my resource. If I executed an operation and soon after see one or more alerts trigger, I should start by investigating if that operation caused the resource to act strangely.<br /><br />Cross-facet correlation occurs in other areas of the Jopr UI (and the sky's the limit to where we can take Jopr from here in the future).<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_I05vMaF_3Ec/ScunlB43P5I/AAAAAAAAABE/i5RaKlluM3U/s1600-h/summary.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 320px; height: 310px;" src="http://1.bp.blogspot.com/_I05vMaF_3Ec/ScunlB43P5I/AAAAAAAAABE/i5RaKlluM3U/s320/summary.png" alt="" id="BLOGGER_PHOTO_ID_5317528039644938130" border="0" /></a> For example, here is the summary tab for my agent. Notice how all the different facets are combined into a single view so at a glance you can see what is currently going on with this agent resource.<br /><br />I can see that I had one operation, out of the past three, that failed. I can see that I recently updated the configuration of this resource several times, one of which failed. I can see what the current measurements are for this resource, and what events and alerts have been logged, when they happened and their severity.<br /><br />All of this data is linked to their respective pages in the UI - if I want to see more about the alerts, I click the alert links. More about the operations? Click the operation links. And so on.<br /><br />And because Jopr has abstracted the facets so they are applicable across any number of managed resources, we can manage all different kinds of resources - JBoss Application Servers, Apache Web Servers, Tomcat Web Application Servers, hardware boxes, operating system services, even Jopr itself - and <span style="font-weight: bold;">we reuse the same UI pages, the same code, and the same look &amp; feel - no additional code needs to be introduced to the server to support additional types of managed resources</span><span style="font-weight: bold;">!</span><br /><br />For example, above you see the summary for the Jopr agent. <a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_I05vMaF_3Ec/ScupiQH-sII/AAAAAAAAABM/ttYjnrdlWCU/s1600-h/summary2.png"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 320px; height: 241px;" src="http://1.bp.blogspot.com/_I05vMaF_3Ec/ScupiQH-sII/AAAAAAAAABM/ttYjnrdlWCU/s320/summary2.png" alt="" id="BLOGGER_PHOTO_ID_5317530190950084738" border="0" /></a>What's the summary information look like for a JBoss Application Server that I am managing? You can see this here. Notice that the same look &amp; feel, the same UI pages and code is used, the same SQL queries - everything is reused. No additional integration is needed on the server side. But notice the difference. The agent resource supports the "configuration facet" - which is why you saw the "Configure" tab and the configuration update information earlier. But the agent does not support the "content facet" (the agent does not push or pull content over the Jopr content subsystem). But look at the JBossAS resource - it does not support the configuration facet (so you don't see the Configure tab and there are no recent config updates) but it <span style="font-style: italic;">does </span>support the content facet (you can see the Content tab along with some recent package history). The JBossAS resource component in the agent will send up to the Jopr server information about what packages (e.g. jar libraries) it has installed. If a resource supports it, you can even ship down updated packages to the resource (for example, to send down new jars that incorporate bug fixes).<br /><br />The above just talks about correlating cross-facet information for a specific resource. What if I want to see information across my entire inventory? What if I want to see all of the alerts that were triggered by Jopr, regardless of the resource that triggered the alert. What if I want to see all the configuration changes made to my environment, regardless of which resource was reconfigured?<br /><br />Again, because Jopr is abstract in nature, we can scan the inventory history and aggregate this kind of information. Below you see that we can view all the alerts and all configuration changes - you can even filter the results if you only care about a subset of the data.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_I05vMaF_3Ec/ScurQLW-WrI/AAAAAAAAABU/E80Iagi6UuY/s1600-h/subsystem-alerts.png"><img style="margin: 0pt 10px 10px 0pt; display: block; text-align: center; cursor: pointer; width: 320px; height: 306px;" src="http://4.bp.blogspot.com/_I05vMaF_3Ec/ScurQLW-WrI/AAAAAAAAABU/E80Iagi6UuY/s320/subsystem-alerts.png" alt="" id="BLOGGER_PHOTO_ID_5317532079456410290" border="0" /></a><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_I05vMaF_3Ec/ScurXfF2cXI/AAAAAAAAABc/vbqdl9vEGx8/s1600-h/subsystem-config.png"><img style="margin: 0pt 0pt 10px 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 250px;" src="http://1.bp.blogspot.com/_I05vMaF_3Ec/ScurXfF2cXI/AAAAAAAAABc/vbqdl9vEGx8/s320/subsystem-config.png" alt="" id="BLOGGER_PHOTO_ID_5317532205012382066" border="0" /></a><br />I think this clearly shows how an abstract management platform that supports multiple management facets can provide tremendous value to anyone managing a network of hardware and software products.<br /><br />Well, that's all I have to say on this subject (for now). I hope this makes it a bit more clear what Jopr brings to the table and its value-add that it can bring to your IT environment.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3173753293795021848?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-45163324646562131662009-03-25T23:02:00.000-07:002009-03-26T00:41:10.990-07:00Today, I rediscovered how nice Jopr really is when I enhanced the agent plugin so it can track the agent log files.<br /><br />In just an hour or two, I added a feature allowing you to enable log tracking for the agent itself. If you enable this feature, you can view the agent's log messages directly in the Jopr UI. This enables you to see what's going on inside an agent, and you can corrolate those log events with other types of changes happening to your agent (e.g. configuration changes, monitoring data, alerts, etc).<br /><br />Let me discuss some of the nice things this allows you to do. Remember, even though this is a concrete example using the agent itself as the managed resource, everything I'm about to discuss can be done for your own managed resources because all of these features are abstract and can be utilized by any plugin, should the plugin developer choose to use them. This is what an abstract management framework provides you and is what Jopr is all about.<br /><br />First, I took between one and two hours to add code to the agent plugin in order for it to utilize the event subsystem provided by Jopr. A little bit of Java code, a little bit of XML and I went from nothing to being able to fully integrate the agent log files into the Jopr events subsystem.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_I05vMaF_3Ec/ScslmG7cStI/AAAAAAAAAAM/Q-woqu-tLvA/s1600-h/events.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 320px; height: 204px;" src="http://2.bp.blogspot.com/_I05vMaF_3Ec/ScslmG7cStI/AAAAAAAAAAM/Q-woqu-tLvA/s320/events.png" alt="" id="BLOGGER_PHOTO_ID_5317385121666190034" border="0" /></a>OK, so, what does this get you? First, and the most obvious, is you can now view the log message events from within the Jopr UI. You do not have to remotely log into the machine where the agent is running to view its log messages. See the image here on the left - this is the event history view. Because the agent emits its log messages as events, this event history view is essentially browsing the log file, with the added bonus of being able to filter the view based on the log message content and the severity of the messages (INFO, WARN, ERROR, etc).<br /><br />Second, you can view the events corrolated with monitoring data - this allows you to see what your resource's measurements looked like at the time<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_I05vMaF_3Ec/ScsohjR0GQI/AAAAAAAAAAc/-Kp8nm-SrFk/s1600-h/monitor-tab.png"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 320px; height: 256px;" src="http://4.bp.blogspot.com/_I05vMaF_3Ec/ScsohjR0GQI/AAAAAAAAAAc/-Kp8nm-SrFk/s320/monitor-tab.png" alt="" id="BLOGGER_PHOTO_ID_5317388341911755010" border="0" /></a> the resource was logging messages (which might help you in diagnosing a problem). Take a look at the bottom of the graphs and you can see the different colored icons to indicate the highest severity of events that occurred in any timeslice shown on the graph. Here you can see INFO (green), WARN (yellow) and DEBUG (blue) messages occurring in different times. And notice how you can corrolate those times with measurement data and event activity.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_I05vMaF_3Ec/ScsqXD2oinI/AAAAAAAAAAk/_DH8zEFG6jY/s1600-h/event-viewer.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 320px; height: 257px;" src="http://2.bp.blogspot.com/_I05vMaF_3Ec/ScsqXD2oinI/AAAAAAAAAAk/_DH8zEFG6jY/s320/event-viewer.png" alt="" id="BLOGGER_PHOTO_ID_5317390360700815986" border="0" /></a>You can even drill down into the log file directly from this view so you can read the first several log messages that occurred within a narrow span of time. Again, this might help you to diagnose a problem if, using this agent resource as an example, you can see the actual ERROR log messages that occurred in or around the same time you saw the average execution time for sent commands starting to go up.<br /><br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_I05vMaF_3Ec/ScsrzY98onI/AAAAAAAAAAs/QdSh42Yrk8M/s1600-h/alert-def.png"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 320px; height: 296px;" src="http://3.bp.blogspot.com/_I05vMaF_3Ec/ScsrzY98onI/AAAAAAAAAAs/QdSh42Yrk8M/s320/alert-def.png" alt="" id="BLOGGER_PHOTO_ID_5317391946916602482" border="0" /></a>Jopr can be more proactive with these events/log messages, as well. You can define an alert definition such that you will get notified (via email for example) if the agent emits a log message at the ERROR severity level. You can even be alerted when a specific log message is emitted (e.g. have you ever wanted to be emailed when your application spits out an OutOfMemoryError log message? Now you can!). The alert definition UI page allows you to set this up.<br /><br />And finally, you can use the summary timeline to further corrolate log message events with other things that have happened to this agent resource. For example, notice that I can see when my event messages occurred (and what their highest severities were) on this timeline correlated with other things that happened to this agent, such as when its configuration changed and when alerts were triggered. You would also see when operations were invoked on this timeline as well had any operations been executed during this timeframe that the timeline is showing. In effect, you get a wholistic view of what happened to this agent resource, across all the different management facets: configuration changes, control executions, alerts, events, etc!<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_I05vMaF_3Ec/ScstTZ0H_sI/AAAAAAAAAA0/TWommqd-G-E/s1600-h/timeline.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 210px;" src="http://2.bp.blogspot.com/_I05vMaF_3Ec/ScstTZ0H_sI/AAAAAAAAAA0/TWommqd-G-E/s320/timeline.png" alt="" id="BLOGGER_PHOTO_ID_5317393596411281090" border="0" /></a><br />Once again, I must emphasize the fact that you can get all of this functionality, too - and all you need is to write a plugin that talks to your managed resource and provides the raw data to Jopr. You get everything else for free - the corrolated timeline, the monitoring graphs, alerting and more.<br /><br />So, even though the above screenshots show this functionality for the agent resource, the UI and all of these capabilities would be the same for <span style="font-style: italic;">any </span>resource you want to manage, so long as that managed resource has a plugin that provides the same kind of raw information. In my case, I just had to spend a couple of hours to get the agent to report events from its log files, and Jopr took care of the rest. For example, I did not have to do anything to enable the alerting capabilities or the corrolating timeline. <span style="font-style: italic;">That </span>is the value that Jopr brings to the table!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4516332464656213166?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2tag:blogger.com,1999:blog-6944050187568241059.post-56710581878196763242009-03-23T11:52:00.001-07:002010-06-17T05:28:30.621-07:00<a href="http://pilhuhn.blogspot.com/">Heiko</a> has just demonstrated another way that the agent-side plugin container can be embedded in any Java VM.<br /><br />We've already proven that this concept works because the plugin container has already been embedded in a few places: not only does the agent itself embed the plugin container, but our unit tests do it when the validity of plugins needs to be tested and also the <a href="http://www.jboss.org/embjopr/">Embedded Jopr</a> project does it by embedding the plugin container directly in a JBossAS5 application server!<br /><br />But <a href="http://pilhuhn.blogspot.com/2009/03/working-on-standalone-plugincontainer.html">what Heiko has done</a> is go a step further by providing a very small, yet useful, wrapper around the plugin container to support plugin developers (it is called the "<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=etc/standalone-pc;hb=master">standalone plugin container</a>"). It is "standalone" because you no longer need to install and run a full Jopr environment (a server and a database) in order to test your plugin's functionality.<br /><br />If you are writing a custom plugin, just use the standalone plugin container and deploy and execute your plugin. This means you just take an existing agent distribution, and use a very simple script to start the standalone plugin container (there are <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=etc/standalone-pc/standalone.bat;hb=master">Windows</a> and <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=etc/standalone-pc/standalone.sh;hb=master">UNIX</a> versions of the script). The simplicity of these scripts border on trivial. Under the covers, all this does is run a new main class that embeds the plugin container, as opposed to the original AgentMain class (which does all the complex agent-to-server communications). This new standalone plugin container will accept commands on the prompt to help you exercise your plugin code - a great help for those writing plugins. See the <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=etc/standalone-pc/README.txt;hb=master">README</a> for some install instructions.<br /><br />This type of capability sort of existed in the old JBoss ON 1.x code base - but its old (now obsolete) plugin model was never this modular and could never have been this easily embedded in so many different ways.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5671058187819676324?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-86948207920971709302009-02-28T20:47:00.000-08:002009-03-04T18:35:24.667-08:00Jopr 2.2 is almost ready for release. I was going to build some nice flash demos, but there is so much new to cover, I didn't know where to begin! I wanted to get the message out quickly though, so, rather than wait for my flash demos to be completed, I decided to simply take some screen snapshots and post them up on the Jopr wiki.<br /><br />Take a <a href="https://www.jboss.org/community/docs/DOC-13381">sneak peek here</a> at what's coming - I think you will find Jopr 2.2 to be a great leap forward. It is much better than anything JBoss ON 1.x ever could or did deliver, and it is a major milestone even from the latest releases of JBoss ON/Jopr.<br /><br />I am very proud of what we have been able to accomplish as a team - this is definitely the most feature-rich release of any previous JBoss ON/Jopr release.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-8694820792097170930?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-86104663270055153362009-02-15T10:47:00.000-08:002009-03-04T18:35:24.667-08:00I had a need to do some quick profiling of a Java application. I didn't want to spend alot of time (or money) setting up a profiler and my app to run within that profiler's environment. But it turns out that SUN's Java 6 distribution ships with some very handy tools that you can use to do what I wanted without setting up anything special - didn't even have to pass in any special set of -D system property definitions.<br /><br />First, run your application and get it into a state that you want to analyze. Then perform the following steps:<br /><ol><br /><li>Run "<span style="font-style: italic;">jps</span>" to determine the process ID (aka pid) of your Java application. BTW: this is a cool nugget in and of itself. I've been so used to doing "ps -elf | grep java" or "pgrep java" to find my running Java applications that I never bothered looking for a simpler way. "jps" dumps the pid and the simple name of the Java main class - and most of the time this is all you are looking for. But you can also get the fully qualified main class names and the argument lists - use "jps -help" to see the usage syntax. This is a very helpful tool all by itself.</li><br /><li>Run "<span style="font-style: italic;">jmap -dump:format=b,file=dump.dat &lt;pid></span>" where &lt;pid> is your Java applications's pid that you found in the previous step. This will dump information about your Java VM's memory in the file "dump.dat" and will be used by jhat in the next step.</li><br /><li>Run "<span style="font-style: italic;">jhat -J-Xmx512m dump.dat</span>" to start the Java heap analysis tool (jhat). This will start an HTTP listener on port 7000. The -J option lets me configure the jhat VM, which is needed if you have a large dump to analyze.</li><br /><li>Point a browser to http://localhost:7000 and begin browsing around.</li><br /></ol><br /><br />This jhat tool let's you examine the objects and classes currently found within your VM. It's nothing glorious and certainly not as well featured as many of the other commercial and open-source profilers out there. But, it comes with the JDK, and requires absolutely no setup. It can't get much easier to perform a quick heap analysis than this.<br /><br />Credit goes to Frank Kieviet, whose <a href="http://blogs.sun.com/fkieviet/entry/how_to_fix_the_dreaded">blog</a> brought my attention to these tools.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-8610466327005515336?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com1tag:blogger.com,1999:blog-6944050187568241059.post-60330160397405366892009-02-09T09:57:00.000-08:002009-03-04T18:35:24.668-08:00Well, OK, Jopr won't let you know if there is a <a href="http://en.wikipedia.org/wiki/Cherry_bomb">cherry bomb</a> strapped to your laptop, but, it can alert you if something is physically wrong with your hardware that may cause it to malfunction, assuming your hardware can provide Jopr with relevent data.<br /><br />Take for example, the <a href="http://www.jroller.com/ghinkle/entry/jopr_and_the_angry_demo">premature ending of Greg Hinkle's Jopr demo</a> a few weeks ago. Greg's computer crashed for what seemed like an unknown reason - but because he was running Jopr on his laptop (which had his experimental hardware plugin deployed), he was actually able to use Jopr to figure out what the problem was. Turns out, his laptop was slowly overheating - which ended up crashing his box. Check out his <a href="http://jroller.com/ghinkle/resource/temp_burn.png">blog</a> to see what the graph looked like.<br /><br />Too bad he didn't have any alerts set up on the temperature metric - he could have received an email from Jopr telling him his machine was about to blow up because his hardware reached 100 degrees :)<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-6033016039740536689?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-3650259723936672932009-01-25T08:38:00.000-08:002009-03-04T18:35:24.668-08:00If you write code that creates classloaders, you need to know about this bug:<br /><br /><a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5041014">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5041014</a><br /><br />It is very insidious and something I just came across myself in some code.<br /><br />You normally only have to worry about this if you are writing code that creates and destroys classloaders (for example, if you have some kind of pluggable architecture where a pluggable component found in a jar file gets its own classloader, and you want that pluggable component to be hot-deployable - that is, you want to be able to overwrite or modify that jar file with updated code). In Jopr's case, this happens on the agent - each "product plugin" (e.g. the JBossAS plugin, or the Postgres plugin, etc) has its own classloader, managed separately and kept independent of other plugin classloaders (there is a dependency model in place, but ignore that for this discussion).<br /><br />Well, this VM bug is so bad it seems that anytime a classloader loads in a jar file, that jar file's file descriptor remains open for the lifetime of that VM (in other words, the classloader never calls JarFile.close() for all the jar files it previously streamed content from). At least that's what the bug report infers and what I'm seeing when I was debugging this. There is a <a href="http://blogs.sun.com/quinn/entry/tool_for_diagnosing_failed_glassfish">nifty tool from Timothy Quinn</a> that he used to track issues in Glassfish, but this tool is useful to track this kind of problem for any application, not just Glassfish - in fact, I used it to debug the issue in the Jopr agent. This bug manifested itself in the Jopr agent when hot-deploying agent plugins on Windows (Windows has the "feature" of not being able to manipulate files that are locked by others). I suspect similar issues will occur on UNIX because, even though UNIX doesn't do the file locking that Windows does, the file descriptors are still open and copying a file with the same name over the opened file will probably just create a second file descriptor.<br /><br />The worst part about this is - there is no real workaround. The Jopr agent has its own classloader implementation - it is very basic and extends java.net.URLClassLoader to reuse most of its functionality. But the Java classloader API has no public, protected or package-scoped method or data field that you can override or access within URLClassLoader to help workaround the problem.<br /><br />To actually fix the problem, it is simple - when you know you are done with a classloader, you just need to have that classloader close all .jar files it previously had opened. Alas, there is no "close" type method on the classloader object - there is absolutely no way to tell a classloader "I am done with you, clean up any resources you have open".<br /><br />Once a classloader opens a jar file, that jar file's file descriptor remains open by the operating system for the lifetime of the VM. I find this completely unacceptable - this is clearly a design flaw that slipped through the cracks when the Java API was conceived and implemented. In order to support hot-deployable Java code, one would need to destroy and recreate classloaders. The current Java implementation does not make it easy to do this (requiring people to write their own classloader implementations from scratch does not meet the definition of "easy-to-do" and doesn't that defeat the purpose of OO and code reuse anyway?).<br /><br />So, how do you support hot-deployable code and not see this bug? There are two main ways to do this as I see it:<br /><br />1) write your own classloader implementation that allows you to close the open file descriptors when the classloader is no longer needed<br />2) copy the jar files that a classloader needs to a temporary location and put the temporary jars in the classloader (NOT the original jar files). When you need to hot-deploy an updated jar file, simply copy that new jar to a new temporary location, throw away the old classloader (which still has the file descriptor open, but its the old temporary jar file) and create a new classloader that opens the new temporary jar file. This sucks because if you hot-deploy frequently, you may run into your limit of the number of allowed open file descriptors (along with the problem that Windows presents - that being you can't delete the old temporary jar files until your VM exits).<br /><br />Anyway, here is some code you can use to "workaround" this issue. It is a major hack - it only works if you are running in a SUN VM and because it relies on the implementation of internal SUN classes and code, you may break in the future should SUN decide to change how these classes are implemented (however, the good thing about this code is it has <i>no</i> compile time dependencies on any SUN-specific classes). I tested this code on SUN's Java6 JRE.<br /><br />This method needs to be placed in your classloader that extends URLClassLoader. It uses reflection to iterate over the set of currently opened jar files as found in a private data member (URLClassLoader.ucp.loaders) of the classloader you want to discard. After running this code, I verified that no more jar files are left open.<br /><br /><hr/><pre>public void close() {<br />try {<br /> Class clazz = java.net.URLClassLoader.class;<br /> java.lang.reflect.Field ucp = clazz.getDeclaredField("ucp");<br /> ucp.setAccessible(true);<br /> Object sun_misc_URLClassPath = ucp.get(this);<br /> java.lang.reflect.Field loaders = <br /> sun_misc_URLClassPath.getClass().getDeclaredField("loaders");<br /> loaders.setAccessible(true);<br /> Object java_util_Collection = loaders.get(sun_misc_URLClassPath);<br /> for (Object sun_misc_URLClassPath_JarLoader :<br /> ((java.util.Collection) java_util_Collection).toArray()) {<br /> try {<br /> java.lang.reflect.Field loader = <br /> sun_misc_URLClassPath_JarLoader.getClass().getDeclaredField("jar");<br /> loader.setAccessible(true);<br /> Object java_util_jar_JarFile = <br /> loader.get(sun_misc_URLClassPath_JarLoader);<br /> ((java.util.jar.JarFile) java_util_jar_JarFile).close();<br /> } catch (Throwable t) {<br /> // if we got this far, this is probably not a JAR loader so skip it<br /> }<br /> }<br />} catch (Throwable t) {<br /> // probably not a SUN VM<br />}<br />return;<br />}<br /></pre><hr/><br /><br />If you happen to be using JNI (native libraries), you might also have to play games like the above to close the JNI jars too (same cavets as above apply regarding this needing to access the SUN implementation code). You can add this code to the close() method above:<br /><br /><hr><pre><br />// now do native libraries<br />clazz = ClassLoader.class;<br />java.lang.reflect.Field nativeLibraries = clazz.getDeclaredField("nativeLibraries");<br />nativeLibraries.setAccessible(true);<br />java.util.Vector java_lang_ClassLoader_NativeLibrary =<br /> (java.util.Vector) nativeLibraries.get(this);<br />for (Object lib : java_lang_ClassLoader_NativeLibrary) {<br /> java.lang.reflect.Method finalize =<br /> lib.getClass().getDeclaredMethod("finalize", new Class[0]);<br /> finalize.setAccessible(true);<br /> finalize.invoke(lib, new Object[0]);<br />}<br /></pre><hr/><br /><br />But even if you do this, I'm still not sure everything will work due to yet more SUN VM bugs (well, I think these are all basically the same bug):<br /><br /><a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4299094">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4299094</a><br /><a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4642062">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4642062</a><br /><a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4286309">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4286309</a><br /><br />In the end, the Jopr agent didn't really need to do the above. I found that in the Jopr agent code, it was creating temporary classloaders unnecessarily which was locking the plugin jars. Once I removed the unnecessary classloaders from being created, the agent hot-deployment worked just fine since the plugin jars no longer got locked. For the record, the Jopr agent uses method #2 as described above to do its hot-deployment.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-365025972393667293?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com7tag:blogger.com,1999:blog-6944050187568241059.post-53122263912691925242009-01-10T12:05:00.000-08:002009-12-03T06:23:48.264-08:00I have completed the agent auto-update functionality. This provides the ability for a Jopr agent running in an environment to automatically detect that it needs to be updated and does so without the need for manual intervention.<br /><br />The cool thing about this is it is completely cross platform! I've testing on Windows and Linux and I see no reason why this wouldn't work on other UNIX flavors such as HP-UX, AIX and MacOS.<br /><br />Here's the basics of how it works:<br /><br />When a Jopr Agent tries to connect or register with a Jopr Server, that server verifies the version of that agent. If the agent is not a compatible version, the server will forbid that agent from connecting/registering and will tell the agent it needs to update itself.<br /><br />At this point, the agent will shutdown all of its internals, download the latest agent update binary (either from the server or some other download location previously configured in the agent), fork another Java VM that will unpackage the new agent binary and update the old agent with the new binary. The old agent will shutdown its VM and the new agent VM will be started.<br /><br />From an administrator's point of view, this all happens under the covers and automatically and the agent just looks like it goes offline for a minute or two before coming back online.<br /><br />Tangential to this, is the addition to several features to the agent plugin. The agent resource metadata now includes several more child services that allow you to configure your agent without having to manually log onto the agent box (i.e. we are using Jopr to manage Jopr!). You can now even change agent JVM settings and restart the agent with those new settings (in case you need to change a -Xmx option, for example). You can read <a href="http://www.rhq-project.org/display/JOPR2/Using+the+Agent+Plugin">this wiki page</a> to learn about these new plugin features.<br /><br />All of this code will be forthcoming in our next RHQ/Jopr release.<br /><br />Here's some additional documentation you can read if curious:<br /><a href="http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-PreparingYourAgentToBeAutoUpdatable"><br /></a><a href="http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-PreparingYourAgentToBeAutoUpdatable"></a><a href="http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-PreparingYourAgentToBeAutoUpdatable">http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-PreparingYourAgentToBeAutoUpdatable</a><br /><a href="http://www.rhq-project.org/display/RHQ/Design-AgentAutoUpdate">http://www.rhq-project.org/display/RHQ/Design-AgentAutoUpdate</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5312226391269192524?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-5003983119613840142008-12-06T10:39:00.000-08:002009-12-03T06:19:45.364-08:00One of the most requested enhancements for Jopr is for an easier way to perform agent installations. Because of this, efforts are underway to ease the pain of agent deployment.<br /><br />For small Jopr environments, you can take the agent distributions as-is, install them and individually set up each agent by answering the setup questions at startup. This can be tedious the more machines you have. It would be nice if you could bundle your own "golden distro", push them out to all your machines and with no additional configuration or manual setup required, have the agents "just work".<br /><br />With the current Jopr code, this is now possible. Following these steps, you can bundle your own agent into one "golden distro". We call it the "golden distro" because it's the one and only distro you will need in order to install all the agents in your environment. That distro will be able to install all your agents and have them start and configure themselves with no further setup required.<br /><br /><ol><br /><li>Unpackage the agent distribution that comes out-of-box. This is the starting point to build your own "golden distro".</li><br /><li>Next, consider what, if any, customized environment variables you need to set for your agents. If there are any, edit <span style="font-weight: bold;">rhq-agent-env.sh</span> (<span style="font-weight: bold;">rhq-agent-env.bat</span> for agents that are to run on Windows). For example, if there are any -XX options you want to pass to your agent's JVM, set <span style="font-weight: bold;">RHQ_AGENT_ADDITIONAL_JAVA_OPTS</span> in the -env script appropriately.</li><br /><li>The next several steps involve editing the <span style="font-weight: bold;">conf/agent-configuration.xml</span> file. So, load that file in an editor. Here is where you will preconfigure your agents in order for the agents to start up and successfully configure and initialize themselves, without requiring an admin to answer the setup quetions.</li><br /><li> Set the configuration preference "rhq.agent.configuration-setup-flag" to "true". This tells the agent that, when it starts up, it should not ask any setup questions. Instead, it will immediately use what configuration preferences it has and attempt to initialize itself automatically. Of course, setting this to true infers that the rest of your agent's configuration in agent-configuration.xml is complete. But that's what we are going to make sure we do next.</li><br /><li>Make sure that the "rhq.agent.name" configuration preference is left undefined (out-of-box, this setting is commented out in the agent-configuration.xml, make sure you keep it that way). Leaving this undefined will force the agent to attempt to auto-generate its own name. It does this by looking up the agent machine's fully qualified domain name and using that as the agent name. This should ensure that all agents will obtain a unique agent name (since by definition, a fully qualified domain name or IP address is unique within a network).</li><br /><li>Next, determine which Jopr Server your agents will use as their "Registration Server". When new agents start up, they must communicate with a Jopr Server in order to register themselves into the Jopr environment. You must decide which of your Jopr Servers will be used to register newly installed agents (we'll call it the "Registration Server"). There is nothing special or different about a "Registration Server" compared to your other servers in your server cloud, i.e. you won't see any configuration settings or UI controls that turn on or off some "registration feature". Any Jopr Server can register any Jopr Agent. However, you must specify <span style="font-style: italic;">something</span> in your golden distro's agent configuration so the agent knows where a server is so the agent can bootstrap itself into the Jopr environment. Once you determine which of your Jopr Servers will be the one to handle all new agent registrations, set that server's endpoint information in your golden distro's agent-configuration.xml settings:<br /><ul><br /><li>rhq.agent.server.transport</li><br /><li>rhq.agent.server.bind-port</li><br /><li>rhq.agent.server.bind-address</li><br /><li>rhq.agent.server.transport-params</li><br /></ul><br />Note that this will not necessarily be the Jopr Server that will be assigned as the agent's primary server. Once the registration is complete, the agent will be assigned a server failover list, with the first server in the list to be designated as its "primary". This primary server may or may not be the same as the settings you provide here.<br /></li><br /><li>If you wish to assign multiple "Registration Servers" to your agent, you may do so by prepopulating a failover list and putting it in your golden distro. This allows you to have more than one Jopr Server assigned to all of your agents as Registration Servers. If the main registration server is down or is in maintenance mode, the agents will be able to failover to your secondary servers as defined in your failover list. Create a directory "data" in your distribution and place a file called "failover-list.dat" in it. Each line in that file must be of the form "address:port" where address is the IP or hostname of a Jopr Server and port is the port number the server is listening on (each server must require the same transport and transport parameters, so the "rhq.agent.server.transport[-params]" settings will be used for all servers). If you prepackage a failover list in your golden distro, you should place your main Registration Server (the one you configured in the previous step) as the top-most server in the list. Each server thereafter can be listed. If the servers at the top of the list are down, the agent will still be able to register because it just moves down the list until it finds a server it can talk to. Note that this prepopulated failover list is only temporary and is used only the first time the agent starts. Once the agent registers, it will be given a new failover list which will overwrite the list shipped in the golden distro. This is what you want because the server maintains an up-to-date failover list for each agent and you want the agent to refresh its list everytime it regsiters and starts up.</li><br /><li>Make sure "rhq.communications.connector.bind-address" is left undefined (out-of-box, it is commented out in agent-configuration.xml, make sure you keep it that way). Leaving this undefined will tell the agent it needs to lookup its local IP address and use that as its bind address. It does so by using the Java API "InetAddress.getLocalHost().getCanonicalHostName()". Therefore, this uses whatever network adapters are installed on the box and chooses one from the list to determine which IP to use - usually it chooses the first network adapter that the operating system reports. <span style="font-style: italic;">(side note: this may choose an IP from the list of available IPs that is different from the one you actually want to use. You usually have to do some special configuration in your network adapters to get InetAddress.getLocalHost() to return the one you want)</span>.</li><br /><li>Repackage the agent installation in a new jar - this is your "golden distro". Take this distro and push it out to all of your agent machines and they can all start up without any additional configuration or setup needed.</li><br /></ol><br /><br /><span style="font-style: italic;"><a href="http://jira.rhq-project.org/browse/RHQ-496">RHQ-496</a> now allows the agent to determine its name at runtime if one was not specifically given to it at setup time. This code is in trunk, but not in any current Jopr releases. So to get the ability to bundle the "golden distro" and deploy it to multiple agents, you must use a trunk build, until we release our next version.</span><br /><br />After you have deployed your golden distro to machines in your network, you are then left with the question of how do I upgrade my agents? This then leads to the desire to perform automatic upgrades of agents already deployed and running in your environment. This feature is not fully complete, but most of the work is done and exists in trunk (see my earlier blog on this topic). To follow the development of this agent auto-update feature, watch the JIRA <a href="http://jira.rhq-project.org/browse/RHQ-110">RHQ-110</a>. The finished implementation will hopefully look like the design described on the <a href="http://www.rhq-project.org/display/RHQ/Design-AgentAutoUpdate">RHQ wiki</a>.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-500398311961384014?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-52746498454431659472008-12-04T18:55:00.000-08:002009-12-03T06:17:21.615-08:00A new feature has been added to trunk, a feature so interesting that it deserves its own blog.<br /><br />I am sure most security-conscious administrators configure their IT infrastructure in a very specific way and they do not want anyone going onto any machine and re-configuring the machine or any of its software components willy-nilly. In fact, if something <b>is</b> reconfigured outside of a business' normal change-control processes, I would think administrators would want to be notified about it. It could be an innocent user mistakenly modifying something they should not be, or it could be an intruder trying to hack into the system. Being notified of configuration changes sounds like it could be a very useful thing.<br /><br />Jopr now has this feature. If a plugin supports the configuration subsystem (i.e. it can retrieve configuration from its managed resource), the alert subsystem will have the ability to detect changes made in that remote managed resource and send notifications when that happens.<br /><br />I've put together a <a href="http://rhq-project.org/display/JOPR2/Demo-ConfigChangeAlert">demo</a> that shows this feature in action. The scenario is quite simple - I have a Fedora box running sshd, and I do not want that sshd daemon process' configuration to change. If, for whatever reason, the configuration of sshd on the box does change, I want to be notified.<br /><br />And because this config-change-notification feature is built into the core engine, <b>any</b> plugin that supports configuration gets this feature for free. So, if Jopr does not have a plugin that supports a particular resource whose configuration you want to monitor for changes, you can quite simply write your own plugin and deploy it into your Jopr environment and have this capability.<br /><br />I can envision watching the following for configuration changes would be something people find helpful (and some of these you can already do today thanks to existing Jopr plugins):<br /><br /><ul><br /><li>JBossAS's main jboss-service.xml configuration file</li><br /><li>JBossAS's authentication configuration (login-config.xml)</li><br /><li>JBossAS's datasource configuration</li><br /><li>/etc/hosts</li><br /><li>Jopr Agent's own configuration</li><br /><li>...and many more...</li><br /></ul><br /><br />And configuration does not have to be stored in a file on a filesystem. The Jopr configuration subsystem makes no distinction between configuration stored in a file, in a database, an LDAP server or whatever you can think of. It's the plugin's job to translate the resource's configuration into configuration data that conforms to the plugin's metadata. Once the configuration data makes it into the core engine, it is treated the same.<br /><br />And finally, if a configuration change is detected, and that change was unauthorized, the Jopr user has the ability to immediately rollback that change by reverting to an earlier configuration set. This configuration-rollback feature is orthogonal to the change-notification feature, but you can see how both can be used hand-in-hand to keep a tight grip on your IT infrastructure's configuration.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-5274649845443165947?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-47594641290327530342008-11-27T19:11:00.000-08:002009-03-04T18:35:24.670-08:00It started out so innocently - running my J2EE app under high load, I would notice this message repeat many times in the JBossAS log file:<br /><br /><pre><blockquote>[com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa]<br />Could not find new XAResource to use for recovering<br />non-serializable XAResource ...</blockquote></pre><br /><br />The desire to hunt down the meaning of this error message set me off on a long voyage that involved learning about the innards of the JBoss Transaction Manager (JBossTS), stepping through its code, writing custom Transaction Manager objects, and reading through many pages of documentation, forum threads and JIRA issues.<br /><br />What I came out of this huge endeavor is the fact that the JBossTS integration with JBossAS is not very well documented and is not easy to use. What I hope to achieve with this blog is to provide what I feel is lacking in the current JBossAS documentation, and that is a single page where you can find all the information you need in order to assure your application running in JBossAS is fully recoverable in the event of an XA transaction failure. (<span style="font-style: italic;">note: I am using JBossAS 4.2.1, hopefully, later JBossAS versions will make what I am going to discuss easier)</span><br /><br />First, let me say this - if you think your application deployed in JBossAS can recover from transaction failures, I would recommend you read this blog and double check your configuration. Because unless you took <span style="font-weight: bold;">very specific</span> steps to configure your JBossAS deployment to support XA transaction recovery, you will not be able to recover from failed transactions - even if you are using JBossAS with JBossTS integrated and even if you are using XA datasources (i.e. your data sources are defined in your -ds.xml files via &lt;xa-datasource>). The only outward indication that you will get that tells you your app is not able to perform transaction recovery is when you actually get a failure, which causes that above message ("Could not find new XAResource to use for recovering non-serializable XAResource") to appear in your log file.<br /><br />Why is this? Because out-of-box, JBossAS is not fully configured to perform recovery. You have to manually configure JBossAS to fully enable this feature - and it's not as simple as just setting some configuration settings or deploying some additional canned services. It also involves writing/deploying your own custom Transaction Manager code and integrating it into the JBossAS server, because there are problems with some of the current code shipped with JBossAS. This is why I think many people think they can recover from transaction failures, but really cannot - because I'm not convinced many people know enough about this issue to write this custom Transaction Manager code and deploy it properly in JBossAS. This blog will hopefully help put all of this information in a single place and get people "on the road to recovery".<br /><br />I will now walk you through everything that happened in my lengthy investigation of this issue. You can follow my train of thought in the <a href="http://www.jboss.com/index.html?module=bb&amp;op=viewtopic&amp;t=146138">JBossTS forum thread</a> I started that shows you the progression of this investigation.<br /><br /><hr /><br /><span style="font-style: italic;">Before we begin, I must point out that for any of this to work at all, you must deploy your data sources to use XA, i.e. your -ds.xml files need to use &lt;xa-datasource>. To learn how to switch your data sources over to use XA, refer to the <a href="http://www.jboss.org/community/docs/DOC-9328">JBossAS wiki</a>, specifically the part that talks about "Parameters specific for javax.sql.XADataSource usage"</span><br /><br /><span style="font-style: italic;">Special Note To Oracle Users: XA recovery will also not work unless you grant special privileges to your XA datasource's user (i.e. the user whose credentials you define in the xa-datasource definition). If you do not do this, "XAException.XAER_RMERR" errors will occur. The privileges you need to grant include:<br /><pre><blockquote>GRANT SELECT ON sys.dba_pending_transactions TO db_user;<br />GRANT SELECT ON sys.pending_trans$ TO db_user;<br />GRANT SELECT ON sys.dba_2pc_pending TO db_user;<br />GRANT EXECUTE ON sys.dbms_system TO db_user;<br />GRANT SELECT ON v$xatrans$ TO db_user;</blockquote></pre><br />The first four you definitely need. The last one is Oracle version dependent. As documented in <a href="http://www.orafaq.com/wiki/XA_FAQ">http://www.orafaq.com/wiki/XA_FAQ</a>, "for Oracle 7.3 databases one needs to run the XAVIEW.SQL SQL script as user SYS. This script will create the V$XATRANS$ view. Grant select access on it to PUBLIC. This script is located in the $ORACLE_HOME/rdbms/admin directory. Please note, XAVIEW.SQL is not required for XA applications running on Oracle8 and above." Talk to your DBA to see which are required for your specific database installation. I've only tested XA on Oracle10g so I can't speak for other versions.</span><br /><br /><hr /><br />First, of course, was the fact that this all started by me getting those XA recovery failure messages in the JBossAS server log. (And for the curious, I believe the entire reason why I was getting those was because, under heavy load, my application was maxing out its connection pool, which actually went over my processes/sessions limit in Oracle - Oracle promptly rejected the extra connection attempts causing my transactions to fail. Bumping up my Oracle processes/sessions configuration seems to have fixed the cause of most, if not all, of my failures). Anyway, back to the XA recovery error message - at the time, it wasn't very intuitive what that log message was saying, but it sounded bad enough for me to search the 'net for this error message and see what others were saying about it. Alot of people reported seeing this, but not much was said as to how you go about correcting it, and more specifically, how to correct it <span style="font-style: italic;">within the context of the JBossAS application server</span>. I found wiki pages that talk about this error message (such as <a href="http://www.jboss.org/community/docs/DOC-12463">this one</a>), but only from the perspective of the JBossTS standalone product. This goes back to my assertion that the JBossAS needs more documentation on its integration with JBossTS, because the JBossTS documentation only gets you so far (and is why I submitted JIRA <a href="https://jira.jboss.org/jira/browse/JBAS-6244">JBAS-6244</a>). The JBossTS documentation itself seemed comprehensive, my concern was the lack of JBossAS documentation on its integration with JBossTS. For example, the JBossTS wiki page tells me this:<br /><br /><pre><blockquote>You need to provide an instance of a XAResourceRecovery<br />implementation and tie it into the recovery process</blockquote></pre><br /><br />I'm sure this makes perfect sense to the developer familiar with the JBossTS API and to the core JBossAS developers themselves that are integrating JBossTS into the server. But to a J2EE developer - the guy who is simply deploying his J2EE/EJB3 app in JBossAS and who should be free from having to worry about the internals of the app server's transaction manager - this is very confusing and leads to more questions than answers. For example, what is "XAResourceRecovery"? How do I provide one? And how do I tie it into the recovery process? I found no easy answers to those questions in the JBossAS documentation.<br /><br />Searching the JBossTS documentation further, I found <a href="http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Transactions_JTA_Programmers_Guide/index.html#sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Shipped_XAResourceRecovery_implementations">information on the XAResourceRecovery class</a>. It turns out this is a JBossTS API that provides the hook necessary to recover from a transaction failure for a particular resource, like a JDBC data source (that answers the question, "what is XAResourceRecovery?"). JBossTS provides a few of its own implementations out-of-box and because JBossAS ships with the JBossTS product, JBossAS itself comes with these XAResourceRecovery implementations out-of-box as well (and this answers the "how do I provide one?" question - but as you will see shortly, that is not the end of this story). From the JBossTS documentation, I see:<br /><br /><pre><blockquote>Recovery of XA datasources can sometimes<br />be implementation dependant, requiring developers to<br />provide their own XAResourceRecovery instances. However,<br />JBossTS ships with several out-of-the-box implementations<br />that may be useful.</blockquote></pre><br /><br />This wiki page lists two implementations. One is specific to Oracle, but since my app needs to support both Oracle and PostgreSQL (and hopefully more in the future), I want to use the second one: <span style="font-style: italic;">com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery</span>. As the JBossTS documentation states, "this recovery implementation <span style="font-style: italic;">should</span> work on any datasource that is exposed via JNDI." (emphasis is mine, because as I will show you shortly, this class is completely unusable when deployed within JBossAS).<br /><br />OK, so I should be golden now. All I need to do in order to enable XA recovery for any data source deployed in my JBossAS is to set those JDBCXARecovery configuration properties specified in that wiki page and tie that JDBCXARecovery implementation into the recovery process. Now, how do I do this? (this is that third question I had asked myself earlier). Unfortunately, there is no JBossAS documentation that I found that describes how to do this. But I did see another <a href="http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp03/html/Transactions_JTA_Programmers_Guide/sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Recovering_XAConnections.html">JBossTS wiki page that describes how to do this for the standalone JBossTS product</a>, which states:<br /><br /><pre><blockquote>To inform the recovery system about each<br />of the XAResourceRecovery instances, it is necessary to<br />specify their class names through property variables.<br />Any property variable found in the properties file, or<br />registered at runtime, which starts with the name<br />com.arjuna.ats.jta.recovery.XAResourceRecovery will be<br />assumed to represent one of these instances, and its<br />value should be the class name.<br />...<br />Additional information that will be passed to the<br />instance when it is created may be specified after<br />a semicolon.<br />...<br />Note: These properties need to go into the JTA section<br />of the property file.<br /></blockquote></pre><br /><br />OK, now I know how to do this for the standalone JBossTS. But how/where do I do this for the JBossAS integration of JBossTS? I found the file "jbossjta-properties.xml" located in the "&lt;jboss-install-dir>/server/default/conf" directory - this, as it turns out, defines the properties that configure the internals of the JBossTS integrated in the JBossAS server. Based on the instructions on how to configure the JBossTS product that was previously discussed in the JBossTS documentation, I added this in that jbossjta-properties.xml file:<br /><br /><blockquote><pre>&lt;properties depends="arjuna" name="jta"><br />&lt;!-- add this to tie in the recovery object to my JBossTS --><br />&lt;property<br />name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC"<br />value="com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery"/><br />&lt;property name="DatabaseJNDIName" value="java:/MyDS"/><br />&lt;property name="UserName" value="my-db-username"/><br />&lt;property name="Password" value="my-db-password"/><br />...</pre></blockquote><br /><br />That is done exactly how it is documented. However, it doesn't work. I get this at runtime:<br /><br /><pre><blockquote>java.lang.NullPointerException<br />at javax.naming.InitialContext.getURLScheme(InitialContext.java:269)<br />at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:318)<br />at javax.naming.InitialContext.lookup(InitialContext.java:392)<br />at com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery.createDataSource(JDBCXARecovery.java:174)<br />...</blockquote></pre><br /><br />I actually had to grab the JBossTS source code and step through it in a debugger to see what's really happening and why this NPE is thrown. I give the full details as to why this NPE occurs in one of my forum thread posts - <a href="http://www.jboss.com/index.html?module=bb&amp;op=viewtopic&amp;t=146138&amp;postdays=0&amp;postorder=asc&amp;start=10">go here for the technical details</a> - but suffice it to say, in order for those three properties (DatabaseJNDIName, UserName, Password) to be read in by the recovery implementation, I had to provide a parameter to the first property (the parameter value could be anything - I could specify "foo" if I wanted - but this parameter is meant to be the URL to a property file, so I specified the name of the property file itself):<br /><br /><blockquote><pre>&lt;property<br />name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC"<br />value="com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery;jbossjta-properties.xml"/></pre></blockquote><br /><br />OK, that hurdle has been jumped. Start up again and... whoops:<br /><br /><pre><blockquote>java.lang.ClassCastException: org.jboss.resource.adapter.jdbc.WrapperDataSource<br />at com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery.createDataSource(JDBCXARecovery.java:174)</blockquote></pre><br /><br />Back to the debugger and I found that JDBCXAResovery is trying to cast the object found from the JNDI lookup to a XADataSource, but JBossAS does not bind that type of object to JNDI - it binds this WrapperDataSource, which is not a XADataSource. Therefore, this JDBCXARecovery object can <span style="font-style: italic;">never</span> work when deployed in JBossAS. This is one reason why relying on documentation for the standalone JBossTS product is insufficient and why the lack of JBossAS integration docs is really needed. This probably works in some cases, but it most certainly does not work (and will never work) when integrated with JBossAS.<br /><br />Now, it turns out this class-cast problem has already been discussed on <a href="http://www.jboss.com/index.html?module=bb&amp;op=viewtopic&amp;t=126628">a prior forum thread</a> and reported in a JIRA - <a href="https://jira.jboss.org/jira/browse/JBTM-319">JBTM-319</a>. I wish I knew that before I started all of this (did I mention we need JBossAS docs on this? :)<br /><br />Reading that JIRA, it looks like there is a XAResourceRecovery implementation written specifically for deployment inside of JBossAS 4.2 (AppServerJDBCXARecovery) and it was introduced in version 4.2.3.SP8 (I'll assume it made it into that version's distribution). However, I'm using an earlier version of JBossAS, so I had to take the <a href="http://anonsvn.labs.jboss.com/labs/jbosstm/branches/JBOSSTS_4_2_3_GA_SP/atsintegration/classes/com/arjuna/ats/internal/jbossatx/jta/AppServerJDBCXARecovery.java">source code for AppServerJDBCXARecovery.java</a>, compile and bundle its binary in a jar file, and deploy that jar into my JBossAS's "server/default/lib" directory.<br /><br />I found that the Javadocs for that class describe how to configure this. In addition, it looks like Jonathan Halliday very recently added <a href="https://www.jboss.org/community/docs/DOC-10789">some documentation on the JBossTS wiki</a> that discusses this as well - it refers to the Javadoc for the technical details. He does, however, confirm my findings that this class is not in earlier versions of JBossAS - "Note that AppServerJDBCXARecovery is not present in JBossAS (you need to download and build it from source) or early EAP releases"<br /><br />The Javadocs say, in part:<br /><br /><pre><blockquote>To use this class, add an XAResourceRecovery<br />entry in the jta section of jbossjta-properties.xml for<br />each datasource for which you need recovery, ensuring the<br />value ends with ;&lt;datasource-name> i.e. the same value<br />as is in the -ds.xml jndi-name element. You also need the<br />XARecoveryModule enabled and appropriate values for<br />nodeIdentifier and xaRecoveryNode set. See the JBossTS<br />recovery guide if you are unclear on how the recovery<br />system works.</blockquote></pre><br /><br /><span style="font-style: italic;">*sigh*</span> - back to reading about the internals of JBossTS to learn what "appropriate values for nodeIdentifier and xaRecoveryNode" means. I'm sure, again, this makes perfect sense to someone familar with the JBossTS product, but I really find it annoying that a J2EE deployer needs to know all of this just to enable XA recovery. But OK, hopefully this will get easier in the future. Marching forward...<br /><br />The Javadoc instructions tell me to refer to the <a href="http://www.jboss.org/jbosstm/docs/4.2.3/manuals/html/core/FailureRecoveryGuide.html">JBossTS Recovery Guide</a>, and it is in there that I read:<br /><br /><pre><blockquote>A value of * will force JBossTS to recover<br />(and possibly rollback) all transactions irrespective of<br />their node identifier and should be used with caution.<br />The contents of com.arjuna.ats.jta.xaRecoveryNode<br />should be alphanumeric and match the values of<br />com.arjuna.ats.arjuna.xa.nodeIdentifier.</blockquote></pre><br /><br />This leads me back to "jbossjta-properties.xml" and lo-and-behold I do see a "com.arjuna.ats.arjuna.xa.nodeIdentifier" property set here - its value is "1". I didn't look deeply into what this actually identifies, but I assume it identifies this JBossTS instance (but I could be wrong on this).<br /><br />So, following the instructions, I went back to my "jbossjta-properties.xml" and configured it to use this new recoverer instead of the unusable JDBCXARecovery implementation and to use the appropriate value for xaRecoveryNode:<br /><br /><blockquote><pre>&lt;property<br />name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC"<br />value="com.arjuna.ats.internal.jdbc.recovery.AppServerJDBCXARecovery;MyDS"/><br />&lt;!-- xaRecoveryNode should match value in nodeIdentifier or be * --><br />&lt;property name="com.arjuna.ats.jta.xaRecoveryNode" value="1"/><br /></pre></blockquote><br /><br />I'm really getting close now! There is one slight problem, however. When I run my application server for the very first time, my data source is not deployed yet! My application requires the user to run through a "post-installation" UI in order to do things like tell me what database vendor the user is using (Postgres or Oracle), the JDBC URL, database username, password, etc. My application then writes out some deployment information and hot deploys the ds.xml at runtime (a great feature provided to me by JBossAS - hot deployment of data sources is very cool).<br /><br />Anyway, this causes problems because before my user runs this "post-install" step, I have no data source deployed and this recovery object will dump an ugly stack trace to the log because it can't find the data source. The exception is an MBeanException with a root cause of "javax.management.InstanceNotFoundException: jboss.jca:name=MyDS,service=ManagedConnectionFactory is not registered." This is to be expected, looking at the code of AppServerJDBCXARecovery.<br /><br />So what I had to do is modify the AppServerJDBCXARecovery code so it can tolerate the times when the data source is not deployed. (I'll post a follow up with a URL to tell you where you can find this modified code, its not checked into svn yet, but will be soon. It is a pretty simple change <span style="font-style: italic;">[update: the source can now be viewed <a href="http://svn.rhq-project.org/repos/rhq/trunk/modules/enterprise/server/container-lib/src/main/java/org/rhq/jbossatx/jta/recovery/AppServerJDBCXARecovery.java">here</a>]</span>).<br /><br />At this point, I recompiled my custom version of AppServerJDBCXARecovery, bundled it in a .jar and placed that jar in my JBossAS's server/default/lib directory and restarted the server. At this point, no errors occur at startup, and after deploying my data source, I confirmed that the recovery object is able to obtain my XADataSource!<br /><br />And that's it, it is that simple. :-) <span style="font-style: italic;">[update: not so fast, after I wrote this blog, I hit another problem that is documented in JIRA <a href="https://jira.jboss.org/jira/browse/JBTM-441">JBTM-441</a>. This is bad because a very common recovery use case (the database or network crashes) causes recovery to fail until you restart your app server, and this is true for all currently released JBossAS versions, 4.3 and under as of today, 12/6/2008) You must build a patched version of AppServerJDBCXARecovery, attached to that JIRA, and deploy it yourself to work around the problem]</span><br /><br />At this point, I have an application with XA data sources deployed and a transaction manager configured to recover any transactions that fail. I plan on writing some test code in which I can force transaction failures to occur, so I can actually test that the recovery features are fully enabled, but at this point, I have very little doubt that things would work. Once I see that JBossTS is able to get my XADataSource, its just a matter of JBossTS doing what it does best - which includes performing this transaction recovery.<br /><br />PHEW! All of this investigation took alot of time and energy, way too much time for my liking. Hopefully, I can save a few hours (or days :) of someone else's time with this information. It could have turned my several days into about 30 minutes. :}<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4759464129032753034?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com12tag:blogger.com,1999:blog-6944050187568241059.post-62291194960597242632008-11-22T22:16:00.000-08:002009-03-04T18:35:24.671-08:00I recently discovered somewhat odd behavior in the way transactions are timed out in my EJB3/JPA app. It was surprising to me how it worked, and though I guess it makes sense when thinking about it, something here still doesn't "feel right".<br /><br />Consider an EJB3/JPA application deployed in JBossAS 4.2 that uses the Arjuna's Transaction Manager (aka JBossTM - although I still call it Arjuna :) and Hibernate as the JPA implementation.<br /><br />Annotating an EJB3 stateless session bean (SLSB) method, is this:<br /><br />@org.jboss.annotation.ejb.TransactionTimeout(60)<br /><br />If my method takes longer than 60 seconds, the transaction it is running in will timeout, and thus rollback. But, what happens if my method is executing a very long running SQL update? Or, maybe my method is simply writing out a very long data file to a remote file system. Or, maybe I'm just stupid and my method is executing "Thread.sleep(30 * 60 * 1000)"? In other words, what happens if my method takes longer than this transaction timeout?<br /><br />What I was assuming would happen is the thread running my method would get interrupted, and (assuming the JDBC driver or file IO subsystem or Thread.sleep or whatever my method was doing at the time can handle the interrupt) the method would abort with that interrupted exception and immediately begin the rollback procedures. It would do this because once the method exits, the EJB3 interceptor chain would begin to unroll and eventually hit the transaction manager interceptors whose job it is to abort the transaction.<br /><br />That is close to what happens, but not exactly.<br /><br />It turns out, the transaction manager detects that the transaction has timed out and aborts it from within an asynchronous thread (different from the thread running my method), but it never sends an interrupt to my method. So, my method continues on even after the timeout period and only when it returns will that interceptor chain finally get a chance to abort the transaction. But by this time, its already been aborted! The interceptor kindly tells me this in a log message, but it's late to the party - the transaction has already been rolled back. This doesn't actually cause any harm because the transaction manager simply says, "this has already been aborted and the rollback was performed earlier, I'll just log a warning and skip the abort procedure".<br /><br />But, I question why my method was not given a chance to abort also. It is possible that my method could run for 60 minutes, do everything completely successfully, yet, because it exceeded the transaction timeout the entire transaction was rolled back and that 60 minutes worth of work now becomes wasted.<br /><br />Of course, the answer to that would be, "just set your transaction timeout to a higher value". But that's not really my point. My point is, why is my thread even allowed to waste its time when the transaction manager knows the timeout has expired and the transaction has been rolled back? I think it should at least attempt to warn the thread about the situation by sending an interrupt to it (at that point, it would be my method's job to handle the thrown InterruptedException).<br /><br />I've tested this to see if an interrupt is sent and I don't see it. See below for the log messages I see after running some test code. I had an SLSB method annotated with a transaction timeout of 2 seconds, and in my SLSB method, I block within a "Thread.sleep(10000)" call (so it pauses in my method for 10 seconds). Take note of the log timestamps and notice that my sleep is allowed to return normally after 10 seconds, but after the 2nd second, you see the transaction manager aborted my transaction! Therefore, my method was allowed to continue until it returned normally - at which time, the interceptor chain tried to abort the transaction a second time:<br /><br /><blockquote><pre><br />2008-11-23 01:09:33 INFO [STDOUT] !!!!!!!!!BEFORE SLEEP<br />2008-11-23 01:09:35 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N]<br />[com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action<br />id a0b0c21:ab6:4928f32d:1225 invoked while multiple threads active within it.<br />2008-11-23 01:09:35 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N]<br />[com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check<br />- atomic action a0b0c21:ab6:4928f32d:1225 aborting with 1 threads active!<br />2008-11-23 01:09:43 INFO [STDOUT] !!!!!!!!!AFTER SLEEP<br />2008-11-23 01:09:43 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N]<br />[com.arjuna.ats.arjuna.coordinator.BasicAction_40] - Abort called on<br />already aborted atomic action a0b0c21:ab6:4928f32d:1225<br />2008-11-23 01:09:43 ERROR [test.slsb] Failed. Cause:<br />java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive]<br />[com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] The transaction is not active!<br />java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive]<br />[com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] The transaction is not active!<br />at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1372)<br />at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)<br />at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)<br />at org.jboss.aspects.tx.TxPolicy.endTransaction(TxPolicy.java:175)<br />at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:87)<br />at org.jboss.aspects.tx.TxInterceptor$RequiresNew.invoke(TxInterceptor.java:262)<br />at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br />at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)<br />at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br />at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)<br />at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br />at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)<br />at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)<br />at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br />at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)<br />at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br />at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)<br />at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)<br />at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)<br />at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)<br />at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)<br /> ...<br /></pre></blockquote><br /><br />It is also interesting to note that the transaction manager knows that a thread is still running my method - see its log message: "aborting with 1 threads active".<br /><br />This is worrisome to me because I could run into a condition where my SLSB methods might waste a whole lot of time processing a query or doing other work well after my timeout has triggered the rollback (unless I set my timeouts to some really high value that I know will never expire but then, what's the point of the timeout?). I would rather have my method be interrupted so I don't waste the system resources doing work that is just going to be (or has already been) rolled back. But, for now, without any other options that I know of, I am just going to have to make sure I set my timeouts to some very large values for those methods that may (but most times may not) take a really long time to complete.<br /><br />I wonder if there is some configuration property I can set on the transaction manager to tell it, "interrupt any active thread that may be running within a transaction that has timed out". I don't know of any such thing, but it just seems like this would be an obvious thing to want to do, so I would not be surprised at all if it turns out that I am just missing a piece of the puzzle. Feel free to comment on this blog if you know of a way to work around this issue or to let me know if there is something I am misunderstanding with respect to the way the transaction manager works (or can be configured to work).<br /><br />As a side note, I was hoping I could prevent doing the extra, wasted work in the database by circumventing the JPA entity manager and using JDBC statements directly, thereby having access to the java.sql.Statement.setQueryTimeout API but, alas, the Postgres driver does not yet implement that.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-6229119496059724263?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com14tag:blogger.com,1999:blog-6944050187568241059.post-7433178032633058622008-11-21T08:37:00.000-08:002009-03-04T18:35:24.671-08:00The <a href="http://www.mobicents.org/">Mobicents Platform</a> has recently announced their integration with Jopr!<br /><br />Jean has some really nice things to say:<br /><br /><a href="http://jeanderuelle.blogspot.com/2008/11/mobicents-sip-servlets-gets-shinny-new.html">http://jeanderuelle.blogspot.com/2008/11/mobicents-sip-servlets-gets-shinny-new.html</a><br /><br />This validates what we are doing - which is developing an extensible management platform that anyone who has a software product that they want to manage/monitor can integrate with (and do so quickly and easily).<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-743317803263305862?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-90382918822747705972008-11-10T22:19:00.000-08:002010-06-17T05:23:30.267-07:00My previous post explains how you can use the Hibernate plugin to manage your Hibernate applications with Jopr. In reality, the Hibernate plugin is just a customized JMX plugin because, after all, everything you do in relation to monitoring Hibernate is through its Hibernate Statistics MBean.<br /><br />So this begs the question, what if my own application has its own custom JMX MBeans? Can I use Jopr to manage and monitor those as well?<br /><br />The answer is a most emphatic yes!.<br /><br />Out-of-box, you already get the generic JMX plugin. This was designed to be extensible by other plugins. In fact, its where Jopr's JBossAS plugin, Tomcat plugin and Hibernate plugin all get alot of their functionality from! It stands to reason that you can do the exact same thing for your own MBeans as the Jopr development team does for JBossAS, Tomcat and Hibernate. There is even already an example custom JMX plugin Maven module that you can use as a starting point, should you want to write your own custom JMX plugin - see <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=etc/samples/custom-jmx-plugin;hb=refs/heads/master">the actual code here</a> which you can copy and customize for your needs. BTW: when I say "custom JMX plugin" - I mean a plugin that can manage your custom JMX MBeans. I do not mean to say this is a customized version of the out-of-box JMX plugin. In other words, the custom JMX plugin is not a replacement for the existing JMX plugin, its merely an extension to it.<br /><br /><a href="http://www.rhq-project.org/display/RHQ/Plugins+-+Demos+-+Writing+Custom+JMX+Plugin">Watch the demo I made</a> to see how you can create your own custom JMX plugin for deployment in any RHQ or Jopr Server (btw: RHQ is where the entire plugin framework lives; because it is upstream to Jopr, your RHQ plugins are completely 100% compatible with Jopr as well - I'll continue to use the Jopr name, but suffice it to say, whenever I say "Jopr", you can assume the same holds true for RHQ). The <a href="http://www.rhq-project.org/download/attachments/1015985/custom-jmx-plugin-demo-code.zip?version=1">source and binaries used by the demo</a> are available for download, in case you want to try it yourself after watching the demo.<br /><br />The generic JMX plugin is very flexible in the way you can extend it. That said, there is still plenty more we can probably do to enhance it. Allowing it to generically handle more data types for operation parameters and results and attribute values are just some of the low-hanging fruit in here - if only we had 25 hours in a day. If anyone is interested in getting started contributing to the RHQ project, that would probably be a very good place to start - its easy to understand and the places that need to be enhanced are localized to only one or maybe a couple Java classes. If you are willing to try this out, feel free to ask about this in our freenode chat room at #rhq or send a message to one of our forums.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-9038291882274770597?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com4tag:blogger.com,1999:blog-6944050187568241059.post-47468718564765701052008-11-09T23:29:00.000-08:002010-06-17T05:24:35.450-07:00I have heard several people ask questions regarding how they can manage and monitor Hibernate from Jopr. Some have even asked how they can do this even if their Hibernate app is not running inside a JBossAS server instance (for example, within its own, standalone J2SE virtual machine).<br /><br />Jopr today has the capability to manage/monitor Hibernate if its running inside Tomcat or JBossAS via its <a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/plugins/hibernate;hb=master">Hibernate plugin</a>. It can even monitor more than one Hibernate application/JVM that is running on your machine, even if they are using different versions of Hibernate!<br /><br />Today, I checked in code to the RHQ core and Jopr Hibernate plugin to have it also support Hibernate that is running in a standalone JVM (in fact, it can now support Hibernate running in <span style="font-weight: bold;">any</span> JVM, so long as it can remotely connect to the Hibernate Statistic MBean's MBeanServer).<br /><br />This is such a cool feature, that I decided to "wink" it. Watch <a href="http://www.rhq-project.org/display/RHQ/Plugins+-+Demos+-+Monitoring+Hibernate">the flash demo</a> to see how you can examine your Hibernate statistics in the Jopr GUI - things such as which queries were executed, how many times they were executed and how long it took to execute them; how many and which entities were created and deleted, etc. This is a very helpful set of features for developers - I can attest to that because I use this to examine the RHQ Server's own Hibernate usage.<br /><br />There are a couple caveats:<br /><br />First, the JVM that Hibernate is running in must have JMX remoting enabled and configured to accept connections from the agent. Google "com.sun.management.jmxremote" and read all about the settings used to configure this. The demo used something like this:<br /><br /><pre class="code-java" style="font-family:courier new;"><span style="font-size:100%;"><br />java -Dcom.sun.management.jmxremote.port=19988 \<br />-Dcom.sun.management.jmxremote.ssl=false \<br />-Dcom.sun.management.jmxremote.authenticate=false \<br />-jar helloworld.jar<br /></span></pre><br /><br />Second, your application must have enabled the Hibernate Statistics MBean. This sounds obvious, but I guess I should explicitly mention it. If you don't tell Hibernate to turn on its statistics, you can't very well get any useful data from it. To do this, your application will have to execute something like this:<br /><br /><pre class="code-java" style="font-family:courier new;"><span style="font-size:100%;"><a href="http://www.hibernate.org/hib_docs/v3/api/org/hibernate/jmx/StatisticsService.html">StatisticsService</a> mBean = <span class="code-keyword">new</span> StatisticsService();<br /><a href="http://www.hibernate.org/hib_docs/v3/api/org/hibernate/SessionFactory.html">SessionFactory</a> sessionFactory = ...get hibernate session factory...<br />mBean.setSessionFactory(sessionFactory);<br />ObjectName objectName = <span class="code-keyword">new</span> ObjectName("Hibernate:application=MY_APP_NAME,type=statistics"<em></em>);<br />MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();<br />mbs.registerMBean(mBean, objectName);<br />sessionFactory.getStatistics().setStatisticsEnabled(<span class="code-keyword">true</span>);<br /></span></pre><br />Once you place your statistics MBean in an MBeanServer that can be remotely accessed, your Jopr Hibernate plugin will do the rest.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4746871856476570105?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com4tag:blogger.com,1999:blog-6944050187568241059.post-31005067391451536222008-11-03T09:55:00.000-08:002009-12-03T06:12:53.328-08:00Now that we have introduced HA capabilities to RHQ, we expect to be able to support at least a few hundred agents within an RHQ ecosystem (if not more).<br /><br />But this poses a new challenge for us and our customers - how do we keep all of those agents up-to-date? Once you get an RHQ environment set up and running, what happens when we release our next version of RHQ? We want to avoid having to remotely log onto each and every agent machine to manually update their old agents.<br /><br />I've recently embarked on the first implementation of an "agent auto-update" feature, to be included in the next release of RHQ.<br /><br />Soon, RHQ will no longer ship a separate agent distribution. We will now ship just a "RHQ distribution" that includes both a server and agent (this will be true of Jopr and JBoss ON as well). This will ensure that when you get a distribution binary, you will receive the server and agent that are compatible with each other.<br /><br />The agent distribution is then directly downloadable straight from the server (e.g. <span style="font-family:courier new;">http://&lt;server><server></server></span><rhq-server><span style="font-family:courier new;">:7080/agentupdate/download</span>). This lets you grab an agent distribution when you want to install an agent manually.<br /><br />However, what happens if you already installed the agent? Soon, what will happen, is the agent will be able to automatically download an updated agent distribution from the server and install the update itself; all without user intervention or manual steps needing to be taken.<br /><br />To follow the progress of this feature or are interested in the technical details, watch the JIRA and feel free to join in the discussion on that developer forum thread.<br /><br /><a href="http://jira.rhq-project.org/browse/RHQ-110">http://jira.rhq-project.org/browse/RHQ-110</a><br /><a href="http://www.rhq-project.org/display/RHQ/Design-AgentAutoUpdate">http://www.rhq-project.org/display/RHQ/Design-AgentAutoUpdate</a><br /><br /><a href="http://forums.rhq-project.org/viewtopic.php?f=5&amp;t=91"></a></rhq-server><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-3100506739145153622?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-45910239897859093072008-11-03T09:42:00.000-08:002009-12-03T06:11:51.356-08:00Due to enormous peer pressure from my current team members, I have embarked on the creation of my first blog. :)<br /><br />I am currently involved in the design and development of the <a href="http://www.rhq-project.org/display/RHQ/Home">RHQ management platform</a> and the <a href="http://www.jboss.org/jopr/">Jopr Middleware Management project</a>. The blogs that I will be writing in the future will provide some additional insight regarding the RHQ and Jopr projects.<br /><br />If your company has a need for a JBoss Middleware Management product, may I suggest you look into <a href="https://docs.jbosson.redhat.com">JBoss Operations Network</a> - which is Red Hat's offering that, at its core, is Jopr/RHQ.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/6944050187568241059-4591023989785909307?l=management-platform.blogspot.com' alt='' /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0