I recently completed the initial implementation of a feature that folks have been asking for lately, so I decided to put together a quick demo to show it.
First, some background. The initial RHQ Bundle Provisioning feature (of which I have blogged about in the past) supported the ability to deploy bundle distributions (which are, essentially, just files of generic content) to a group of platforms. That is, given a set of files bundled together, RHQ can ship out those files to a group of platform resources where the agents on those platforms will unbundle the files to the root file system.
This new feature (submitted as enhancement request BZ 644328) now allows you to target your bundle deployments to a group of non-platform resources if you wish. For example (as the demo shows), we can now target bundle deployments to JBossAS Server resources. This means you can bundle up a WAR or an EAR, and push that bundle to a group of JBoss AS Servers such that the WAR or EAR gets deployed directly to the deploy/ directory (and hence gets deployed into your application servers).
The demo is 10 minutes long and shows what a simple workflow looks like to deploy a WAR bundle to a group of JBoss EAP application servers.
View the demo here.
[Note: unlike my previous demos which were built on a Windows laptop using Wink and formatted as Flash, I decided to try Istanbul on my Fedora desktop. So the demo is formatted in .ogg format, as opposed to Flash. Hopefully, this doesn't limit the audience that is able to view the demo.]
Monday, June 20, 2011
Monday, May 23, 2011
Detaching Hibernate Objects to pass to GWT
One thing we encountered pretty quickly when we started our GWT work was the fact that you can't serialize objects over the wire from server to GWT client if those objects were obtained via a Hibernate/JPA entity manager.
If you've ever worked with Hibernate/JPA, you'll know that when you get back entity POJOs whose fields are not loaded (i.e. marked for lazy loading and you didn't ask for the data to be loaded), your entity POJO instance will have Hibernate proxies where you would expect a "null" object to be (this is to allow you to load the data later, if your object is still attached to the entity manager session).
Having these proxies even after leaving a JPA entity manager session is a problem in the GWT world because the GWT client sitting in your browser doesn't have Hibernate classes available to it! Trying to send these entity POJO instances that have references to Hibernate proxies causes serialization errors and your GWT client will fail to operate properly.
This is a known issue and is discussed here.
We pretty quickly decided against using DTOs. As that page above mentioned, "if you have many Hibernate objects that need to be translated, the DTO / copy method creation process can be quite a hassle". We have a lot of domain objects that are used server side in RHQ. There was no reason why we shouldn't be able to reuse our domain objects both server side and client side - introducing DTOs just so we could workaround this serialization issue seemed ill-advised. It would have just added bloat and unnecessary complexity.
I can't remember how mature the Gilead project was at the time we started our GWT work, or maybe we just didn't realize it existed. Gilead does require you to have your domain objects and server side impl classes extend certain Java classes (LightEntity for example), so it has a slight downside that it requires you to modify all your domain objects. In any event, we do not use Gilead to do this detaching of hibernate proxies.
RHQ's solution was to write our own "Hibernate Detach Utility". This is a single static utility that you use to process your objects just prior to sending them over the wire to your GWT client. Essentially it scrubs your object of all Hibernate proxies, cleaning it such that it can be serialized over the wire successfully.
We also used this when we originally developed a web services interface to the RHQ remote API.
Here is the HibernateDetachUtility source code in case you are interested in seeing how we do it - maybe you could use this in your own GWT/Hibernate application. I think it is reuseable - not much custom RHQ stuff is going on in here.
If you've ever worked with Hibernate/JPA, you'll know that when you get back entity POJOs whose fields are not loaded (i.e. marked for lazy loading and you didn't ask for the data to be loaded), your entity POJO instance will have Hibernate proxies where you would expect a "null" object to be (this is to allow you to load the data later, if your object is still attached to the entity manager session).
Having these proxies even after leaving a JPA entity manager session is a problem in the GWT world because the GWT client sitting in your browser doesn't have Hibernate classes available to it! Trying to send these entity POJO instances that have references to Hibernate proxies causes serialization errors and your GWT client will fail to operate properly.
This is a known issue and is discussed here.
We pretty quickly decided against using DTOs. As that page above mentioned, "if you have many Hibernate objects that need to be translated, the DTO / copy method creation process can be quite a hassle". We have a lot of domain objects that are used server side in RHQ. There was no reason why we shouldn't be able to reuse our domain objects both server side and client side - introducing DTOs just so we could workaround this serialization issue seemed ill-advised. It would have just added bloat and unnecessary complexity.
I can't remember how mature the Gilead project was at the time we started our GWT work, or maybe we just didn't realize it existed. Gilead does require you to have your domain objects and server side impl classes extend certain Java classes (LightEntity for example), so it has a slight downside that it requires you to modify all your domain objects. In any event, we do not use Gilead to do this detaching of hibernate proxies.
RHQ's solution was to write our own "Hibernate Detach Utility". This is a single static utility that you use to process your objects just prior to sending them over the wire to your GWT client. Essentially it scrubs your object of all Hibernate proxies, cleaning it such that it can be serialized over the wire successfully.
We also used this when we originally developed a web services interface to the RHQ remote API.
Here is the HibernateDetachUtility source code in case you are interested in seeing how we do it - maybe you could use this in your own GWT/Hibernate application. I think it is reuseable - not much custom RHQ stuff is going on in here.
RHQ 4 Has Been Released
It has been a very long road for this one, but we managed to release RHQ 4.
We managed to standardize on GWT as our user interface framework. Here's an example of what the new GWT based UI looks like:
There are still a few JSP pages around, but for the most part, the RHQ GUI is now a GWT application with SmartGWT components. One of the fun parts of this job is to learn new and exciting technologies (though I guess "new" is relative - but I still consider GWT a "new" GUI technology, compared to things like Struts and JSP).
Take a look, give it a test drive and see what you think.
We managed to standardize on GWT as our user interface framework. Here's an example of what the new GWT based UI looks like:
There are still a few JSP pages around, but for the most part, the RHQ GUI is now a GWT application with SmartGWT components. One of the fun parts of this job is to learn new and exciting technologies (though I guess "new" is relative - but I still consider GWT a "new" GUI technology, compared to things like Struts and JSP).
Take a look, give it a test drive and see what you think.
Saturday, February 5, 2011
Alerting and Remote Script Execution
RHQ has the ability to invoke operations on any resource when it triggers alerts. When you combine this with the ability of the Script plugin to run remote scripts via its "Execute" operation, you have a very powerful mechanism to integrate your own processes and rules to help correct or workaround abnormal conditions that occur in your managed environment.
Because I've heard several people ask if RHQ has this ability, I put together a flash demo that shows how you do it. The demo shows how you can execute any script on a remote machine when an alert is triggered by RHQ. For example, if you set up an alert that detects when your app server is using an abnormally large number of threads (a possible indication of heavy load), you can have RHQ execute a custom script on your app server machine to help alleviate problems that might occur due to that condition (such a script could be one that reconfigures a load balancer to help redirect load away from your app server).
The use-cases for this feature are virtually endless. Any set of alert conditions on any managed resource can trigger the execution of any script you have. And as the demo illustrates, it is really easy to set this up within RHQ.
Because I've heard several people ask if RHQ has this ability, I put together a flash demo that shows how you do it. The demo shows how you can execute any script on a remote machine when an alert is triggered by RHQ. For example, if you set up an alert that detects when your app server is using an abnormally large number of threads (a possible indication of heavy load), you can have RHQ execute a custom script on your app server machine to help alleviate problems that might occur due to that condition (such a script could be one that reconfigures a load balancer to help redirect load away from your app server).
The use-cases for this feature are virtually endless. Any set of alert conditions on any managed resource can trigger the execution of any script you have. And as the demo illustrates, it is really easy to set this up within RHQ.
Wednesday, January 26, 2011
Bundle Provisioning Via RHQ
A certain amount of enhancements and UI clean up were added to the RHQ Bundle/Provisioning feature. So, I figure now would be a good time to re-introduce it in a new blog entry. I also put together a flash demo if you would like to see the UI in action.
Let me recap what this RHQ Bundle/Provisioning feature is all about. RHQ allows you to bundle up a set of files and push them out to remote machines. You can install and upgrade these sets of files as well as revert back to a previous version of the files or purge the bundle files completely. You sometimes see this mentioned as the "Provisioning" feature, and other times you will see it referred to as the "Bundle" subsystem. (I prefer the term "Bundle" since that is the term that the RHQ user interface uses).
There are a few concepts you must know in order to understand how RHQ does its thing. This is covered on the wiki, but I'll try to explain it briefly here, too.
First is the concept of "bundle". A "bundle" is a logical concept and basically refers to an application ("Pet Store Application" or "My Wiki Server"). A bundle has one or more "versions". A "bundle version" refers to an actual set of files that you want to push out to a set of remote machines. Think of it as your application distribution. Each bundle version has its own "recipe" which tells RHQ what files exist in the bundle, and how to configure and provision those files to the remote machines. Developers or application packagers are responsible for writing the recipe and 'bundling up' the application's files (hence the name 'bundle') with the recipe into an RHQ "bundle version".
What you do with a bundle version brings us to the next set of concepts. A bundle "destination" is associated with a specific bundle and is simply a place (or places) where you want to deploy your bundle. A destination specifies two things - a group (which contains one or more remote machines) and a destination directory (which specifies where on the remote machines' filesystems the bundle files should go). Once you have a "bundle destination" in place, you can begin to deploy one or more of its bundle's versions to that destination. A bundle "deployment" represents one deployment of a bundle version to a destination.
It may make more sense if I give an example. Suppose I have a web application (call it "My Application") that runs inside a JBoss application server. This is my "bundle". I actually have two versions of my application, 1.0 and 2.0. These are my "bundle versions".
Now suppose that I have a QA environment that consists of two machines - a Windows machine and a Linux machine. I want to test my application on my two QA machines. So I need to install my application on both of them. I want to install my application on each machine's "/home/mazz/opt/myapp" directory. This group of two QA machines, along with the destination directory, is a "bundle destination" for my application bundle (call it the "QA destination"). I also have a group of three Linux machines that make up my production environment. After my application passes all tests, I want to deploy my application to that production environment in the "/opt" directory. That group of three production machines, along with the "/opt" directory specification, is another bundle destination associated with my application bundle - call it the "production destination".
Once I tell RHQ to deploy the "1.0" bundle version of my application to my "QA destination", I will have a "bundle deployment". This bundle deployment will be considered the "live" deployment because its the last one I pushed out. I can then test that version while it is on my QA machines. Suppose I find that I want to upgrade my QA environment with the newer "2.0" version of my application. I simply deploy that bundle version to the "QA destination" and now I have a second "bundle deployment". This second deployment is now considered live. If I find that I do not like this newer "2.0" version of my application, I can ask RHQ to revert back to the last live deployment (which was my "1.0" bundle version) - this revert becomes yet another "bundle deployment" (the third) but it reverts back to the "1.0" bundle version content. Once I pass all of my QA tests, I can then deploy whatever bundle version I deem appropriate to my "production destination".
Most of what I describe above is actually demonstrated in my flash demo. The only thing I do not show in the demo is the use of a second "production destination", but it is the same effort to deploy to a second destination as it is to deploy to the first destination.
One new feature that has been introduced to RHQ is the ability to "purge" a destination of all bundle content. If, for example, you want to remove all bundle files completely from the QA destination, you can ask RHQ to purge that destination. What RHQ will do is remove all bundle content from the remote machines that were associated with that destination.
Another new feature that has been added is the ability for RHQ to deploy a bundle into an already existing deployment directory that may have other non-managed content that should be left alone. Such would be the case if you want to deploy an EAR or WAR to a JBossAS deploy/ directory (which obviously has other files inside of it). This deserves some additional explanation.
Typically, you will want to deploy a set of application files into its own directory on some file system. For example, if you have a JBoss application server, you want to install it in something like "/opt/my-jboss". All of your application server files are in that directory, but no other files are in there. If you want to remove your JBoss application server installation, it is as simple as "rm -rf /opt/my-jboss".
However, what if you deployed a bundle version in that directory already, but you then upgrade that bundle deployment with a new bundle version? In this case, you will already have files in /opt/my-jboss (the original bundle version content). RHQ will actually overwrite, backup or ignore conflicting files that it finds following strict upgrade rules. If, for example, RHQ finds files in that /opt/my-jboss directory that don't belong to the new bundle version, they will be removed. RHQ calls this "managing the root directory".
This is usually what you want if you are deploying a standalone software product. If there are any unknown files in the deployment directory, RHQ has to remove them to make sure the bundle deployment directory is exactly in the state the new bundle version recipe wants it to be. However, this is not what you want if you desire to deploy an EAR or WAR to an already existing JBossAS's deploy/ directory. That's because we already know there will be unrelated files in this deploy/ directory that must remain intact and in place. RHQ must leave any files it finds in that destination directory alone - even though they aren't part of our bundle deployment. In other words, we do not want RHQ to manage this root deployment directory.
RHQ now supports this by allowing the Ant recipe author to specify the manageRootDir="false" attribute in the rhq:deployment-unit task. This new feature is documented in Bugzilla 659142 and this new attribute is documented on the RHQ wiki.
You can read more about the Provisioning/Bundle feature on the RHQ wiki.
Let me recap what this RHQ Bundle/Provisioning feature is all about. RHQ allows you to bundle up a set of files and push them out to remote machines. You can install and upgrade these sets of files as well as revert back to a previous version of the files or purge the bundle files completely. You sometimes see this mentioned as the "Provisioning" feature, and other times you will see it referred to as the "Bundle" subsystem. (I prefer the term "Bundle" since that is the term that the RHQ user interface uses).
There are a few concepts you must know in order to understand how RHQ does its thing. This is covered on the wiki, but I'll try to explain it briefly here, too.
First is the concept of "bundle". A "bundle" is a logical concept and basically refers to an application ("Pet Store Application" or "My Wiki Server"). A bundle has one or more "versions". A "bundle version" refers to an actual set of files that you want to push out to a set of remote machines. Think of it as your application distribution. Each bundle version has its own "recipe" which tells RHQ what files exist in the bundle, and how to configure and provision those files to the remote machines. Developers or application packagers are responsible for writing the recipe and 'bundling up' the application's files (hence the name 'bundle') with the recipe into an RHQ "bundle version".
What you do with a bundle version brings us to the next set of concepts. A bundle "destination" is associated with a specific bundle and is simply a place (or places) where you want to deploy your bundle. A destination specifies two things - a group (which contains one or more remote machines) and a destination directory (which specifies where on the remote machines' filesystems the bundle files should go). Once you have a "bundle destination" in place, you can begin to deploy one or more of its bundle's versions to that destination. A bundle "deployment" represents one deployment of a bundle version to a destination.
It may make more sense if I give an example. Suppose I have a web application (call it "My Application") that runs inside a JBoss application server. This is my "bundle". I actually have two versions of my application, 1.0 and 2.0. These are my "bundle versions".
Now suppose that I have a QA environment that consists of two machines - a Windows machine and a Linux machine. I want to test my application on my two QA machines. So I need to install my application on both of them. I want to install my application on each machine's "/home/mazz/opt/myapp" directory. This group of two QA machines, along with the destination directory, is a "bundle destination" for my application bundle (call it the "QA destination"). I also have a group of three Linux machines that make up my production environment. After my application passes all tests, I want to deploy my application to that production environment in the "/opt" directory. That group of three production machines, along with the "/opt" directory specification, is another bundle destination associated with my application bundle - call it the "production destination".
Once I tell RHQ to deploy the "1.0" bundle version of my application to my "QA destination", I will have a "bundle deployment". This bundle deployment will be considered the "live" deployment because its the last one I pushed out. I can then test that version while it is on my QA machines. Suppose I find that I want to upgrade my QA environment with the newer "2.0" version of my application. I simply deploy that bundle version to the "QA destination" and now I have a second "bundle deployment". This second deployment is now considered live. If I find that I do not like this newer "2.0" version of my application, I can ask RHQ to revert back to the last live deployment (which was my "1.0" bundle version) - this revert becomes yet another "bundle deployment" (the third) but it reverts back to the "1.0" bundle version content. Once I pass all of my QA tests, I can then deploy whatever bundle version I deem appropriate to my "production destination".
Most of what I describe above is actually demonstrated in my flash demo. The only thing I do not show in the demo is the use of a second "production destination", but it is the same effort to deploy to a second destination as it is to deploy to the first destination.
One new feature that has been introduced to RHQ is the ability to "purge" a destination of all bundle content. If, for example, you want to remove all bundle files completely from the QA destination, you can ask RHQ to purge that destination. What RHQ will do is remove all bundle content from the remote machines that were associated with that destination.
Another new feature that has been added is the ability for RHQ to deploy a bundle into an already existing deployment directory that may have other non-managed content that should be left alone. Such would be the case if you want to deploy an EAR or WAR to a JBossAS deploy/ directory (which obviously has other files inside of it). This deserves some additional explanation.
Typically, you will want to deploy a set of application files into its own directory on some file system. For example, if you have a JBoss application server, you want to install it in something like "/opt/my-jboss". All of your application server files are in that directory, but no other files are in there. If you want to remove your JBoss application server installation, it is as simple as "rm -rf /opt/my-jboss".
However, what if you deployed a bundle version in that directory already, but you then upgrade that bundle deployment with a new bundle version? In this case, you will already have files in /opt/my-jboss (the original bundle version content). RHQ will actually overwrite, backup or ignore conflicting files that it finds following strict upgrade rules. If, for example, RHQ finds files in that /opt/my-jboss directory that don't belong to the new bundle version, they will be removed. RHQ calls this "managing the root directory".
This is usually what you want if you are deploying a standalone software product. If there are any unknown files in the deployment directory, RHQ has to remove them to make sure the bundle deployment directory is exactly in the state the new bundle version recipe wants it to be. However, this is not what you want if you desire to deploy an EAR or WAR to an already existing JBossAS's deploy/ directory. That's because we already know there will be unrelated files in this deploy/ directory that must remain intact and in place. RHQ must leave any files it finds in that destination directory alone - even though they aren't part of our bundle deployment. In other words, we do not want RHQ to manage this root deployment directory.
RHQ now supports this by allowing the Ant recipe author to specify the manageRootDir="false" attribute in the rhq:deployment-unit task. This new feature is documented in Bugzilla 659142 and this new attribute is documented on the RHQ wiki.
You can read more about the Provisioning/Bundle feature on the RHQ wiki.
Tuesday, October 26, 2010
Using Byteman To Analyze the RHQ Agent
I recently had a need to run some quick and dirty performance testing over a few method calls in one of the RHQ plugins that was deployed in my agent. I didn't have a profiler installed and ready to go so I used Byteman. I quickly wrote up a couple of Byteman rules (thanks to Jay S. for correcting some problems I had with those rules), ran the RHQ Agent with Byteman installed and got some really good data.
It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.
Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.
I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.
I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).
Read the wiki page on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.
It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.
Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.
I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.
I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).
Read the wiki page on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.
Thursday, August 5, 2010
Remotely Installing An Agent
One new feature coming soon to RHQ is the ability to remotely install an RHQ Agent on to any machine in your network (with the caveat that the remote machine must be accessible via SSH).
What this now means is you do not have to manually log onto a remote machine and do the tasks of downloading the agent update binary distribution, installing it and running it. RHQ can now do this for you all remotely - all you need to provide to the RHQ GUI is the machine name, your SSH credentials and the location where you want to install the agent.
I think this could still use some enhancements - there is no way to customize each agent's configuration (i.e. providing custom answers to the startup setup questions). But that's for another day. If you want to customize the new agent, you can do so by just importing the RHQ Agent resource into inventory and going to its Configuration tab and adjusting its configuration (don't forget to restart the agent so it can pick up the changed configuration- you can restart the agent using this new RHQ GUI page! I'll talk about that next).
This mechanism will also allow you to stop and start an existing agent that is already installed. This is very useful if you have not started the agent yet but wish to bring it back online, or for those cases where you have not commited the RHQ Agent resource in inventory yet but still need a way to shutdown or restart an agent.
I put together a flash demo that shows this stuff in action.
What this now means is you do not have to manually log onto a remote machine and do the tasks of downloading the agent update binary distribution, installing it and running it. RHQ can now do this for you all remotely - all you need to provide to the RHQ GUI is the machine name, your SSH credentials and the location where you want to install the agent.
I think this could still use some enhancements - there is no way to customize each agent's configuration (i.e. providing custom answers to the startup setup questions). But that's for another day. If you want to customize the new agent, you can do so by just importing the RHQ Agent resource into inventory and going to its Configuration tab and adjusting its configuration (don't forget to restart the agent so it can pick up the changed configuration- you can restart the agent using this new RHQ GUI page! I'll talk about that next).
This mechanism will also allow you to stop and start an existing agent that is already installed. This is very useful if you have not started the agent yet but wish to bring it back online, or for those cases where you have not commited the RHQ Agent resource in inventory yet but still need a way to shutdown or restart an agent.
I put together a flash demo that shows this stuff in action.
Subscribe to:
Posts (Atom)