RHQ has the ability to invoke operations on any resource when it triggers alerts. When you combine this with the ability of the Script plugin to run remote scripts via its "Execute" operation, you have a very powerful mechanism to integrate your own processes and rules to help correct or workaround abnormal conditions that occur in your managed environment.
Because I've heard several people ask if RHQ has this ability, I put together a flash demo that shows how you do it. The demo shows how you can execute any script on a remote machine when an alert is triggered by RHQ. For example, if you set up an alert that detects when your app server is using an abnormally large number of threads (a possible indication of heavy load), you can have RHQ execute a custom script on your app server machine to help alleviate problems that might occur due to that condition (such a script could be one that reconfigures a load balancer to help redirect load away from your app server).
The use-cases for this feature are virtually endless. Any set of alert conditions on any managed resource can trigger the execution of any script you have. And as the demo illustrates, it is really easy to set this up within RHQ.
Saturday, February 5, 2011
Wednesday, January 26, 2011
Bundle Provisioning Via RHQ
A certain amount of enhancements and UI clean up were added to the RHQ Bundle/Provisioning feature. So, I figure now would be a good time to re-introduce it in a new blog entry. I also put together a flash demo if you would like to see the UI in action.
Let me recap what this RHQ Bundle/Provisioning feature is all about. RHQ allows you to bundle up a set of files and push them out to remote machines. You can install and upgrade these sets of files as well as revert back to a previous version of the files or purge the bundle files completely. You sometimes see this mentioned as the "Provisioning" feature, and other times you will see it referred to as the "Bundle" subsystem. (I prefer the term "Bundle" since that is the term that the RHQ user interface uses).
There are a few concepts you must know in order to understand how RHQ does its thing. This is covered on the wiki, but I'll try to explain it briefly here, too.
First is the concept of "bundle". A "bundle" is a logical concept and basically refers to an application ("Pet Store Application" or "My Wiki Server"). A bundle has one or more "versions". A "bundle version" refers to an actual set of files that you want to push out to a set of remote machines. Think of it as your application distribution. Each bundle version has its own "recipe" which tells RHQ what files exist in the bundle, and how to configure and provision those files to the remote machines. Developers or application packagers are responsible for writing the recipe and 'bundling up' the application's files (hence the name 'bundle') with the recipe into an RHQ "bundle version".
What you do with a bundle version brings us to the next set of concepts. A bundle "destination" is associated with a specific bundle and is simply a place (or places) where you want to deploy your bundle. A destination specifies two things - a group (which contains one or more remote machines) and a destination directory (which specifies where on the remote machines' filesystems the bundle files should go). Once you have a "bundle destination" in place, you can begin to deploy one or more of its bundle's versions to that destination. A bundle "deployment" represents one deployment of a bundle version to a destination.
It may make more sense if I give an example. Suppose I have a web application (call it "My Application") that runs inside a JBoss application server. This is my "bundle". I actually have two versions of my application, 1.0 and 2.0. These are my "bundle versions".
Now suppose that I have a QA environment that consists of two machines - a Windows machine and a Linux machine. I want to test my application on my two QA machines. So I need to install my application on both of them. I want to install my application on each machine's "/home/mazz/opt/myapp" directory. This group of two QA machines, along with the destination directory, is a "bundle destination" for my application bundle (call it the "QA destination"). I also have a group of three Linux machines that make up my production environment. After my application passes all tests, I want to deploy my application to that production environment in the "/opt" directory. That group of three production machines, along with the "/opt" directory specification, is another bundle destination associated with my application bundle - call it the "production destination".
Once I tell RHQ to deploy the "1.0" bundle version of my application to my "QA destination", I will have a "bundle deployment". This bundle deployment will be considered the "live" deployment because its the last one I pushed out. I can then test that version while it is on my QA machines. Suppose I find that I want to upgrade my QA environment with the newer "2.0" version of my application. I simply deploy that bundle version to the "QA destination" and now I have a second "bundle deployment". This second deployment is now considered live. If I find that I do not like this newer "2.0" version of my application, I can ask RHQ to revert back to the last live deployment (which was my "1.0" bundle version) - this revert becomes yet another "bundle deployment" (the third) but it reverts back to the "1.0" bundle version content. Once I pass all of my QA tests, I can then deploy whatever bundle version I deem appropriate to my "production destination".
Most of what I describe above is actually demonstrated in my flash demo. The only thing I do not show in the demo is the use of a second "production destination", but it is the same effort to deploy to a second destination as it is to deploy to the first destination.
One new feature that has been introduced to RHQ is the ability to "purge" a destination of all bundle content. If, for example, you want to remove all bundle files completely from the QA destination, you can ask RHQ to purge that destination. What RHQ will do is remove all bundle content from the remote machines that were associated with that destination.
Another new feature that has been added is the ability for RHQ to deploy a bundle into an already existing deployment directory that may have other non-managed content that should be left alone. Such would be the case if you want to deploy an EAR or WAR to a JBossAS deploy/ directory (which obviously has other files inside of it). This deserves some additional explanation.
Typically, you will want to deploy a set of application files into its own directory on some file system. For example, if you have a JBoss application server, you want to install it in something like "/opt/my-jboss". All of your application server files are in that directory, but no other files are in there. If you want to remove your JBoss application server installation, it is as simple as "rm -rf /opt/my-jboss".
However, what if you deployed a bundle version in that directory already, but you then upgrade that bundle deployment with a new bundle version? In this case, you will already have files in /opt/my-jboss (the original bundle version content). RHQ will actually overwrite, backup or ignore conflicting files that it finds following strict upgrade rules. If, for example, RHQ finds files in that /opt/my-jboss directory that don't belong to the new bundle version, they will be removed. RHQ calls this "managing the root directory".
This is usually what you want if you are deploying a standalone software product. If there are any unknown files in the deployment directory, RHQ has to remove them to make sure the bundle deployment directory is exactly in the state the new bundle version recipe wants it to be. However, this is not what you want if you desire to deploy an EAR or WAR to an already existing JBossAS's deploy/ directory. That's because we already know there will be unrelated files in this deploy/ directory that must remain intact and in place. RHQ must leave any files it finds in that destination directory alone - even though they aren't part of our bundle deployment. In other words, we do not want RHQ to manage this root deployment directory.
RHQ now supports this by allowing the Ant recipe author to specify the manageRootDir="false" attribute in the rhq:deployment-unit task. This new feature is documented in Bugzilla 659142 and this new attribute is documented on the RHQ wiki.
You can read more about the Provisioning/Bundle feature on the RHQ wiki.
Let me recap what this RHQ Bundle/Provisioning feature is all about. RHQ allows you to bundle up a set of files and push them out to remote machines. You can install and upgrade these sets of files as well as revert back to a previous version of the files or purge the bundle files completely. You sometimes see this mentioned as the "Provisioning" feature, and other times you will see it referred to as the "Bundle" subsystem. (I prefer the term "Bundle" since that is the term that the RHQ user interface uses).
There are a few concepts you must know in order to understand how RHQ does its thing. This is covered on the wiki, but I'll try to explain it briefly here, too.
First is the concept of "bundle". A "bundle" is a logical concept and basically refers to an application ("Pet Store Application" or "My Wiki Server"). A bundle has one or more "versions". A "bundle version" refers to an actual set of files that you want to push out to a set of remote machines. Think of it as your application distribution. Each bundle version has its own "recipe" which tells RHQ what files exist in the bundle, and how to configure and provision those files to the remote machines. Developers or application packagers are responsible for writing the recipe and 'bundling up' the application's files (hence the name 'bundle') with the recipe into an RHQ "bundle version".
What you do with a bundle version brings us to the next set of concepts. A bundle "destination" is associated with a specific bundle and is simply a place (or places) where you want to deploy your bundle. A destination specifies two things - a group (which contains one or more remote machines) and a destination directory (which specifies where on the remote machines' filesystems the bundle files should go). Once you have a "bundle destination" in place, you can begin to deploy one or more of its bundle's versions to that destination. A bundle "deployment" represents one deployment of a bundle version to a destination.
It may make more sense if I give an example. Suppose I have a web application (call it "My Application") that runs inside a JBoss application server. This is my "bundle". I actually have two versions of my application, 1.0 and 2.0. These are my "bundle versions".
Now suppose that I have a QA environment that consists of two machines - a Windows machine and a Linux machine. I want to test my application on my two QA machines. So I need to install my application on both of them. I want to install my application on each machine's "/home/mazz/opt/myapp" directory. This group of two QA machines, along with the destination directory, is a "bundle destination" for my application bundle (call it the "QA destination"). I also have a group of three Linux machines that make up my production environment. After my application passes all tests, I want to deploy my application to that production environment in the "/opt" directory. That group of three production machines, along with the "/opt" directory specification, is another bundle destination associated with my application bundle - call it the "production destination".
Once I tell RHQ to deploy the "1.0" bundle version of my application to my "QA destination", I will have a "bundle deployment". This bundle deployment will be considered the "live" deployment because its the last one I pushed out. I can then test that version while it is on my QA machines. Suppose I find that I want to upgrade my QA environment with the newer "2.0" version of my application. I simply deploy that bundle version to the "QA destination" and now I have a second "bundle deployment". This second deployment is now considered live. If I find that I do not like this newer "2.0" version of my application, I can ask RHQ to revert back to the last live deployment (which was my "1.0" bundle version) - this revert becomes yet another "bundle deployment" (the third) but it reverts back to the "1.0" bundle version content. Once I pass all of my QA tests, I can then deploy whatever bundle version I deem appropriate to my "production destination".
Most of what I describe above is actually demonstrated in my flash demo. The only thing I do not show in the demo is the use of a second "production destination", but it is the same effort to deploy to a second destination as it is to deploy to the first destination.
One new feature that has been introduced to RHQ is the ability to "purge" a destination of all bundle content. If, for example, you want to remove all bundle files completely from the QA destination, you can ask RHQ to purge that destination. What RHQ will do is remove all bundle content from the remote machines that were associated with that destination.
Another new feature that has been added is the ability for RHQ to deploy a bundle into an already existing deployment directory that may have other non-managed content that should be left alone. Such would be the case if you want to deploy an EAR or WAR to a JBossAS deploy/ directory (which obviously has other files inside of it). This deserves some additional explanation.
Typically, you will want to deploy a set of application files into its own directory on some file system. For example, if you have a JBoss application server, you want to install it in something like "/opt/my-jboss". All of your application server files are in that directory, but no other files are in there. If you want to remove your JBoss application server installation, it is as simple as "rm -rf /opt/my-jboss".
However, what if you deployed a bundle version in that directory already, but you then upgrade that bundle deployment with a new bundle version? In this case, you will already have files in /opt/my-jboss (the original bundle version content). RHQ will actually overwrite, backup or ignore conflicting files that it finds following strict upgrade rules. If, for example, RHQ finds files in that /opt/my-jboss directory that don't belong to the new bundle version, they will be removed. RHQ calls this "managing the root directory".
This is usually what you want if you are deploying a standalone software product. If there are any unknown files in the deployment directory, RHQ has to remove them to make sure the bundle deployment directory is exactly in the state the new bundle version recipe wants it to be. However, this is not what you want if you desire to deploy an EAR or WAR to an already existing JBossAS's deploy/ directory. That's because we already know there will be unrelated files in this deploy/ directory that must remain intact and in place. RHQ must leave any files it finds in that destination directory alone - even though they aren't part of our bundle deployment. In other words, we do not want RHQ to manage this root deployment directory.
RHQ now supports this by allowing the Ant recipe author to specify the manageRootDir="false" attribute in the rhq:deployment-unit task. This new feature is documented in Bugzilla 659142 and this new attribute is documented on the RHQ wiki.
You can read more about the Provisioning/Bundle feature on the RHQ wiki.
Tuesday, October 26, 2010
Using Byteman To Analyze the RHQ Agent
I recently had a need to run some quick and dirty performance testing over a few method calls in one of the RHQ plugins that was deployed in my agent. I didn't have a profiler installed and ready to go so I used Byteman. I quickly wrote up a couple of Byteman rules (thanks to Jay S. for correcting some problems I had with those rules), ran the RHQ Agent with Byteman installed and got some really good data.
It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.
Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.
I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.
I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).
Read the wiki page on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.
It didn't take me more than 15 minutes to get the agent and Byteman integrated and running with an initial set of rules and it was very simple to explain it to Jay so he could run with the Agent/Byteman combo for his own testing he was doing. It took us longer to determine what rules we needed, since we were using different rules to probe around the agent runtime to see different method timings before we narrowed down a potential problem area.
Another nice thing about this is how easily it is to share rules in order to collaborate with others during testing. When Jay came up with a set of rules that produced an interesting set of data - he just sent his rules to me over IM, and I quickly installed them in my agent's Byteman instance. This allowed me to attempt to replicate the same behavior Jay was seeing with his own agent. Within a few minutes we were able to confirm the behavior on his environment was the same as my environment thanks to the data emitted by his rules.
I realized that this is really a handle tool for not only RHQ core developers, but also for plugin developers out in the community and those that have RHQ deployed and running in a managed environment. If a support engineer needs to get some data from deep within the agent or one of its plugins and log files can't provide this data, using Byteman is an easy, fast and effective way to do it - all without asking the user to perform some heavyweight install of a third-party profiler or to redeploy a "debug version" of the agent or plugin jars.
I bundled up the one required Byteman jar, a sample rules file and a script that you can use to run the RHQ Agent with Byteman installed and preconfigured to run the rules in the rules file. All you have to do is edit the rules file with your own rules and use the provided script to start the agent. No additional third party downloads/installs are needed, you can use the same RHQ Agent you have installed already and no additional reconfiguration of the RHQ Agent is required (except in the case where you have configured a custom RHQ_AGENT_ADDITIONAL_JAVA_OPTS value in rhq-agent-env.sh - if you do, see the comments in rhq-agent-with-byteman.sh to see what you need to do).
Read the wiki page on this for more information if you are interested. You could even tweek the rhq-agent-with-byteman.sh to have it start your own Java app if you want to do this kind of thing outside of an RHQ environment.
Thursday, August 5, 2010
Remotely Installing An Agent
One new feature coming soon to RHQ is the ability to remotely install an RHQ Agent on to any machine in your network (with the caveat that the remote machine must be accessible via SSH).
What this now means is you do not have to manually log onto a remote machine and do the tasks of downloading the agent update binary distribution, installing it and running it. RHQ can now do this for you all remotely - all you need to provide to the RHQ GUI is the machine name, your SSH credentials and the location where you want to install the agent.
I think this could still use some enhancements - there is no way to customize each agent's configuration (i.e. providing custom answers to the startup setup questions). But that's for another day. If you want to customize the new agent, you can do so by just importing the RHQ Agent resource into inventory and going to its Configuration tab and adjusting its configuration (don't forget to restart the agent so it can pick up the changed configuration- you can restart the agent using this new RHQ GUI page! I'll talk about that next).
This mechanism will also allow you to stop and start an existing agent that is already installed. This is very useful if you have not started the agent yet but wish to bring it back online, or for those cases where you have not commited the RHQ Agent resource in inventory yet but still need a way to shutdown or restart an agent.
I put together a flash demo that shows this stuff in action.
What this now means is you do not have to manually log onto a remote machine and do the tasks of downloading the agent update binary distribution, installing it and running it. RHQ can now do this for you all remotely - all you need to provide to the RHQ GUI is the machine name, your SSH credentials and the location where you want to install the agent.
I think this could still use some enhancements - there is no way to customize each agent's configuration (i.e. providing custom answers to the startup setup questions). But that's for another day. If you want to customize the new agent, you can do so by just importing the RHQ Agent resource into inventory and going to its Configuration tab and adjusting its configuration (don't forget to restart the agent so it can pick up the changed configuration- you can restart the agent using this new RHQ GUI page! I'll talk about that next).
This mechanism will also allow you to stop and start an existing agent that is already installed. This is very useful if you have not started the agent yet but wish to bring it back online, or for those cases where you have not commited the RHQ Agent resource in inventory yet but still need a way to shutdown or restart an agent.
I put together a flash demo that shows this stuff in action.
Thursday, July 8, 2010
RHQ 3.0.0 Has Been Released
The RHQ team is proud to announce the immediate availability of RHQ 3.0.0.
This release features several months of hard work by the development team and external contributors. Many of the changes have been already made available in the past through seven community releases.
You can browse the full release notes on the RHQ wiki.
The release can be downloaded via the RHQ web site or directly from SourceForge.
Download it and try it out. The documentation wiki has full install instructions.
This release features several months of hard work by the development team and external contributors. Many of the changes have been already made available in the past through seven community releases.
You can browse the full release notes on the RHQ wiki.
The release can be downloaded via the RHQ web site or directly from SourceForge.
Download it and try it out. The documentation wiki has full install instructions.
Wednesday, June 2, 2010
Provisioning Content via RHQ
Work has been steadily progressing in RHQ to introduce a new "provisioning" feature. Our beta will soon be released, and as part of our getting ready of that release, I put together a quick flash demo that illustrates our current state of this provisioning feature.
There is still much more work to get done - the new GWT GUI is continually improving and we need more provisioning support for things like starting, stopping and installing services being provisioning, orchestration between machines in a cluster (to support provisioning software within a cluster) and other things. But what we have is a good base framework to build upon - watch the demo if you want to see what we have now.
For more information, we have documentation on our wiki:
http://rhq-project.org/display/JOPR2/Provisioning
There is still much more work to get done - the new GWT GUI is continually improving and we need more provisioning support for things like starting, stopping and installing services being provisioning, orchestration between machines in a cluster (to support provisioning software within a cluster) and other things. But what we have is a good base framework to build upon - watch the demo if you want to see what we have now.
For more information, we have documentation on our wiki:
http://rhq-project.org/display/JOPR2/Provisioning
Thursday, March 25, 2010
Hibernate Performance Monitoring
Joseph Marques has a very good blog (good in both substance and technical details) on how you can analyze the performance of your Hibernate application through the use of a simple set of utility classes. He even goes into how you can do this inside your EJB3 app. We do this when analyzing performance in RHQ (in addition to using RHQ itself to do some analysis). Joe's code examples show that it's a solution that is easily transferable to any Java application that uses Hibernate as its ORM layer.
Subscribe to:
Posts (Atom)