Wednesday, May 8, 2013

Creating Https Connection Without javax.net.ssl.trustStore Property

Question: How can you use the simple Java API call java.net.URL.openConnection() to obtain a secure HTTP connection without having to set or use the global system property "javax.net.ssl.trustStore"? How can you make a secure HTTP connection and not even need a truststore?

I will show you how you can do both below.

First, some background. Java has a basic API to make a simple HTTP connection to any URL via URL.openConnection(). If your URL uses the "http" protocol, it is very simple to use this to make basic HTTP connections.

Problems creep in when you want a secure connection over SSL (via the "https" protocol). You can still use that API - URL.openConnection() will return a HttpsURLConnection if the URL uses the https protocol - however, you must ensure your JVM can find and access your truststore in order to authenticate the remote server's certificate.

[note: I won't discuss how you get your trusted certificates and how you put them in your truststore - I'll assume you know, or can find out, how to do this.]

You tell your JVM where your truststore is by setting the system property "javax.net.ssl.trustStore" and you tell your JVM how to access your truststore by giving your JVM the password via the system property "javax.net.ssl.trustStorePassword".

The problem is these are global settings (you often see instructions telling you to set these values via the -D command line arguments when starting your Java process) so everything running in your JVM must use that truststore. And you can't alter those system properties during runtime and expect those changes to take effect. Once you ask the JVM to make a secure connection, those system property values appear to be cached in the JVM and are used thereafter for the life of the JVM (I don't know exactly where in the JRE code these values are cached, but my experience shows me that they are). Changing those system properties later on in the lifetime of the JVM has no effect; the original values are forever used.

Another problem that some people run into is having the need for a truststore in the first place. Sometimes you don't have a requirement to authenticate the server endpoint; however, you would still like to send your data encrypted over the wire. You can't do this readily since the connection you obtain from URL.openConnection() will, by default, expect to use your truststore located at the path pointed to by the system property javax.net.ssl.trustStore.

To allow me to use different truststores for different connections, or to allow me to encrypt a connection but not authenticate the endpoint, I wrote a Java utility object that allows you to do just this.

The main constructor is this:

public SecureConnector(String secureSocketProtocol,
                       File   truststoreFile,
                       String truststorePassword,
                       String truststoreType,
                       String truststoreAlgorithm)

You pass it a secure socket protocol (such as "TLS") and your truststore file location. If the truststore file is null, the SecureConnector object will assume you do not want to authenticate the remote server endpoint and you only want to encrypt your over-the-wire traffic. If you do provide a truststore file, you need to provide its password, its type (e.g. "JKS"), and its algorithm (e.g. "SunX509") - if you pass in null for type and/or algorithm, the JVM defaults are used.

Once you create the object, just obtain a secure connection to any URL via a call to SecureConnector.openSecureConnection(URL). This expects your URL to have a protocol of "https". If successful, an HttpsURLConnection object is returned and you can use it like any other connection object. You do not need to set javax.net.ssl.trustStore (or any other javax.net.ssl system property) and, as explained above, you don't even need to provide a truststore at all (assuming you don't need to do any authentication).

The code for this is found inside of RHQ's agent - you can read its javadoc and look through SecureConnector code here.

The core code is found in openSecureConnection and looks like this, I'll break it down:

First, it simply obtains the HTTPS connection object from the URL itself:
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
Then it prepares a custom SSLContext object using the given secure socket protocol:
TrustManager[] trustManagers;
SSLContext sslContext = SSLContext.getInstance(getSecureSocketProtocol());
If no truststore file was provided, it will build its own "no-op" trust manager and "no-op" hostname verifier. What these "no-op" objects will do is always accept all certificates and hostnames thus they will always allow the SSL communications to flow. This is how the authentication is by-passed:
if (getTruststoreFile() == null) {
    // configured to not care about authenticating server, encrypt but don't worry about certificates
    trustManagers = new TrustManager[] { NO_OP_TRUST_MANAGER };
    connection.setHostnameVerifier(NO_OP_HOSTNAME_VERIFIER);
If a truststore file was provided, then it will be loaded in memory and stored in a KeyStore instance:
} else {
    // need to configure SSL connection with truststore so we can authenticate the server.
    // First, create a KeyStore, but load it with our truststore entries.
    KeyStore keyStore = KeyStore.getInstance(getTruststoreType());
    keyStore.load(new FileInputStream(getTruststoreFile()), getTruststorePassword().toCharArray());
The truststore file's content (now stored in a KeyStore object) is used to initialize a trust manager. Unlike the "no-op" trust manager that was created above (if a truststore file was not provided), this trust manager really does perform authentication and it uses the provided truststore's certificates to authorize the server being communicated with. This is why we no longer need to worry about the system properties "javax.net.ssl.trustStore" and "javax.net.ssl.trustStorePassword" - this builds its own trust manager using the data provided by the caller:
    // create truststore manager and initialize it with KeyStore we created with all truststore entries
    TrustManagerFactory tmf = TrustManagerFactory.getInstance(getTruststoreAlgorithm());
    tmf.init(keyStore);
    trustManagers = tmf.getTrustManagers();
}
Finally, the SSL context is initialized with the trust manager that was created earlier (either the "no-op" trust manager, or the trust manager that was initialized with the truststore's certificates). That SSL context is handed off to the SSL connection so the connection can use the context when it needs to perform authentication:
sslContext.init(null, trustManagers, null);
connection.setSSLSocketFactory(sslContext.getSocketFactory());
The connection is finally returned to the caller, fully configured and ready to be used.
return connection;
This is helpful for certain use cases. First, it is helpful when you have multiple truststores that you need to choose from when connecting to different servers as well as being able to switch truststores at runtime (remember, the system property values of javax.net.ssl.trustStore, et. al. are fixed for the lifetime of the JVM - this helps bypass that restriction). This is also helpful in local testing, debugging and demo scenarios when you don't really need or care about setting up truststores and certificates but you do want to connect over https.

Thursday, March 14, 2013

Deleting RHQ Agent Made Easier

In the past, if you wanted to remove an RHQ Agent from your RHQ environment, the simple answer was "just uninventory the platform" which would, under the covers, also remove the agent record completely.

However, in some cases, users found it difficult to remove their agent. Usually, what happens is they try to install the RHQ Agent, run into problems, and then get their RHQ system in a state that causes their RHQ Agent to not be able to register with the RHQ Server. For example, this could happen when a person runs the agent as a different user from before or with the -L command line option - both of which essentially purges the agent's security token and could cause the RHQ Server to reject future registration requests from the agent.

If a person does not understand the linkage between platform and agent, or if the agent's platform was never committed to inventory, it became difficult to understand how to get out of the quadmire.

This has now been addressed as an enhancement as requested by BZ 849711. Now, the answer is simple - regardless of whether the platform is in inventory or not, and even if the agent's resources do not yet show up in the discovery queue - you have a way to quickly purge your agent from the system. This will allow you to get back to a clean slate and attempt to re-install your agent.

You do this by going to the top Adminstration page and selecting the "Agents" item. From here, you see the list of all the agents currently registered in your RHQ environment. If you select one or more of them, you now have the option of pressing the new "Delete" button at the bottom. This will do a few things. First, if the agent's platform is already in inventory, it will uninventory that platform. This means that platform and all its child servers and services will be removed (so be careful and make sure you really want to do this - you will lose all manageability and all audit history for all resources previously managed by that agent). Once that is done, all resources will disappear from the inventory and you won't even see any resources for that agent in the Discovery Queue. Finally, the agent's record itself is removed - so the Administration>Agents page will show that the agents have disappeared.

 
With the agent and its resources completely removed, you have the option to attempt to re-install the agent if you wish to bring it back.

You can also use this feature if your managed infrastructure has changed and you no longer want to manage a machine. Just select the agent that was responsible for managing that machine and delete it.

Note that if your agent is still running, it will attempt to re-register itself! So if you no longer wish to manage a machine, make sure you shutdown the agent as well (you'll obviously want to do this anyway, since you won't want an RHQ Agent consuming resources on a machine that you no longer want managed by RHQ).

Friday, February 8, 2013

"Nested Transactions" and Timeouts

While coding up some EJB3 SLSB methods, the following question came up:

If a thread is already associated with a transaction, and that thread calls another EJB3 SLSB method annotated with "REQUIRES_NEW", the thread gets a new (or "pseudo-nested" or for ease-of-writing what I'm simply call the "nested" transaction, even though it is not really "nested" in the true sense of the term) transaction but what happens to the parent transaction while the "nested" transaction is active? Does the parent transaction's timeout keep counting down or is it suspended and will start back up counting down only when the "nested" transaction completes?

For example, suppose I enter a transaction context by calling method A and this method has a transaction timeout of 1 minute. Method A then immediatlely calls a REQUIRES_NEW method B, which itself has a 5 minute timeout. Now suppose method B takes 3 minutes to complete. That is within B's allotted 5 minute timeout so it returns normally to A. A then immediately returns.

But A's timeout is 1 minute! B took 3 minutes on its own. Even though the amount of time A took within itself was well below its allotted 1 minute timeout, its call to B took 3 minutes.

What happens? Does A's timer "suspend" while its "nested" transaction (created from B) is still active?  Or does A's timer keep counting down, regardless of whether or not B's "nested" transaction is being counted down at the same time (and hence A will abort with a timeout)?

Here's some code to illustrate the use-case (this is what I actually used to test this):

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
@TransactionTimeout(value = 5, unit = TimeUnit.SECONDS)
public void testNewTransaction() throws InterruptedException {
   log.warn("~~~~~ Starting new transaction with 5s timeout...");
   LookupUtil.getTest().testNewTransaction2();
   log.warn("~~~~~ Finishing new transaction with 5s timeout...");
}
@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
@TransactionTimeout(value = 10, unit = TimeUnit.SECONDS)
public void testNewTransaction2() throws InterruptedException {
   log.warn("~~~~~ Starting new transaction with 10s timeout...sleeping for 8s");
   Thread.sleep(8000);
   log.warn("~~~~~ Finishing new transaction with 10s timeout...");
}
I don't know what any of the EE specs say about this, but it doesn't matter - all I need to know is how JBossAS7 behaves :) So I ran this test on JBossAS 7.1.1.Final and here's what the log messages say:
17:51:22,935 ~~~~~ Starting new transaction with 5s timeout...
17:51:22,947 ~~~~~ Starting new transaction with 10s timeout...sleeping for 8s
17:51:27,932 WARN  ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffc0a80102:-751071d8:5115811c:449 in state  RUN
17:51:27,936 WARN  ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffffc0a80102:-751071d8:5115811c:449
17:51:30,948 ~~~~~ Finishing new transaction with 10s timeout...
17:51:30,949 ~~~~~ Finishing new transaction with 5s timeout...
17:51:30,950 WARN  ARJUNA012077: Abort called on already aborted atomic action 0:ffffc0a80102:-751071d8:5115811c:449
17:51:30,951 ERROR JBAS014134: EJB Invocation failed on component TestBean for method public abstract void org.rhq.enterprise.server.test.TestLocal.testNewTransaction() throws java.lang.InterruptedException: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
...
Caused by: javax.transaction.RollbackException: ARJUNA016063: The transaction is not active!
   at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1155) [jbossjts-4.16.2.Final.jar:]
   at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) [jbossjts-4.16.2.Final.jar:]
   at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
   at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
So it is clear that, at least for JBossAS7's transaction manager, the parent transaction's timer is not suspended, even if a "nested" transaction is activated. You see I enter the first method (which activates my first transaction) at 17:51:22, and immediately enter the second method (which activates my second, "nested", transaction at the same time of 17:51:22). My first transaction has a timeout of 5 seconds, my second "nested" transaction has a timeout of 10 seconds. My second method sleeps for 8 seconds, so it should finish at 17:51:30 (and it does if you look at the log messages at that time). BUT! Prior to that, my first transaction is aborted by the transaction manager at 17:51:27 - exactly 5 seconds after my first transaction was started. So, clearly my first transaction's timer was not suspended and was continually counting down even as my "nested" transaction was active.

So, in short, the answer is (for JBossAS7 at least) - a transaction timeout is always counting down and starts as soon as the transaction activates. It never pauses nor suspends due to "nested" transactions.

Wednesday, January 23, 2013

PostgreSQL Tool To Analyze DB Performance

I needed to do some fine-grained performance analysis of my PostgreSQL database setup and came across an interesting tool. It's pgBadger and it is used to take a snapshot of your PostgreSQL performance data. It is not a realtime monitoring system - but it is easy to install and run and it outputs a nice HTML report that is a snapshot of your logged performance data.

I didn't have to do anything special to get it built and installed on my Fedora 15 box. I just followed their simple instructions and it built and ran fine.

I did have to configure PostgreSQL to spit out performance data in its logs, but again that was easy to do following their instructions. It just required simple changes to postgresql.conf and a restart of the DB. Once I did that, PostgreSQL started logging performance data in log files located in the PostgreSQL data/pg_log directory.

I then just ran pgbadger, passing in the names of the logs files as command line arguments, and after a few seconds it spit out an HTML report. A typical snapshot report that it generates out can be seen here.


Monday, November 26, 2012

Monitoring IP Endpoints Via RHQ

RHQ has a little known plugin called the "netservices" plugin. Someone was asking about how RHQ can monitor HTTP endpoints on the #rhq freenode chat room - you use this netservices plugin to do this.

There are actually two different resource types that plugin provides - one allows you to monitor an HTTP URL endpoint (e.g. so you can monitor for different HTTP status codes). The other is a very basic resource type that just verifies that you can ping a specific IP or hostname.

Here's the RHQ 4.5.1 version of that plugin and its source:

http://mirrors.ibiblio.org/maven2/org/rhq/rhq-netservices-plugin/4.5.1/

(note: if anyone in the community is interested in doing some work on RHQ and looking for something simple to do - here's a perfect opportunity. It would be nice to have some documentation written for that plugin, but more importantly, I think we could beef up this plugin. For example, it would be nice to have another resource type (similar to that basic PingService) that is able to confirm that a particular port on a particular IP/host can be connected to. We also could use some testing done on this plugin to make sure it still works as expected.)

Thursday, August 2, 2012

Fedorahosted GIT links have changed (again)

Well, fedorahosted did it again.

All of my links to RHQ source code in all of my blog entries (and everywhere else I used them - bugzilla entries, wiki pages, etc) will be invalid now, because fedorahosted once again changed their git URLs and AFAICS the old URLs no longer work and do not redirect to the new URLs.

This is at least the second time that I know of that they did this - it may have happened more. Previously, I went through all of my blog entries and updated my source links, but I am not going to do it this time. It is too time consuming and I am not convinced this won't happen again.

So, if you click a link to source code in my blogs and get an error message saying the link is invalid, you know why. :-/

(Heiko is right, we need to move to github :-)

Friday, July 13, 2012

Co-Located Management Client For JBossAS 7

Back in the "olden days" (pre-JBossAS 7 :-), the JBoss Application Server provided management capabilities through MBeans hosted in its MBeanServer. Your own code could remotely manage the app server by connecting to its MBeanServer through remote connectors, but, if your code was running co-located in the same app server that you wanted to manage, your code could talk directly to the MBeanServer through the normal JMX API.

JBossAS 7 has a new management subsystem and its management API has been completely redesigned and is no longer dependent on JMX. The new AS7 management subsystem is exposed in several ways; in fact, as I see it, there are five different ways to manage AS7 - two of which are programmatic:

  1. The HTTP management endpoint which uses JSON and a de-typed RPC style API.
  2. "ModelControllerClient" (found in the jboss-as-controller-client library) which provides a factory that allows you to remotely connect to an AS7 instance and utilizes jboss-dmr as its de-typed API.
  3. The administration console (aka "the web interface") which is a GWT application that remotely manages AS7 via the previously mentioned HTTP management endpoint.
  4. The command line interface (aka "the CLI") which is a console-based and Swing GUI-based user interface that remotely manages AS7 via the previously mentioned ModelControllerClient.
  5. Manual editing of XML configuration files.
You can use (1) and (2) programmatically to build management tools. And, in fact, you can see this is what the AS7 folks did when they built AS7's own management tools (the web interface (3) utilizes the HTTP endpoint and the CLI (4) utilizes the ModelControllerClient).

However, as you may have noticed, one thing is conspicuously absent - that being a way to manage an application server which is co-located with the management tool itself without going over some sort of remote connector or endpoint. In the previous JMX-based way of doing JBossAS management, if your management code was running in the app server itself, you could obtain the app server's MBeanServer without going through some sort of remote connector thereby allowing you to call the management API by directly accessing that MBeanServer. However, in AS7, there is nothing readily available that gives you local, intra-VM, access to the underlying management capabilities. Even though your code may be running in the same AS7 instance that you want to manage, you still have to make a "remote" round-trip connection back to yourself (using either an HTTP connection or the standard ModelControllerClient factory).

In this blog I propose* a sixth mechanism to manage AS7 - that being, a local mechanism that circumvents any need for a remote connection and allows you to directly access the management API.
(* full credit to this idea actually belongs to Kabir Khan who kindly gave me the original prototype implementation - thanks Kabir!)

Obviously, this sixth mechanism is only useful if your management code is co-located with the AS7 being managed. But, when you have this situation, it does seem that this need has been, up to now, left unfulfilled. This mechanism actually will utilize the same ModelControllerClient and jboss-dmr API as the CLI uses, however, it does not require that you connect the client to the app server over a remote connector. Rather, what this will do is activate a service within AS7 - that service will get injected with AS7's internal ModelController object and then it will expose that object via a ModelControllerClient. With that client, any code running inside the AS7 instance can talk directly to AS7's ModelController thus giving you direct access to the underlying management functionality without the need for going through a remote connector.

Here's one way you can do this. In this example, let's assume there is a WAR which will contain an AS7 MSC service (I'll just refer to it as a "service"). This AS7 service will expose the local ModelControllerClient to the WAR via a very simple singleton pattern. I could conceive of other ways in which you could expose the local client - you could even do so through JMX! Just put the ModelControllerClient in an attribute on some MBean that the AS7 service registers in a local MBeanServer. Your local JMX clients could then obtain the ModelControllerClient via JMX. But I digress.

First, you need to declare that your WAR deployment has a ServiceActivator. You do this very simply - by placing a file called "org.jboss.msc.service.ServiceActivator" in your WAR's META-INF/services directory (note: there is currently a bug that requires META-INF/services/org.jboss.msc.service.ServiceActivator to actually be placed under the WAR's WEB-INF/classes directory. Refer to that bug report for more information). The content of this file is the fully qualified class name of a ServiceActivator implementation class that can be found in the WAR (in my example, this would be the full class name of the ManagementService class that I will describe below).

Second, because this mechanism requires that our service utilize some AS7 specific libraries, you need to add some dependencies to the WAR so it can access those AS7 libraries. The dependencies are identified by module names (these are the names that AS7 uses to identify the libraries). Some of the libraries that the WAR needs access to are identified by module names such as org.jboss.msc, org.jboss.as.controller-client, et. al. You can specify these additional dependencies by simply adding a Dependencies entry in the WAR's META-INF/MANIFEST.MF file like this:
Dependencies: org.jboss.msc,org.jboss.as.controller-client,org.jboss.a s.controller,org.jboss.as.server
Third, you need to actually implement this ServiceActivator class and place it in the WAR (e.g. it can go in WEB-INF/classes). For this example, I will call it "ManagementService" because it is a service that simply exposes the local management client to the WAR. Here is the code for this service:

public class ManagementService implements ServiceActivator {
   private static volatile ModelController controller;
   private static volatile ExecutorService executor;


   public static ModelControllerClient getClient() {
      return controller.createClient(executor);
   }


   @Override
   public void activate(ServiceActivatorContext context) throws ServiceRegistryException {
      final GetModelControllerService service = new GetModelControllerService();
      context
          .getServiceTarget()
          .addService(ServiceName.of("management", "client", "getter"), service)
          .addDependency(Services.JBOSS_SERVER_CONTROLLER, ModelController.class, service.modelControllerValue)
          .install();
   }


   private class GetModelControllerService implements Service[Void> {
      private InjectedValue[ModelController> modelControllerValue = new InjectedValue[ModelController>();


      @Override
      public Void getValue() throws IllegalStateException, IllegalArgumentException {
         return null;
      }


      @Override
      public void start(StartContext context) throws StartException {
         ManagementService.executor = Executors.newFixedThreadPool(5, new ThreadFactory() {
             @Override
             public Thread newThread(Runnable r) {
                 Thread t = new Thread(r);
                 t.setDaemon(true);
                 t.setName("ManagementServiceModelControllerClientThread");
                 return t;
             }
         });
         ManagementService.controller = modelControllerValue.getValue();
      }


      @Override
      public void stop(StopContext context) {
         try {
            ManagementService.executor.shutdownNow();
         } finally {
            ManagementService.executor = null;
            ManagementService.controller = null;
         }
      }
   }
}

You'll notice that its activate() method simply injects the existing AS7 management controller (whose name is defined by Services.JBOSS_SERVER_CONTROLLER) into the service being activated. This AS7 management controller is the "secret sauce" - it is the object that interacts with the AS7's management subsystem. The actual service is defined via a small private class (GetModelControllerService) whose main job is simply to take the injected ModelController and store it in the ManagementService.controller static field.

Once this service is activated (which happens when the WAR is deployed), classes deployed in the WAR itself can access the management API by obtaining a ModelControllerClient via ManagementService.getClient(). This wraps the ModelController in a client object which interacts with the ModelController thus allowing any class in the WAR to start managing the AS7 instance via the client. But unlike the typical usage of ModelControllerClient (like what you see in the CLI) it doesn't go through a remote connector. It literally talks directly to the underlying AS7 ModelController object giving you a direct channel to the AS7 management subsystem.