Wednesday, September 20, 2017

Prometheus JMX Exporter Configuration For WildFly 10

I'm currently looking at WildFly JMX MBeans, particular for exposing their attributes as metrics via the Prometheus JMX Exporter.

Here's the JMX Exporter configuration I've tentatively come up with. Let me know in the comments if I've forgotten something or did something wrong.

In short, this JMX Exporter config should collect the metrics for:

* Deployment (EARs)
* Subdeployment (WARs inside of EARs)
* JMS Topics and Queues
* Datasources
* Transactions
* Web Subsystem (i.e. Undertow)
* Servlets
* EJBs (Stateless, Stateful, Singleton, and Message Driven)

It will also store server paths under the "wildfly_config" metric family with the label name identifying the path such as "jboss_home_dir" or "jboss_server_config_dir" and the label value being the path itself.

And of course the default behavior of the JMX Exporter is to also emit metrics for the JVM statistics as well (such as heap and non-heap memory usage) so we get those "for free".

The config is as follows:


---
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:

  # CONFIG

  - pattern: "^jboss.as<path=(.+)><>path:(.+)"
    attrNameSnakeCase: true
    name: wildfly_config
    labels:
      $1: $2
    value: 1

  # DEPLOYMENTS

  - pattern: "^jboss.as<deployment=(.+), subdeployment=(.+), subsystem=undertow><>(.+_sessions|session_.+):"
    attrNameSnakeCase: true
    name: wildfly_deployment_$3
    type: GAUGE
    labels:
      name: $1
      subdeployment: $2

  - pattern: "^jboss.as<deployment=(.+), subsystem=undertow><>(.+_sessions|session_.+):"
    attrNameSnakeCase: true
    name: wildfly_deployment_$2
    type: GAUGE
    labels:
      name: $1

  # MESSAGING

  - pattern: "^jboss.as<subsystem=messaging-activemq, server=(.+), jms-(queue|topic)=(.+)><>(.+):"
    attrNameSnakeCase: true
    name: wildfly_messaging_$4
    type: GAUGE
    labels:
      server: $1
      $2: $3

  # DATASOURCES

  - pattern: "^jboss.as<subsystem=datasources, (?:xa-)*data-source=(.+), statistics=(.+)><>(.+):"
    attrNameSnakeCase: true
    name: wildfly_datasource_$2_$3
    type: GAUGE
    labels:
      name: $1

  # TRANSACTIONS

  - pattern: "^jboss.as<subsystem=transactions><>number_of_(.+):"
    attrNameSnakeCase: true
    name: wildfly_transactions_$1
    type: GAUGE

  # WEB SUBSYSTEM

  - pattern: "^jboss.as<subsystem=undertow, server=(.+), (https?)-listener=(.+)><>(bytes_.+|error_count|processing_time|request_count):"
    attrNameSnakeCase: true
    name: wildfly_undertow_$4
    type: GAUGE
    labels:
      server: $1
      listener: $3
      protocol: $2

  # SERVLET

  - pattern: "^jboss.as<deployment=(.+), subdeployment=(.+), subsystem=undertow, servlet=(.+)><>(.+_time|.+_count):"
    attrNameSnakeCase: true
    name: wildfly_servlet_$4
    type: GAUGE
    labels:
      name: $3
      deployment: $1
      subdeployment: $2

  - pattern: "^jboss.as<deployment=(.+), subsystem=undertow, servlet=(.+)><>(.+_time|.+_count):"
    attrNameSnakeCase: true
    name: wildfly_servlet_$3
    type: GAUGE
    labels:
      name: $2
      deployment: $1

  # EJB

  - pattern: "^jboss.as<deployment=(.+), subdeployment=(.+), subsystem=ejb3, (stateless-session|stateful-session|message-driven|singleton)-bean=(.+)><>(.+):"
    attrNameSnakeCase: true
    name: wildfly_ejb_$5
    type: GAUGE
    labels:
      name: $4
      type: $3
      deployment: $1
      subdeployment: $2

  - pattern: "^jboss.as<deployment=(.+), subsystem=ejb3, (stateless-session|stateful-session|message-driven|singleton)-bean=(.+)><>(.+):"
    attrNameSnakeCase: true
    name: wildfly_ejb_$4
    type: GAUGE
    labels:
      name: $3
      type: $2
      deployment: $1

Friday, September 1, 2017

Hawkular Alerts with Jaeger / OpenTracing

Two recent blogs discuss how OpenTracing instrumentation can be used to collect application metrics:

A further interesting integration can be the addition of Hawkular Alerts to the environment.

As the previous blog and demo discuss, Hawkular Alerts is a generic, federated alerts system that can trigger events, alerts, and notifications from different, independent systems such as Prometheus, ElasticSearch, and Kafka.

Here we can combine the two. Let's follow the directions for the OpenTracing demo (using the Jaeger implementation) and add Hawkular Alerts.

What this can show is OpenTracing application metrics triggering alerts when (as in this example) OpenTracing spans encounter a larger-than-expected error rates.

(Note: these instructions assume you are using Kubernetes / Minikube - see the Hawkular OpenTracing blogs linked above for more details on these instructions)

1. START KUBERNETES

Here we start minikube giving it enough resources to run all of the pods necessary for this demo. We also start up a browser pointing to the Kubernetes dashboard, so you can follow the progress of the remaining instructions.
  • minikube start --cpus 4 --memory 8192
  • minikube dashboard
2. DEPLOY PROMETHEUS
  • kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml
  • kubectl create -f https://raw.githubusercontent.com/objectiser/opentracing-prometheus-example/master/prometheus-kubernetes.yml
    • (Note: The above command might not work depending on your version - if you get errors, download a copy of prometheus-kubernetes.yml and edit it, changing “v1alpha1” to “v1”)
3. DEPLOY JAEGER
  • kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml
The following will build and deploy the Jaeger example code that will produce the OpenTracing data for the demo:
  • mkdir -p ${HOME}/opentracing ; cd ${HOME}/opentracing
  • git clone git@github.com:objectiser/opentracing-prometheus-example.git
  • cd opentracing-prometheus-example/simple
  • eval $(minikube docker-env)
  • mvn clean install docker:build
  • kubectl create -f services-kubernetes.yml
    • (Note: The above command might not work depending on your version - if you get errors, edit services-kubernetes.yml and edit it, changing “v1alpha1” to “v1”)
4. DEPLOY HAWKULAR-ALERTS AND CREATE ALERT TRIGGER

The following will deploy Hawkular Alerts and create the trigger definition that will trigger an alert when the Jaeger OpenTracing data indicates an error rate that is over 30%
  • kubectl create -f https://raw.githubusercontent.com/hawkular/hawkular-alerts/master/dist/hawkular-alerts-k8s.yml
  • Use “minikube service hawkular-alerts --url” to determine the Hawkular Alerts URL and point your browser to the path “/hawkular/alerts/ui” at that URL (i.e. http://host:port/hawkular/alerts/ui).
  • From the browser page running the Hawkular Alerts UI, enter a tenant name in the top right text field (“my-organization” for example) and click the “Change” button.
  • Navigate to the “Triggers” page (found in the left-hand nav menu).
  • Click the kabob menu icon at the top and select “New Trigger”.
  • In the text area, enter the following to define a new trigger that will trigger alerts when the Prometheus query shows that there is a 30% error rate or greater in the accountmgr or ordermgr servers:
  •  {
       "trigger":{
          "id":"jaeger-prom-trigger",
          "name":"High Error Rate",
          "description":"Data indicates high error rate",
          "severity":"HIGH",
          "enabled":true,
          "autoDisable":false,
          "tags":{
             "prometheus":"Test"
          },
          "context":{
             "prometheus.url":"http://prometheus:9090"
          }
       },
       "conditions":[
          {
             "type":"EXTERNAL",
             "alerterId":"prometheus",
             "dataId":"prometheus-test",
             "expression":"(sum(increase(span_count{error=\"true\",span_kind=\"server\"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind=\"server\"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) > 0.3"
          }
       ]
    }
  • Now navigate back to the “Dashboard” page (again via the left-hand nav menu). From this Dashboard page, look for alerts when they are triggered. We'll next start generating the data that will trigger these alerts.
5. GENERATE SOME SAMPLE OPEN TRACING APPLICATION DATA
  • export ORDERMGR=$(minikube service ordermgr --url)
  • ${HOME}/opentracing/opentracing-prometheus-example/simple/genorders.sh
Once the data starts to be collected, you will see alerts in the Hawkular Alerts UI as error rates become over 30% in the past minute (as per the Prometheus query).

If you look at the alerts information in the Hawkular Alerts UI, you’ll see the conditions that triggered the alerts, such as:
Time: 2017-09-01 17:41:17 -0400
External[prometheus]: prometheus-test[Event [tenantId=my-organization, id=1a81471d-340d-4dba-abe9-5b991326dc80, ctime=1504302077288, category=prometheus, dataId=prometheus-test, dataSource=_none_, text=[1.504302077286E9, 0.3333333333333333], context={service=ordermgr, version=0.0.1}, tags={}, trigger=null]] matches [(sum(increase(span_count{error="true",span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) > 0.3]
Notice the “ordermgr” service (version "0.0.1") had an error rate of 0.3333 (33%) which caused the alert since it is above the allowed 30% threshold.

At this point, the Hawkular Alerts UI provides the ability for system admins to log notes about the issue, acknowledge the alert and mark the alert resolved if the underlying issue has been fixed. These lifecycle functions (also available as REST operations) are just part of the value add of Hawkular-Alerts.

You could do more complex things such as only trigger this alert if this Prometheus query generated results AND some other condition was true (say, ElasticSearch logs match a particular pattern, or if a Kafka topic had certain data). This demo merely scratches the surface, but does show how Hawkular Alerts can be used to work with OpenTracing to provide additional capabilities that may be found useful by system administrators and IT support personnel.