tag:blogger.com,1999:blog-69440501875682410592024-03-13T22:24:56.675-07:00Thoughts From A Management Platform DeveloperJohn Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.comBlogger86125tag:blogger.com,1999:blog-6944050187568241059.post-54418843448438675702017-09-20T15:56:00.001-07:002017-09-20T16:10:22.938-07:00Prometheus JMX Exporter Configuration For WildFly 10I'm currently looking at WildFly JMX MBeans, particular for exposing their attributes as metrics via the Prometheus JMX Exporter.<br />
<br />
Here's the JMX Exporter configuration I've tentatively come up with. Let me know in the comments if I've forgotten something or did something wrong.<br />
<br />
In short, this JMX Exporter config should collect the metrics for:<br />
<br />
* Deployment (EARs)<br />
* Subdeployment (WARs inside of EARs)<br />
* JMS Topics and Queues<br />
* Datasources<br />
* Transactions<br />
* Web Subsystem (i.e. Undertow)<br />
* Servlets<br />
* EJBs (Stateless, Stateful, Singleton, and Message Driven)<br />
<br />
It will also store server paths under the "wildfly_config" metric family with the label name identifying the path such as "jboss_home_dir" or "jboss_server_config_dir" and the label value being the path itself.<br />
<br />
And of course the default behavior of the JMX Exporter is to also emit metrics for the JVM statistics as well (such as heap and non-heap memory usage) so we get those "for free".<br />
<br />
The config is as follows: <br />
<br />
<pre>
---
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
# CONFIG
- pattern: "^jboss.as<path=(.+)><>path:(.+)"
attrNameSnakeCase: true
name: wildfly_config
labels:
$1: $2
value: 1
# DEPLOYMENTS
- pattern: "^jboss.as<deployment=(.+), subdeployment=(.+), subsystem=undertow><>(.+_sessions|session_.+):"
attrNameSnakeCase: true
name: wildfly_deployment_$3
type: GAUGE
labels:
name: $1
subdeployment: $2
- pattern: "^jboss.as<deployment=(.+), subsystem=undertow><>(.+_sessions|session_.+):"
attrNameSnakeCase: true
name: wildfly_deployment_$2
type: GAUGE
labels:
name: $1
# MESSAGING
- pattern: "^jboss.as<subsystem=messaging-activemq, server=(.+), jms-(queue|topic)=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_messaging_$4
type: GAUGE
labels:
server: $1
$2: $3
# DATASOURCES
- pattern: "^jboss.as<subsystem=datasources, (?:xa-)*data-source=(.+), statistics=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_datasource_$2_$3
type: GAUGE
labels:
name: $1
# TRANSACTIONS
- pattern: "^jboss.as<subsystem=transactions><>number_of_(.+):"
attrNameSnakeCase: true
name: wildfly_transactions_$1
type: GAUGE
# WEB SUBSYSTEM
- pattern: "^jboss.as<subsystem=undertow, server=(.+), (https?)-listener=(.+)><>(bytes_.+|error_count|processing_time|request_count):"
attrNameSnakeCase: true
name: wildfly_undertow_$4
type: GAUGE
labels:
server: $1
listener: $3
protocol: $2
# SERVLET
- pattern: "^jboss.as<deployment=(.+), subdeployment=(.+), subsystem=undertow, servlet=(.+)><>(.+_time|.+_count):"
attrNameSnakeCase: true
name: wildfly_servlet_$4
type: GAUGE
labels:
name: $3
deployment: $1
subdeployment: $2
- pattern: "^jboss.as<deployment=(.+), subsystem=undertow, servlet=(.+)><>(.+_time|.+_count):"
attrNameSnakeCase: true
name: wildfly_servlet_$3
type: GAUGE
labels:
name: $2
deployment: $1
# EJB
- pattern: "^jboss.as<deployment=(.+), subdeployment=(.+), subsystem=ejb3, (stateless-session|stateful-session|message-driven|singleton)-bean=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_ejb_$5
type: GAUGE
labels:
name: $4
type: $3
deployment: $1
subdeployment: $2
- pattern: "^jboss.as<deployment=(.+), subsystem=ejb3, (stateless-session|stateful-session|message-driven|singleton)-bean=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_ejb_$4
type: GAUGE
labels:
name: $3
type: $2
deployment: $1
</pre>
John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-78518693895423064102017-09-01T16:12:00.002-07:002017-09-01T16:12:47.933-07:00Hawkular Alerts with Jaeger / OpenTracingTwo recent blogs discuss how OpenTracing instrumentation can be used to collect application metrics:<br />
<br />
<ul>
<li><a href="http://www.hawkular.org/blog/2017/06/26/opentracing-appmetrics.html">http://www.hawkular.org/blog/2017/06/26/opentracing-appmetrics.html</a></li>
<li><a href="http://www.hawkular.org/blog/2017/08/opentracing-appmetrics-canary.html">http://www.hawkular.org/blog/2017/08/opentracing-appmetrics-canary.html</a></li>
</ul>
A further interesting integration can be the addition of Hawkular Alerts to the environment.<br />
<br />
As the previous <a href="http://www.hawkular.org/blog/2017/08/alerts-multiple-sources.html" target="_blank">blog</a> and <a href="https://www.youtube.com/watch?v=mM1mwJneKO4" target="_blank">demo</a> discuss, Hawkular Alerts is a generic, federated alerts system that can trigger events, alerts, and notifications from different, independent systems such as Prometheus, ElasticSearch, and Kafka.<br />
<br />
Here we can combine the two. Let's follow the directions for the OpenTracing demo (using the Jaeger implementation) and add Hawkular Alerts.<br />
<br />
What this can show is OpenTracing application metrics triggering alerts when (as in this example) OpenTracing spans encounter a larger-than-expected error rates.<br />
<br />
(Note: these instructions assume you are using Kubernetes / Minikube - see the Hawkular OpenTracing blogs linked above for more details on these instructions)<br />
<br />
1. START KUBERNETES<br />
<br />
Here we start minikube giving it enough resources to run all of the pods necessary for this demo. We also start up a browser pointing to the Kubernetes dashboard, so you can follow the progress of the remaining instructions.<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">minikube start --cpus 4 --memory 8192 </span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">minikube dashboard</span></li>
</ul>
2. DEPLOY PROMETHEUS<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml </span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">kubectl create -f https://raw.githubusercontent.com/objectiser/opentracing-prometheus-example/master/prometheus-kubernetes.yml</span></li>
<ul>
<li>(Note: The above command might not work depending on your version - if you get errors, download a copy of prometheus-kubernetes.yml and edit it, changing “v1alpha1” to “v1”)</li>
</ul>
</ul>
3. DEPLOY JAEGER <br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml </span></li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace;"></span>The following will build and deploy the Jaeger example code that will produce the OpenTracing data for the demo:<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">mkdir -p ${HOME}/opentracing ; cd ${HOME}/opentracing</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">git clone git@github.com:objectiser/opentracing-prometheus-example.git</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">cd opentracing-prometheus-example/simple</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">eval $(minikube docker-env)</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">mvn clean install docker:build</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">kubectl create -f services-kubernetes.yml</span></li>
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;"></span>(Note: The above command might not work depending on your version - if you get errors, edit services-kubernetes.yml and edit it, changing “v1alpha1” to “v1”)</li>
</ul>
</ul>
4. DEPLOY HAWKULAR-ALERTS AND CREATE ALERT TRIGGER<br />
<br />
The following will deploy Hawkular Alerts and create the trigger definition that will trigger an alert when the Jaeger OpenTracing data indicates an error rate that is over 30%<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">kubectl create -f https://raw.githubusercontent.com/hawkular/hawkular-alerts/master/dist/hawkular-alerts-k8s.yml</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: inherit;">Use “<span style="font-family: "courier new" , "courier" , monospace;">minikube service hawkular-alerts --url</span>” to determine the Hawkular Alerts URL and point your browser to the path “/hawkular/alerts/ui” at that URL (i.e. <span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">http://host:port/hawkular/alerts/ui</span>).</span></li>
<li><span style="font-family: inherit;">From the browser page running the Hawkular Alerts UI, enter a tenant name in the top right text field (“my-organization” for example) and click the “Change” button.</span></li>
<li><span style="font-family: inherit;">Navigate to the “Triggers” page (found in the left-hand nav menu).</span></li>
<li><span style="font-family: inherit;">Click the kabob menu icon at the top and select “New Trigger”.</span></li>
<li><span style="font-family: inherit;">In the text area, enter the following to define a new trigger that will trigger alerts when the Prometheus query shows that there is a 30% error rate or greater in the accountmgr or ordermgr servers:</span></li>
</ul>
<blockquote class="tr_bq">
<ul>
<li><span style="font-family: inherit;"> <span style="font-family: "courier new" , "courier" , monospace;">{</span></span><span style="font-family: "courier new" , "courier" , monospace;"><br /></span><span style="font-family: inherit;"><span style="font-family: "courier new" , "courier" , monospace;"> "trigger":{<br /> "id":"jaeger-prom-trigger",<br /> "name":"High Error Rate",<br /> "description":"Data indicates high error rate",<br /> "severity":"HIGH",<br /> "enabled":true,<br /> "autoDisable":false,<br /> "tags":{<br /> "prometheus":"Test"<br /> },<br /> "context":{<br /> "prometheus.url":"http://prometheus:9090"<br /> }<br /> },<br /> "conditions":[<br /> {<br /> "type":"EXTERNAL",<br /> "alerterId":"prometheus",<br /> "dataId":"prometheus-test",<br /> "expression":"(sum(increase(span_count{error=\"true\",span_kind=\"server\"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind=\"server\"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) > 0.3"<br /> }<br /> ]<br />}</span></span></li>
</ul>
</blockquote>
<ul>
<li><span style="font-family: inherit;">Now navigate back to the “Dashboard” page (again via the left-hand nav menu). From this Dashboard page, look for alerts when they are triggered. We'll next start generating the data that will trigger these alerts.</span></li>
</ul>
5. GENERATE SOME SAMPLE OPEN TRACING APPLICATION DATA<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">export ORDERMGR=$(minikube service ordermgr --url)</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">${HOME}/opentracing/opentracing-prometheus-example/simple/genorders.sh</span></li>
</ul>
Once the data starts to be collected, you will see alerts in the Hawkular Alerts UI as error rates become over 30% in the past minute (as per the Prometheus query).<br />
<br />
If you look at the alerts information in the Hawkular Alerts UI, you’ll see the conditions that triggered the alerts, such as:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Time: 2017-09-01 17:41:17 -0400<br />External[prometheus]: prometheus-test[Event [tenantId=my-organization, id=1a81471d-340d-4dba-abe9-5b991326dc80, ctime=1504302077288, category=prometheus, dataId=prometheus-test, dataSource=_none_, text=[1.504302077286E9, <b>0.3333333333333333</b>], context={<b>service=ordermgr, version=0.0.1</b>}, tags={}, trigger=null]] matches [(sum(increase(span_count{error="true",span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind) / sum(increase(span_count{span_kind="server"}[1m])) without (pod,instance,job,namespace,endpoint,transaction,error,operation,span_kind)) > 0.3]</span></blockquote>
Notice the “<b>ordermgr</b>” service (version "<b>0.0.1</b>") had an error rate of <b>0.3333</b> (33%) which caused the alert since it is above the allowed 30% threshold.<br />
<br />
At this point, the Hawkular Alerts UI provides the ability for system admins to log notes about the issue, acknowledge the alert and mark the alert resolved if the underlying issue has been fixed. These lifecycle functions (also available as REST operations) are just part of the value add of Hawkular-Alerts.<br />
<br />
You could do more complex things such as only trigger this alert if this Prometheus query generated results AND some other condition was true (say, ElasticSearch logs match a particular pattern, or if a Kafka topic had certain data). This demo merely scratches the surface, but does show how Hawkular Alerts can be used to work with OpenTracing to provide additional capabilities that may be found useful by system administrators and IT support personnel.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-15788360863024740202017-08-11T13:01:00.000-07:002017-08-11T13:01:27.318-07:00Hawkular Alerts with Prometheus, ElasticSearch, Kafka<div style="-webkit-text-stroke-width: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<h2 id="user-content-federated-alerts" style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; font-size: 1.5em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#federated-alerts" id="user-content-federated-alerts" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Federated Alerts</h2>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Hawkular Alerts aims to be a federated alerting system. That is to say, it can fire alerts and send notifications that are triggered by data coming from a number of third-party external systems.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Thus, Hawkular Alerts is more than just an alerting system for use with Hawkular Metrics. In fact, Hawkular Alerts can be used independently of Hawkular Metrics. This means you do not even have to be using Hawkular Metrics to take advantage of the functionality provided by Hawkular Alerts.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
This is a key differentiator between Hawkular Alerts and other alerting systems. Most alerting systems only alert on data coming from their respective storage systems (e.g. the Prometheus Alert Engine alerts only on Prometheus data). Hawkular Alerts, on the other hand, can trigger alerts based on data from various systems.</div>
</div>
<div style="box-sizing: border-box;">
<h3 id="user-content-alerts-vs-events" style="box-sizing: border-box; font-size: 1.25em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#alerts-vs-events" id="user-content-alerts-vs-events" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Alerts vs. Events</h3>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Before we begin, a quick clarification is in order. When it is said that Hawkular Alerts fires an "alert" it means some data came into Hawkular Alerts that matched some conditions which triggered the creation of an alert in Hawkular Alerts backend storage (which can then trigger additional actions such as sending emails or calling a webhook). An "alert" typically refers to a problem that has been detected, and someone should take action to fix it. An alert has a lifecycle attached to it - alerts are <i style="box-sizing: border-box;">opened</i>, then <i style="box-sizing: border-box;">acknowledged</i> by some user who will hopefully fix the problem, then <i style="box-sizing: border-box;">resolved</i> when the problem can be considered closed.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
However, there can be conditions that occur that do not represent problems but nevertheless are events you want recorded. There is no lifecycle associated with events and no additional actions are triggered by events, but "events" are fired by Hawkular Alerts in the same general manner as "alerts" are.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
In this document, when it is said that Hawkular Alerts can fire "alerts" based on data coming from external third-party systems such as Prometheus, ElasticSearch, and Kakfa, this also means events can be fired as well as alerts. What this means is you can record any event (not just a "problem", aka "alert") that can be gleaned from this data coming from external third-party systems.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
See <a href="http://www.hawkular.org/community/docs/developer-guide/alerts.html#_alerting_philosophy" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration: none;">alerting philosophy</a> for more.</div>
</div>
</div>
</div>
</div>
<div style="-webkit-text-stroke-width: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<h2 id="user-content-demo" style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; font-size: 1.5em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#demo" id="user-content-demo" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Demo</h2>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
There is a <a href="https://youtu.be/mM1mwJneKO4" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration: none;">recorded demo found here</a> that will illustrate what this document is describing. After you read this document, you should watch the demo to gain further clarity on what is being explained. The demo is the <code style="background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">multiple-sources</code> example which you can run yourself found <a href="https://github.com/hawkular/hawkular-alerts/tree/master/examples/multiple-sources" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration: none;">here</a> (note: at the time of writing, this example is only found in the <a href="https://github.com/hawkular/hawkular-alerts/tree/next/examples/multiple-sources" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration: none;">next branch</a>, to be merged in master soon).</div>
</div>
</div>
</div>
<div style="-webkit-text-stroke-width: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<h2 id="user-content-prometheus" style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; font-size: 1.5em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#prometheus" id="user-content-prometheus" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Prometheus</h2>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Hawkular Alerts can take the results of Prometheus metric queries and use the queried data for triggers that can fire alerts.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
This Hawkular Alerts trigger will fire an alert (and send an email) when a Prometheus metric indicates our store’s inventory of widgets is consistently low (as defined by the Prometheus query you see in the "expression" field of the condition):</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div class="highlight highlight-source-json" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>trigger<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>id<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>low-stock-prometheus-trigger<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>name<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Low Stock<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>description<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>The number of widgets in stock is consistently low.<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>severity<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>MEDIUM<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>enabled<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">true</span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>tags<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: {
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>prometheus<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Prometheus<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actions<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:[
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actionPlugin<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>email<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actionId<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>email-notify-owner<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
}
]
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>conditions<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:[
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>type<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>EXTERNAL<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>alerterId<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>prometheus<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>dataId<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>prometheus-dataid<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>expression<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>rate(products_in_inventory{product=<span class="pl-cce" style="box-sizing: border-box;">\"</span>widget<span class="pl-cce" style="box-sizing: border-box;">\"</span>}[30s])<2 class="pl-pds" span="" style="box-sizing: border-box; color: #032f62;">"
}
]</2></span></pre>
</div>
</div>
</div>
</div>
</div>
<br />
<div style="box-sizing: border-box;">
<h3 id="user-content-integration-with-prometheus-alert-engine" style="box-sizing: border-box; font-size: 1.25em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#integration-with-prometheus-alert-engine" id="user-content-integration-with-prometheus-alert-engine" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"></svg></a>Integration with Prometheus Alert Engine</h3>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
As a side note, though not demostrated in the example, Hawkular Alerts also has an integration with Prometheus' own Alert Engine. This means the alerts generated by Prometheus itself can be forward to Hawkular Alerts which can, in turn, be used for additional processing, perhaps for use with data that is unavailable to Prometheus that can tell Hawkular Alerts to fire other alerts. For example, Hawkular Alerts can take Prometheus alerts as input and feed it back into other conditions that trigger on the Prometheus alert along with ElasticSearch logs.</div>
</div>
</div>
<div style="-webkit-text-stroke-width: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<h2 id="user-content-elasticsearch" style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; font-size: 1.5em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#elasticsearch" id="user-content-elasticsearch" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>ElasticSearch</h2>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Hawkular Alerts can examine logs stored in ElasticSearch and trigger alerts based on patterns that match within the ElasticSearch log messages.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
This Hawkular Alerts trigger will fire an alert (and send an email) when ElasticSearch logs indicate sales are being lost due to inventory being out of stock of items (as defined by the condition which looks for a log category of "FATAL" which happens to mean a lost sale in the case of the store’s logs). Notice dampening is enabled on this trigger - this alert will only fire when the logs indicate lost sales every 3 times.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div class="highlight highlight-source-json" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>trigger<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>id<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>lost-sale-elasticsearch-trigger<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>name<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Lost Sale<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>description<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>A sale was lost due to inventory out of stock.<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>severity<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>CRITICAL<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>enabled<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">true</span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>tags<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: {
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Elasticsearch<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Localhost instance<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>context<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: {
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>timestamp<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>@timestamp<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>filter<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>{<span class="pl-cce" style="box-sizing: border-box;">\"</span>match<span class="pl-cce" style="box-sizing: border-box;">\"</span>:{<span class="pl-cce" style="box-sizing: border-box;">\"</span>category<span class="pl-cce" style="box-sizing: border-box;">\"</span>:<span class="pl-cce" style="box-sizing: border-box;">\"</span>inventory<span class="pl-cce" style="box-sizing: border-box;">\"</span>}}<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>interval<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>10s<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>index<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>store<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>mapping<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>level:category,@timestamp:ctime,message:text,category:dataId,index:tags<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actions<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:[
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actionPlugin<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>email<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actionId<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>email-notify-owner<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
}
]
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>dampenings<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: [
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>triggerMode<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>FIRING<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>type<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>STRICT<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>evalTrueSetting<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>
}
],
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>conditions<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:[
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>type<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>EVENT<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>dataId<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>inventory<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>expression<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>category == 'FATAL'<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
}
]</pre>
</div>
</div>
</div>
</div>
</div>
<div style="-webkit-text-stroke-width: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<h2 id="user-content-kafka" style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; font-size: 1.5em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#kafka" id="user-content-kafka" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Kafka</h2>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Hawkular Alerts can examine data retrieved from Kafka message streams and trigger alerts based that Kafka data.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
This Hawkular Alerts trigger will fire an alert when data over a Kakfa topic indicates a large purchase was made to fill the store’s inventory (as defined by the condition which evaluates to true when any number over 17 is received on the Kafka topic):</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div class="highlight highlight-source-json" style="box-sizing: border-box; margin-bottom: 16px;">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;"><span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>trigger<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>id<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>large-inventory-purchase-kafka-trigger<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>name<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Large Inventory Purchase<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>description<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>A large purchase was made to restock inventory.<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>severity<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>LOW<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>enabled<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">true</span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>tags<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: {
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Kafka<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>Localhost instance<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>context<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: {
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>topic<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>store<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>kafka.bootstrap.servers<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>localhost:9092<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>kafka.group.id<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>hawkular-alerting<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>actions<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:[ ]
},
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>conditions<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>:[
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>type<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>THRESHOLD<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>dataId<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>store<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>operator<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>GT<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span>threshold<span class="pl-pds" style="box-sizing: border-box; color: #032f62;">"</span></span>: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">17</span>
}
]</pre>
</div>
</div>
</div>
</div>
</div>
<div style="-webkit-text-stroke-width: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; margin-bottom: 0px !important; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<h2 id="user-content-but-wait-there-s-more" style="border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; font-size: 1.5em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#but-wait-theres-more" id="user-content-but-wait-theres-more" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>But, Wait! There’s More!</h2>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
The above only mentions the different ways Hawkular Metrics retrieves data for use in determining what alerts to fire. What is not covered here is the fact that Hawkular Alerts can stream data in the other direction as well - Hawkular Alerts can send alert and event data <i style="box-sizing: border-box;">to</i> things like an ElasticSearch server or a Kafka broker. There are additional examples (mentioned below) that can demonstrate this capability.</div>
</div>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
The point is Hawkular Alerts should be seen as a shared, common alerting engine that can be shared for use by multiple third-party systems and can be used as both a consumer and producer - as a consumer of the data from external third-party systems (which is used to fire alerts and events) and as a producer to send notifications of alerts and events to external third-party systems.</div>
</div>
<div style="box-sizing: border-box;">
<h3 id="user-content-more-examples" style="box-sizing: border-box; font-size: 1.25em; font-weight: 600; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">
<a class="anchor" href="https://github.com/jmazzitelli/hawkular.github.io/blob/25f6037477c20aded1fb975c2bcfd0a10fb36487/src/main/jbake/content/blog/2017/08/alerts-multiple-sources.adoc#more-examples" id="user-content-more-examples" style="background-color: transparent; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg class="octicon octicon-link" height="16" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>More Examples</h3>
<div style="box-sizing: border-box;">
<div style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;">
Take a look at the Hawkular Alerts <a href="https://github.com/hawkular/hawkular-alerts/tree/master/examples" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration: none;">examples</a> for more examples on using external systems as data to be used for triggering alerts. (note: at the time of writing, some examples are currently in the <a href="https://github.com/hawkular/hawkular-alerts/tree/next/examples" style="background-color: transparent; box-sizing: border-box; color: #0366d6; text-decoration: none;">next branch</a> such as the Kafka ones).</div>
</div>
</div>
</div>
</div>
John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com1tag:blogger.com,1999:blog-6944050187568241059.post-78121252602917669442017-08-01T11:23:00.001-07:002017-08-01T11:23:35.889-07:00Hawkular Alerts 2.0 UI WIP<h2>
Hawkular Alerts 2.0 UI</h2>
A quick 10-minute demo has been published to illustrate the progress that was made by the hAlerts team on the new UI.<br />
<br />
This is a work-in-progress, and things will change, but the UI is actually functional now.<br />
<br />
It is best the video is viewed in full screen mode. The video link is: <a href="https://www.youtube.com/watch?v=bb9SaJudPlU">https://www.youtube.com/watch?v=bb9SaJudPlU</a><br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/bb9SaJudPlU/0.jpg" src="https://www.youtube.com/embed/bb9SaJudPlU?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-75428843938528393162016-11-21T11:52:00.005-08:002016-11-21T11:52:55.874-08:00Hawkular OpenShift Demo - Running Outside OpenShiftBelow is a quick 8 minute <a href="https://www.youtube.com/watch?v=Fj_OriyvMc0" target="_blank">demo</a> of the Hawkular OpenShift Agent.<br />
<br />
For more information, see: <a href="https://github.com/hawkular/hawkular-openshift-agent">https://github.com/hawkular/hawkular-openshift-agent</a> <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/Fj_OriyvMc0/0.jpg" src="https://www.youtube.com/embed/Fj_OriyvMc0?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-36107413111805014312016-11-14T13:36:00.001-08:002016-11-14T13:36:45.264-08:00Hawkular OpenShift Agent - First DemoBelow is a quick 10 minute <a href="https://www.youtube.com/watch?v=jvOPlz7lzyM" target="_blank">demo</a> of the Hawkular OpenShift Agent.<br />
<br />
For more information, see: <a href="https://github.com/hawkular/hawkular-openshift-agent">https://github.com/hawkular/hawkular-openshift-agent</a><br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/jvOPlz7lzyM/0.jpg" src="https://www.youtube.com/embed/jvOPlz7lzyM?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-55361034280584827252016-10-20T13:05:00.000-07:002016-10-20T13:05:12.022-07:00Hawkular OpenShift Agent is BornA new Hawkular agent has been published on github.com - <a href="https://github.com/hawkular/hawkular-openshift-agent" target="_blank">Hawkular OpenShift Agent</a>. <br />
<br />
It is implemented in Go and the main use case for which it was created is to be able to collect metrics from OpenShift pods. The idea is you run Hawkular OpenShift Agent (HOSA) on an OpenShift node and HOSA will listen for pods to come up and down on the node. As pods come online, the pods will tell the agent what (if any) metrics should be collected. As pods go down, the agent will stop collecting metrics from all endpoints running on that pod.<br />
<br />
Today, only Prometheus endpoints (using either the binary or text protocol) can be scraped with Jolokia endpoints next on the list to be implemented. So HOSA will be able to support collecting metrics from either type of endpoint in the near future.<br />
<br />
For more information - how to build and configure it - refer to the <a href="https://github.com/hawkular/hawkular-openshift-agent/blob/master/README.adoc" target="_blank">Hawkular OpenShift Agent README</a>.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-69877690096952115642016-10-17T09:12:00.001-07:002016-10-17T09:13:22.250-07:00Pulling in a Go Dependency From a Fork, Branch, or Github PR using GlideWhile writing a Go app, I decided to use <a href="https://glide.sh/" target="_blank">Glide</a> as the dependency management system (I tried Godep first, but even on the first day of using it, my dependencies were getting screwed up, lost, builds would mysteriously break - so I decided to switch to Glide, which seems much better).<br />
<br />
I was using the Hawkular Go Client library because I needed to write metric data to Hawkular Metrics. So in my glide.yaml, I had this:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">- package: github.com/hawkular/hawkular-client-go<br /> subpackages:<br /> - metrics</span></blockquote>
Which simply tells Glide that I want to use the latest master of the client library (I'm not using a versioned library yet. I guess I should start doing that).<br />
<br />
Anyway, I needed to add a <a href="https://github.com/hawkular/hawkular-client-go/issues/8" target="_blank">feature</a> to the Hawkular Go Client. So I forked the git repository, created a branch in my fork where I implemented the new feature, and submitted a Github pull request from my own branch. Rather than wait for the PR to be merged, I wanted Glide to pull in my own branch in my forked repo so I could immediately begin using the new feature. It was as simple as adding three lines to my glide.yaml and running "glide update":<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">- package: github.com/hawkular/hawkular-client-go<br /><b> repo: git@github.com:jmazzitelli/hawkular-client-go.git<br /> vcs: git<br /> ref: issue-8</b><br /> subpackages:<br /> - metrics</span></blockquote>
This tells Glide that the Hawkular Go client package is now located at a different repository (my fork located at github.com) under a branch called "issue-8".<br />
<br />
Running "glide update" pulled in the Hawkuar Go client from my fork's branch and placed it in my vendors/ directory. I can now start using my new feature in my Go app without waiting for the PR to be merged. Once the PR is merged, I can remove those three lines, "glide update" again, and things should be back to normal.<br />
<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-60269799638038642452016-10-05T21:04:00.000-07:002016-10-06T06:57:12.041-07:00Installing Open Shift Origin and Go for a Development EnvironmentThe Hawkular team is doing some work to develop a Hawkular Agent for running within an Open Shift environment. Since the Go Programming Language ("Go") seems to be the language de jour and many things within the Open Shift infrastructure is developed in Go, the first attempt at this new Hawkular Agent will also be implemented in Go.<br />
<br />
Because of this, I needed to get a development environment up and running that included both Open Shift and Go. This blog is simply my notes on how I did this.<br />
<br />
<span style="font-size: x-large;">INSTALL "GO"</span><br />
<br />
Go isn’t necessary to install and run Open Shift, but because I want to write a Go application, I need it.<br />
<br />
<span style="font-size: large;">INSTALL CORE GO SYSTEM</span><br />
<br />
* Create the directory where Go is to be installed<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> mkdir $HOME/bin/go-install</span> (this is where Go will be installed)<br />
<br />
* Create the Go workspace (GOPATH will end up pointing to here)<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> mkdir $HOME/source/go</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> mkdir $HOME/source/go/bin</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> mkdir $HOME/source/go/pkg</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> mkdir $HOME/source/go/src</span><br />
<br />
* Download go package from <a href="https://golang.org/dl/" target="_blank">https://golang.org/dl/ </a><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd /tmp</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz</span><br />
<br />
* Unpack the tar in the $HOME/bin/go-install - should end up inside a subdirectory "go"<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd $HOME/bin/go-install</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> tar xzvf /tmp/go*.tar.gz</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
* Set up your shell environment so you can run Go<br />
** Put the following Inside .bashrc<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> export GOROOT=${HOME}/bin/go-install/go</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> export PATH=${PATH}:${GOROOT}/bin</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> export GOPATH=${HOME}/source/go</span><br />
<br />
* Make sure Go is working before going on - run “<span style="font-family: "courier new" , "courier" , monospace;">go version</span>” to confirm. I like to log completely out and then back in again so my bash script gets loaded for all my shells.<br />
<br />
<span style="font-size: large;">INSTALL ADDITIONAL GO TOOLS</span><br />
<span style="font-size: large;"></span><br />
* Download and install guru<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd $HOME/bin/go-install/go/bin<br /> go get golang.org/x/tools/cmd/guru<br /> go build golang.org/x/tools/cmd/guru</span><br />
<br />
** The above puts the "guru" executable in your current directory (which should be $HOME/bin/go-install/go/bin)<br />
<br />
* Download and install gocode<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd $HOME/bin/go-install/go/bin<br /> go get github.com/nsf/gocode<br /> go build github.com/nsf/gocode</span><br />
<br />
** The above puts the "gocode" executable in your current directory (which should be $HOME/bin/go-install/go/bin)<br />
<br />
* Download and install godef<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd $HOME/bin/go-install/go/bin<br /> go get github.com/rogpeppe/godef<br /> go build github.com/rogpeppe/godef</span><br />
<br />
** The above puts the "godef" executable in your current directory (which should be $HOME/bin/go-install/go/bin)<br />
<br />
* Download and install godep<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd $HOME/bin/go-install/go/bin<br /> go get github.com/tools/godep<br /> go build github.com/tools/godep</span><br />
<br />
** The above puts the "godep" executable in your current directory (which should be $HOME/bin/go-install/go/bin)<br />
<br />
* If you use Eclipse, install <a href="https://github.com/GoClipse/goclipse/tree/latest/documentation" target="_blank">GoClipse</a>:<br />
** Plugin Site: <a href="http://goclipse.github.io/releases/">http://goclipse.github.io/releases/</a><br />
** Make sure the GoClipse configuration knows where your Go installation is along with your guru, gocode, and godef executables. See the Go configuration settings in Eclipse preferences.<br />
<br />
<span style="font-size: x-large;">INSTALL OPEN SHIFT</span><br />
<br />
These instructions will install Open Shift in a virtual machine. So you need to get Vagrant and VirtualBox first before doing anything. And of course Vagrant needs Ruby 2 so you need that before anything. Then you install the Open Shift image.<br />
<br />
Note that you may have to change your BIOS settings to enable virtualization. To avoid a VERR_VMX_MSR_ALL_VMX_DISABLED error, I had to do <a href="http://amiduos.com/support/knowledge-base/article/enabling-virtualization-in-lenovo-systems" target="_blank">this</a> on my Lenovo laptop in the Security->Virtualization section of the BIOS settings. <br />
<br />
These notes follow the instructions found at <a href="https://www.openshift.org/vm">https://www.openshift.org/vm</a><br />
<br />
<span style="font-size: x-small;">(October 5, 2016: Note that the instructions there tell you to not use Vagrant 1.8.5 nor VirtualBox 5.1 - unfortunately, they tell you this all the way at the bottom <i>after</i> the instructions. So if you start at the top and work your way down, you will be frustrated beyond belief until you decide to skip all the way to the bottom and realize this. I emailed the folks maintaining that page to put those notices at the top so people can see these warnings before they start downloading the wrong stuff.)</span><br />
<br />
<span style="font-size: large;">INSTALL RUBY </span><br />
<br />
If you already have Ruby 2+ installed, you can skip this. This will install RVM and then use that to install Ruby 2. If you skip this step, make sure you have a Ruby 2 installation available before going on to installing and running Vagrant.<br />
<br />
* Download and install RVM and Ruby<br />
** You need to accept the GPG key for RVM and install RVM <br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> gpg2 --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> curl -L https://get.rvm.io | bash -s stable</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">rvm autolibs packages</span> </span><br />
<br />
** Now install Ruby 2<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> rvm install 2.3.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> rvm use --default 2.3.0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ruby -v</span><br />
<br />
<span style="font-size: large;">INSTALL VAGRANT</span><br />
<br />
* Download and install Vagrant .rpm<br />
** <a href="https://releases.hashicorp.com/vagrant/">https://releases.hashicorp.com/vagrant/</a><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> cd /tmp<br /> wget https://releases.hashicorp.com/vagrant/1.8.4/vagrant_1.8.4_x86_64.rpm</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> rpm -i </span><span style="font-family: "courier new" , "courier" , monospace;">vagrant_*.rpm</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant version </span><br />
<br />
<span style="font-size: large;">INSTALL VIRTUAL BOX</span><br />
<br />
* Download and install Virtual Box<br />
** <a href="https://www.virtualbox.org/wiki/Downloads">https://www.virtualbox.org/wiki/Downloads</a><br />
** <a href="https://www.virtualbox.org/wiki/Download_Old_Builds">https://www.virtualbox.org/wiki/Download_Old_Builds</a> <br />
** You must grab it from the Oracle yum repo - we first must add that repo to our system:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> sudo wget -P </span></span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">/etc/yum.repos.d </span></span>http://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo</span></span><br />
<br />
** If anything goes wrong during the install, you'll want to update your system and reboot - so you may want to do this now:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> dnf update</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
** Make sure the kernel version is expected - these should output the same version string - otherwise, reboot<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> rpm -qa kernel | sort -V | tail -n 1</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> uname -r</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
** Install VirtualBox now<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> dnf install VirtualBox-5.0</span></span><br />
<br />
I ran into some problems when I tried to run VirtualBox before I realized I needed to download VirtualBox from the Oracle yum repo. If you download from, say, RPMFusion, these instructions may not work (they did not for me). If you run into problems, you might have to install some additional packages via dnf (such as kernel-devel, kernel-headers, dkms) and perform some additional magic which I do not know which explains why I received a Dreadful grade on my O.W.L. exam.<br />
<br />
<span style="font-size: large;">INSTALL THE OPEN SHIFT IMAGE</span><br />
<br />
* Go to an empty directory where you want to prepare your Vagrantfile<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> mkdir ${HOME}/openshift</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd ${HOME}/openshift</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
* Create a Vagrant file that initializes the OpenShift image.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant init openshift/origin-all-in-one</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
* Run Open Shift inside a VM within VirtualBox<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant up --provider=virtualbox</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
At this point you should have an OpenShift environment running. If any errors occur, a log message should tell you how you can proceed.<br />
<br />
You will want to download the Open Shift command line client "oc". Rather than regurgitate the instructions here, simply log into Open Shift using the default "admin" user (password "admin") and go to <a href="https://10.2.2.2:8443/console/command-line">https://10.2.2.2:8443/console/command-line</a> and follow the instructions there to download, install, and use "oc".<br />
<br />
Note that you can SSH into your VM by executing "<span style="font-family: "courier new" , "courier" , monospace;">vagrant ssh</span>"<br />
<br />
The Open Shift self-signed certificate found at "<span style="font-family: "courier new" , "courier" , monospace;">/var/lib/origin/openshift.local.config/master/ca.crt</span>" can be used to authenticate clients.<br />
<br />
<span style="font-size: large;">UPGRADE OPEN SHIFT</span><br />
<br />
Should you wish to upgrade the Open Shift VM in the future, the following steps should do it.<br />
<br />
* Go to the directory where you ran the VM:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd ${HOME}/openshift</span><br />
<br />
* Update the image<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant box update --box openshift/origin-all-in-one</span><br />
<br />
* Destroy and re-create the VM environment<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> vagrant destroy --force</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant up --provider=virtualbox</span><br />
<br />
<span style="font-size: large;">REMOVE OPEN SHIFT</span><br />
<br />
Should you wish to remove the Open Shift VM in the future, the following steps should do it.<br />
<br />
* Go to the directory where your Vagrantfile is<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> cd ${HOME}/openshift</span><br />
<br />
* Remove Open Shift VM<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant halt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant destroy --force</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> vagrant box remove --force openshift/origin-all-in-one</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"></span>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2tag:blogger.com,1999:blog-6944050187568241059.post-55770683958561018492016-07-12T14:28:00.000-07:002016-07-13T08:05:47.568-07:00Collecting Prometheus Data and Storing in HawkularMy last blog entry talked about how to <a href="http://management-platform.blogspot.com/2016/07/collecting-jmx-data-and-storing-in.html" target="_blank">collect JMX data via Jolokia and store that data in Hawkular</a>. Another relatively unknown feature similar to that, which I will now describe in this blog entry, is the ability to collect Prometheus metric data and store that in <a href="http://www.hawkular.org/" target="_blank">Hawkular</a>.<br />
<br />
<a href="https://prometheus.io/" target="_blank">Prometheus</a> is itself a metric collection and storage system. However, Prometheus data endpoints (that is, endpoints that emit Prometheus metric data) follow <a href="https://prometheus.io/docs/instrumenting/exposition_formats/" target="_blank">a specific format</a> for the emitted metric data. The Hawkular WildFly Agent has the ability to parse this Prometheus metric data format and push that metric data for storage into Hawkular which can then, of course, be used by Hawkular and its clients (for the purposes of graphing the metric data, alerting on the metric data, etc.).<br />
<br />
I will explain how you can quickly get Hawkular to collect metric data from any Prometheus endpoint and store that data into Hawkular.<br />
<br />
First, you need a Prometheus endpoint that emits metric data! For this simple demo, I simply ran the Prometheus Server which itself emits metrics. I did this via docker:<br />
<br />
1. Make sure your docker daemon is running: <span style="font-family: "courier new" , "courier" , monospace;">sudo docker daemon</span><br />
2. Run the Prometheus Server docker image: <span style="font-family: "courier new" , "courier" , monospace;">sudo docker run -p 9090:9090 prom/prometheus</span><br />
<br />
That's it! You now have a Prometheus Server running and listening on port 9090. To see the metrics it, itself, emits, go to http://localhost:9090/metrics. We'll be asking the Hawkular WildFly Agent to collect these metrics and push them into a Hawkular Server.<br />
<br />
Second, you need to run a Hawkular Server. I won't go into details here on how to do that. Suffice it to say, either <a href="https://github.com/hawkular/hawkular-services" target="_blank">build</a> or <a href="http://www.hawkular.org/downloads.html" target="_blank">download</a> a Hawkular Server distribution and run it (if it is not a developer build, make sure you run your external Cassandra Server prior to starting your Hawkular Server - e.g. <span style="font-family: "Courier New",Courier,monospace;">sudo docker run cassandra -p 9042:9042</span>).<br />
<br />
Now you want to run a Hawkular WildFly Agent to collect some of that Prometheus metric data and store it in the Hawkular Server. In this demo, I'll be running the Swarm Agent, which is simply a Hawkular WildFly Agent packaged in a single jar that you can run as a standalone app. However, its agent subsystem configuration is the same as if you were running the agent in a full WildFly Server so the configuration I'll be describing can be used no matter how you have deployed your agent.<br />
<br />
Rather than rely on <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-swarm-agents/hawkular-swarm-agent-dists/hawkular-swarm-agent-dist/src/main/resources/hawkular-swarm-agent-config.xml" target="_blank">the default configuration file that comes with the Swarm Agent</a> I extracted the default configuration file and edited it as I describe below.<br />
<br />
I deleted all the "-dmr" related configuration settings (metric-dmr, resource-type-dmr, remote-dmr, etc). I want my agent to only collect data from my Prometheus endpoint, so no need to define all these DMR metadata. (NOTE: the Swarm Agent configuration file does already, out-of-box, support Prometheus. I will skip that for this demo - I want to explicitly explain the parts of the agent configuration that is needed to be in the configuration file).<br />
<br />
The Prometheus portion of the agent configuration is very small. Here it is:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><managed-servers><br /> <remote-prometheus name="My Prometheus Endpoint"<br /> enabled="true"<br /> interval="30"<br /> time-units="seconds"<br /> metric-tags="feed=%FeedId"<br /> metric-id-template="%FeedId_%MetricName"<br /> url="http://127.0.0.1:9090/metrics"/><br /></managed-servers></span></blockquote>
That's it. <remote-prometheus> tells the agent where the Prometheus endpoint is and how often to pull the metric data from it. Every metric emitted by that Prometheus endpoint will be collected and stored in Hawkular.<br />
<br />
Notice I can associate my managed server with metric tags (remote-prometheus is one type of "managed server"). For every metric that is collected for this remote-prometheus managed server, those tags will be added to those metrics in the Hawkular Server (specifically in the Hawkular Metrics component). All metrics will have these same tags. In addition, any labels associated with the emitted Prometheus metrics (Prometheus metric data can have name/value pairs associated with them - Prometheus calls these labels) will be added as Hawkular tags. Similarly, the ID used to store the metrics in the Hawkular Metric component can also be customized. Both metric-tags and metric-id-template are optional. You can also place those attributes on individual metric definitions (which I describe below) which is most useful if you have specific tags you want to add only to metrics of a specific metric type but not on all of the metrics collected for your managed server.<br />
<br />
If you wish to have the agent only collect a subset of the metrics emitted by that endpoint, then you must tell the agent which metrics you want collected. You do this via metric sets:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><metric-set-prometheus name="My Prometheus Metrics"><br /> <metric-prometheus name="http_requests_total" /><br /> <metric-prometheus name="go_memstats_heap_alloc_bytes" /><br /> <metric-prometheus name="process_resident_memory_bytes" /><br /></metric-set-prometheus></span></blockquote>
Once you defined your metric-set-prometheus entries, you specify them in the metric-sets attribute on your <remote-prometheus> element (e.g. metric-sets="My Prometheus Metrics").<br />
<br />
OK, now that I've got my Swarm Agent configuration in place (call it
"agent.xml" for simplicity), I can run the agent and point it to my
configuration file:<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">java -jar hawkular-swarm-agent-dist-*-swarm.jar agent.xml </span></blockquote>
At this point I have my agent running along side of my Hawkular Server
and the Prometheus Server which is the source of our metric data. The agent is collecting
information from the Prometheus endpoint and pushing the collected data to Hawkular
Server.<br />
<br />
In order to visualize this collected data, I'm using the experimental developer tool <a href="https://github.com/pilhuhn/hawkfx" target="_blank">HawkFX</a>.
This is simply a browser that let's you see what data is in Hawkular
Inventory as well as Hawkular Metrics. When I log in, I can see all the metric data that comes directly from the Prometheus endpoint.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiELwGFbFrmW1UGBxzV1FMGC32czVa7KVV_nbr21XVmiDClVEKzTxvGB9_H732RSe-nIcsurR0ZXSSgHXCDA2vEofHnmbmLIbep3eaCyTYxK81Pjyd3vS57JlBxDiKBtAZPsStKdyUdFuGP/s1600/big-metric-list.png" imageanchor="1"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiELwGFbFrmW1UGBxzV1FMGC32czVa7KVV_nbr21XVmiDClVEKzTxvGB9_H732RSe-nIcsurR0ZXSSgHXCDA2vEofHnmbmLIbep3eaCyTYxK81Pjyd3vS57JlBxDiKBtAZPsStKdyUdFuGP/s320/big-metric-list.png" width="234" /></a><br />
<br />
I can select a metric to see its graph:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy_w7Tvu2nmsWuS6cRBgbf2DW4uEg2sp_hJR9hLldkW8Q_WsULX2UzCDJY9dqz3x8EXCgNBsWBTqFg1_9jtY0ssOqLxH2Hl3gt6EMeN2Vpw7m38-65cuF6GZkph6GHDsd2sSXYJ-WUoYi2/s1600/big-metric-list-graph.png" imageanchor="1"><img border="0" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy_w7Tvu2nmsWuS6cRBgbf2DW4uEg2sp_hJR9hLldkW8Q_WsULX2UzCDJY9dqz3x8EXCgNBsWBTqFg1_9jtY0ssOqLxH2Hl3gt6EMeN2Vpw7m38-65cuF6GZkph6GHDsd2sSXYJ-WUoYi2/s320/big-metric-list-graph.png" width="320" /></a><br />
<br />
If I were to have configured my agent to only collect a subset of metrics (as I had shown earlier), I would see only those metrics that I asked to collect - all the other metrics emitted by the Prometheus endpoint are ignored:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4hAxwtZo9Uln0YYY0Ulu1-i6UV4JguGHesXhzGVSRgXfBa7ovVMU6IIf2wwyLqEV11QbeskZSTafUagABwbdDz3U9VQ42egnSi5zWFVAO5e8us3Zk3gBB1ijZCxT07fuaqJsww8zhXF8J/s1600/small-metric-list.png" imageanchor="1"><img border="0" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4hAxwtZo9Uln0YYY0Ulu1-i6UV4JguGHesXhzGVSRgXfBa7ovVMU6IIf2wwyLqEV11QbeskZSTafUagABwbdDz3U9VQ42egnSi5zWFVAO5e8us3Zk3gBB1ijZCxT07fuaqJsww8zhXF8J/s320/small-metric-list.png" width="320" /></a><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTWPwPP8JwRmfkvKS6DlvQhcgIb_IiSWJvFwd0IgOfPX0NCjk8GWzz0_qWkP81KNq19Uej7h2OIoTD-ZT9BHBkxthkRbp3XAIATuu09JjvJzPvoJYLxjIw4jVoHgFaQMM6ofJcJIzfEne8/s1600/small-metric-list-graph.png" imageanchor="1"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTWPwPP8JwRmfkvKS6DlvQhcgIb_IiSWJvFwd0IgOfPX0NCjk8GWzz0_qWkP81KNq19Uej7h2OIoTD-ZT9BHBkxthkRbp3XAIATuu09JjvJzPvoJYLxjIw4jVoHgFaQMM6ofJcJIzfEne8/s320/small-metric-list-graph.png" width="320" /></a><br />
<br />
What this all shows is that you can use Hawkular WildFly Agent to collect metric data from a Prometheus endpoint and store that data inside Hawkular.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-66100385290426887422016-07-12T12:14:00.000-07:002016-07-13T08:06:14.571-07:00Collecting JMX Data and Storing in HawkularThe <a href="https://github.com/hawkular/hawkular-agent" target="_blank">Hawkular WildFly Agent</a> has the ability to not only monitor WildFly Servers but also JMX MBean Servers via <a href="https://jolokia.org/" target="_blank"> Jolokia</a> (and also Prometheus endpoints, but that's for another blog - let's focus on Jolokia-enabled JMX MBean Servers for now).<br />
<br />
What this means is if you have a Jolokia-enabled server, you can collect JMX data from it and store that data in <a href="http://www.hawkular.org/" target="_blank">Hawkular</a>. This includes both metric data as well as resource information.<br />
<br />
This blog will attempt to quickly show how this is done.<br />
<br />
First, you need a Jolokia-enabled server! For this demo, here's the quick steps I did to get this running on my box:<br />
<br />
1. I <a href="http://wildfly.org/downloads/" target="_blank">downloaded WildFly 10.0.0.Final</a> and unzipped it.<br />
2. I <a href="http://repo1.maven.org/maven2/org/jolokia/jolokia-war/" target="_blank">downloaded the latest Jolokia .war file</a> and copied it to my new WildFly Server's <span style="font-family: "courier new" , "courier" , monospace;">standalone/deployments</span> directory<br />
3. I started my WildFly Server and bound it to some IP address that I dedicated to it (in this case, it was simply a loopback IP that I dedicated to this WildFly Server):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">bin/standalone.sh -b 127.0.0.5 -bmanagement=127.0.0.5</span><br />
<br />
At this point, I now have a server with some JMX data exposed over the Jolokia endpoint.<br />
<br />
Second, I need to run a Hawkular Server. I won't go into details here on how to do that. Suffice it to say, either <a href="https://github.com/hawkular/hawkular-services" target="_blank">build</a> or <a href="http://www.hawkular.org/downloads.html" target="_blank">download</a> a Hawkular Server distribution and run it (if it is not a developer build, make sure you run your external Cassandra Server prior to starting your Hawkular Server - e.g. <span style="font-family: "Courier New",Courier,monospace;">sudo docker run cassandra -p 9042:9042</span>).<br />
<br />
Now I want to run a Hawkular WildFly Agent to collect some of that JMX data from my WildFly Server and store it in Hawkular Server. In this demo, I'll be running the Swarm Agent, which is simply a Hawkular WildFly Agent packaged in a single jar that you can run as a standalone app. However, its agent subsystem configuration is the same as if you were running the agent in a full WildFly Server so the configuration I'll be describing can be used no matter how you have deployed your agent.<br />
<br />
Rather than rely on <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-swarm-agents/hawkular-swarm-agent-dists/hawkular-swarm-agent-dist/src/main/resources/hawkular-swarm-agent-config.xml" target="_blank">the default configuration file that comes with the Swarm Agent</a> (which is designed to collect and store data from a WildFly Server's DMR management endpoint, not Jolokia) I extracted the default configuration file and edited it as I describe below.<br />
<br />
I deleted all the "-dmr" related configuration settings (metric-dmr, resource-type-dmr, remote-dmr, etc). I want my agent to only collect data from my Jolokia endpoint, so no need to define all these DMR metadata.<br />
<br />
I then added metadata that describes the JMX data I want to collect. For example, I collect availability metrics (to tell me if an MBean is available or not) and gauge metrics (to graph things like used memory). I also collect resource properties that some MBeans expose as JMX attributes. I assign these to different resources by defining resource metadata which point to specific JMX MBean ObjectNames. I then define the details of my Jolokia-enabled WildFly Server in a <remote-jmx> so my agent knows where my Jolokia-enabled WildFly Server is.<br />
<br />
Some example configuration is:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><avail-jmx name="Memory Pool Avail"<br /> interval="30"<br /> time-units="seconds"<br /> attribute="Valid"<br /> up-regex="[tT].*"/></span></blockquote>
This defines an availability metric that says for the attribute "Valid" if its value matches the regex "[tT].*" consider its availability UP (note this regex matches the string "true", case-insensitive), otherwise it is DOWN. We will attach this availability metric to a resource below.<br />
<br />
Here is an example of a gauge metric:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><metric-jmx name="Pool Used Memory"<br /> interval="1"<br /> time-units="minutes"<br /> metric-units="bytes"<br /> attribute="Usage#used"/></span></blockquote>
Notice the data can come from a sub-attribute of a composite value (the "used" value within the composite attribute "Usage").<br />
<br />
You group availability metrics and numeric metrics in metric sets (avail-set-jmx and metric-set-jmx respectively) and then associate those metric sets to specific resource types. Resource types define resources that you want to monitor (resources in JMX are simply MBeans identified with ObjectNames). For example, in my demo, I want to monitor my Memory Pools. So I create a resource type definition that describe the Memory Pools:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><resource-type-jmx name="Memory Pool MBean"<br /> parents="Runtime MBean"<br /> resource-name-template="%type% %name%"<br /> object-name="java.lang:type=MemoryPool,name=*"<br /> metric-sets="MemoryPoolMetricsJMX"<br /> avail-sets="MemoryPoolAvailsJMX"<br /> <resource-config-jmx name="Type"</span><span style="font-family: "courier new" , "courier" , monospace;"><br /> attribute="Type"/><br /></resource-type-jmx></span></blockquote>
Here you can see my resource type "Memory Pool Bean" refers to all resources that match the JMX query "java.lang:type=MemoryPool,name=*". For all the resources that match that query, I associate with them the availability and numeric metrics defined in the sets mentioned in the metric-sets and avail-sets attributes. I also want a resource configuration property collected for each resource - "Type". Each Memory Pool MBean has a Type attribute that we want to collect and store. Notice also that all of these resources are to be considered children of the parent resource whose resource type name is "Runtime MBean" (which I defined elsewhere in my configuration).<br />
<br />
Once all of my metadata is configured (I've configured the agent to collect all the configuration properties, availability and numeric metrics of all the JMX MBeans I want), I now configure the agent with the Jolokie-enabled endpoint. This tells the agent how to connect to the Jolokia endpoint and what I want the agent to monitor in that endpoint:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><remote-jmx name="My Remote JMX"<br /> enabled="true"<br /> resource-type-sets="MainJMX,MemoryPoolJMX"<br /> metric-tags="server=%ManagedServerName,host=127.0.0.5"<br /> metric-id-template="%ResourceName_%MetricTypeName"<br /> url="http://127.0.0.5:8080/jolokia-war-1.3.3"/></span></blockquote>
Here I configure the URL endpoint of my WildFly Server's Jolokia war. I then tell it what resource types I want to monitor in that Jolokia endpoint (I've grouped all my resource types into two different resource type sets called MainJMX and MemoryPoolJMX). The grouping is all up to you - if you want one big resource type set, that's fine. For metrics, you can have one big availability metric set and one big numeric metric set - it doesn't matter how you organize your sets.<br />
<br />
One final note before I run the agent. Notice I can associate my managed server with metric tags (remote-jmx is one type of "managed server"). For every metric that is collected for this remote-jmx managed server, those tags will be added to those metrics in the Hawkular Server (specifically in the Hawkular Metrics component). So for the "Pool Used Memory" metric I defined earlier, when that metric is stored in the Hawkular Server, it will be tagged with "server" and "host" where the values of those tags are the name of my managed server (in this case, %ManagedServerName is replaced with "My Remote JMX") and "127.0.0.5" respectively. All metrics will have the same tags. Similarly, the ID used to store the metrics in the Hawkular Metric component can also be customized though this is a rarely used feature and you probably will never need it. Both metric-tags and metric-id-template are optional. You can also place those attributes on individual metrics which is most useful if you have specific tags you want to add only to metrics of a specific metric type but not on all of the metrics collected for your managed server.<br />
<br />
OK, now that I've got my Swarm Agent configuration in place (call it "agent.xml" for simplicity), I can run the agent and point it to my configuration file:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">java -jar hawkular-swarm-agent-dist-*-swarm.jar agent.xml </span></blockquote>
At this point I have my agent running along side of my Hawkular Server and WildFly Server enabled with Jolokia. The agent is collecting information from Jolokia and pushing the collected data to Hawkular Server.<br />
<br />
In order to visualize this collected data, I'm using the experimental developer tool <a href="https://github.com/pilhuhn/hawkfx" target="_blank">HawkFX</a>. This is simply a browser that let's you see what data is in Hawkular Inventory as well as Hawkular Metrics. When I log in, I can see all the resources stored in Hawkular that comes directly from Jolokia - these resources represent the different JMX MBeans we asked the agent to monitor:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm8aZaDlXBiNgNG-xujHdYpBSk8DUJpv5rCuT4Hx0yQS5sjqaQW70jfKG7O3QEabFBCU1_LNUy7SQNTLd_Vgy-qX_wMzYo6BHdkkqDO96SlQ5HgEdXFzNO7tE6527L-URBySYZtaw7CJAL/s1600/inventory-tree.png" imageanchor="1"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm8aZaDlXBiNgNG-xujHdYpBSk8DUJpv5rCuT4Hx0yQS5sjqaQW70jfKG7O3QEabFBCU1_LNUy7SQNTLd_Vgy-qX_wMzYo6BHdkkqDO96SlQ5HgEdXFzNO7tE6527L-URBySYZtaw7CJAL/s320/inventory-tree.png" width="320" /></a><br />
<br />
You can see "My Remote JMX Runtime MBean" is my parent resource, it has one availability metric "VM Avail", three numeric metrics and six child resources (those are the Memory Pool resources described above when we added them to the configuration).<br />
<br />
You can drill down and see the metrics associated with the children as well. For example, the Memory Pool for the PS Old Gen has a metric "Pool Used Memory" that we can graph (the metric data was pulled from Jolokia, stored in Hawkular Metrics, which is then graphed by HawkFx as you see here):<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0x459p9MyFmfguOXX5GR5M8skLj1UtxzRoIKayEMU7PZDX3jvRGRVuEAU06-0P-VbEDMQKocd28I6O4k9g1dr0OHAUCqjrTTpvOOCaBwEggtmSZQksRTvNJ5Mr7GI1SRSfvipUao0Hj3n/s1600/metric-graph.png" imageanchor="1"><img border="0" height="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0x459p9MyFmfguOXX5GR5M8skLj1UtxzRoIKayEMU7PZDX3jvRGRVuEAU06-0P-VbEDMQKocd28I6O4k9g1dr0OHAUCqjrTTpvOOCaBwEggtmSZQksRTvNJ5Mr7GI1SRSfvipUao0Hj3n/s400/metric-graph.png" width="400" /></a><br />
<br />
Finally, you can use HawkFx to confirm that the resource configuration properties were successfully collected from Jolokia and stored into Hawkular. For example, here you can see the "Type" property we configured earlier - the type of this memory pool is "HEAP".<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1Oado8PmqSde_h54x96ocaI-aa76HxVW5KgzPTEPn9nP2WrIIq30SLmWncnMIJA-WKUOKIIPfgFUVjF1-f1qQ8z3YyoJIJ_cBoxqrlztfWuGvsjGwIDQaDg8S6C_jvPiTd9He5IV1GAwo/s1600/config-props.png" imageanchor="1"><img border="0" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1Oado8PmqSde_h54x96ocaI-aa76HxVW5KgzPTEPn9nP2WrIIq30SLmWncnMIJA-WKUOKIIPfgFUVjF1-f1qQ8z3YyoJIJ_cBoxqrlztfWuGvsjGwIDQaDg8S6C_jvPiTd9He5IV1GAwo/s400/config-props.png" width="400" /></a><br />
<br />
What this all shows is that you can use Hawkular WildFly Agent to collect resource information and metric data from JMX over a Jolokia endpoint and store that data inside Hawkular.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-74663628787348735112016-04-18T09:14:00.000-07:002016-04-19T10:54:26.199-07:00Prometheus Metric Endpoint Parser for JavaI have a need for a Java-based parser that can parse metric data from any Prometheus endpoint.<br />
<br />
Prometheus has two main data formats - a binary format and a text format. <a href="https://prometheus.io/docs/instrumenting/exposition_formats/" target="_blank">You can read about those formats here</a>. That document says that "Clients must support at least one of these two alternate formats." So I needed a Java-based parser that can parse both.<br />
<br />
The Prometheus team has <a href="https://github.com/prometheus/client_model" target="_blank">published parsers for several different languages</a> (e.g. C++, Go, Python, Ruby, and Java). Some only support the binary formats, Java being one of those with only binary support. In addition, the Prometheus team may delete the Java parser entirely since it is relatively unused by the community. As of this writing, the latest <a href="http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22io.prometheus.client%22%20AND%20a%3A%22model%22" target="_blank">release of the Prometheus Java parser</a> is version 0.0.2 from July 2013 which also doesn't support histograms (though version 0.0.3-SNAPSHOT in the master branch does support it - so if/when 0.0.3 is released, histogram support will be avaialble).<br />
<br />
So I needed to write my own Java-based parser for the text format to ensure I could read any Prometheus metric endpoint (even though the documentation says clients must support one or the other, in practice it seems all endpoints support the text format and only some (mainly Go endpoints) support the binary format). So even if the Java-based binary parser support goes away, having a text parser should still be able to read all Prometheus endpoints (in other words, those endpoints with binary-format support should also have text-format support as well).<br />
<br />
<a href="https://github.com/hawkular/hawkular-agent/tree/master/prometheus-scraper" target="_blank">Here is my Java-based Prometheus Metrics Scraper code</a>. There is a README for a quick synopsis. It supports both binary and text formats and utilizes content negotiation with the URL endpoint to determine what format to expect. You can also programatically process files as opposed to URL endpoints.<br />
<br />
This Prometheus Metrics Scraper comes with a CLI that you can run via a simple Java command:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">java -jar prometheus-scraper*-cli.jar [--simple | --xml | --json] {url}</span></blockquote>
It can output any URL endpoint's metric data in several formats (JSON and XML being the two more interesting ones). If you'd like to try it out, grab the latest release from <a href="https://repository.jboss.org/nexus/content/groups/public/org/hawkular/agent/prometheus-scraper/" target="_blank">here</a> and run it. For example, you can <a href="https://repository.jboss.org/nexus/content/groups/public/org/hawkular/agent/prometheus-scraper/0.17.1.Final/prometheus-scraper-0.17.1.Final-cli.jar" target="_blank">download the 0.17.1Final CLI jar here</a>.<br />
<br />
Programmatically you use this by simply passing a URL (or File) to <span style="font-family: "courier new" , "courier" , monospace;">PrometheusScraper</span> and calling its <span style="font-family: "courier new" , "courier" , monospace;">scrape()</span> method. This will return a list of <span style="font-family: "courier new" , "courier" , monospace;">MetricFamily</span> objects, which contain all the metric data found in the endpoint URL.<br />
<br />
See the code's Javadoc for more complete documentation.<br />
<br />
There are a few things still missing that would be nice to enhance for the future.<br />
<br />
First is histogram support for binary formatted data (but once the jar artifact "<span style="font-family: "courier new" , "courier" , monospace;">io.prometheus.client:model</span>" version 0.0.3 is released by the Prometheus team, it would just be a matter of uncommenting one block of code for my Java-based parser to begin supporting it). Of course, histograms are fully supported in the text parser.<br />
<br />
Secondly, the URL endpoint is assumed to be unsecured. If SSL certificates or authentication is required to access the metric data over the given URL, the scraper will fail to process the data.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com3tag:blogger.com,1999:blog-6944050187568241059.post-88331389585535569652016-01-21T11:59:00.000-08:002016-01-21T13:40:44.932-08:00Hawkular Command Gateway ClientsThis document is to briefly explain how clients can use the command gateway to send requests to the <a href="http://www.hawkular.org/" target="_blank">Hawkular</a> Server and receive responses. The typical use case (though certainly not the only way to use this feature) is for a browser to send requests to Hawkular WildFly Agents routed through the Hawkular Server, with the agent’s responses routed back to the browser (back through the server intermediary) that sent the request.<br />
<h4>
Brief Summary</h4>
The typical workflow is the following:<br />
<ol>
<li>Client makes a websocket connection to the server.</li>
<li>Client immediately receives a WelcomeResponse JSON message from the server notifying the client what its session ID is.</li>
<li>Client can send JSON requests over the Web Socket connection.</li>
<li>Client can receive JSON responses over the Web Socket connection. These messages are received asynchronous from its requests and may not even be tied to a specific request.</li>
<li>Client can keep the connection open as long as it wants, and can disconnect when it wants.</li>
</ol>
<h4>
Make the Connection</h4>
Clients first need to make a WebSocket connection to the Hawkular Server. That is done by connecting to the URL: "ws://<hawkular-server><hawkular-server>:8080/ui/ws".</hawkular-server></hawkular-server><br />
<br />
Note that a secure connection via SSL can be made by connecting to “wss://<hawkular-server><hawkular-server>:8443” but this requires the server to be configured properly with a certificate and a security realm defined. Such details are out of scope for this document. For more, see <a href="http://www.hawkular.org/docs/user/secure-comm.html">http://www.hawkular.org/docs/user/secure-comm.html</a></hawkular-server></hawkular-server><br />
<h4>
Receiving the Welcome Message</h4>
For each client websocket connection that is made, the server sends an initial WelcomeResponse message immediately to it. This message contains the client’s session ID. This is the session ID that the server will use to identify that client. This session ID is actually not needed by the client today, but it is available for future functionality. For now, clients can actually ignore this session ID.<br />
<br />
An example WelcomeResponse message that a client could receive is:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">WelcomeResponse={“sessionId”:”abc:123:xyz:789”}</span></blockquote>
<br />
The WelcomeResponse JSON schema is <a href="https://github.com/hawkular/hawkular-command-gateway/blob/master/hawkular-command-gateway-api/src/main/resources/schema/WelcomeResponse.schema.json" target="_blank">defined here</a>.<br />
<h4>
Sending a Request Message</h4>
Once connected, clients can immediately begin sending any valid request over the web socket connection. Because clients are able to send in different kinds of requests, the client must ensure the request message’s JSON content is prefixed with the JSON schema name followed by “=” . Valid JSON schemas can be <a href="https://github.com/hawkular/hawkular-command-gateway/tree/master/hawkular-command-gateway-api/src/main/resources/schema" target="_blank">found here</a>.<br />
<br />
Within the JSON content, there must be an authentication node containing the credentials of the client. The authentication node’s JSON schema is <a href="https://github.com/hawkular/hawkular-command-gateway/blob/master/hawkular-command-gateway-api/src/main/resources/schema/A0AuthMessage.schema.json" target="_blank">defined here</a>.<br />
<br />
So, for example, if you want to execute the “Undeploy” operation on a web deployment resource, the request sent over the web socket connection will look something like this:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">ExecuteOperationRequest={“authentication”:{“username”:”joe”,”password”:”pw”},“operationName”:”Undeploy”,”resourcePath”:”/t;tenant-id/e;env-id/r;resource-id-of-the-deployment-resource”}</span></blockquote>
<br />
Note that in order for a client to request something related to a specific resource in inventory, that client must know the resource’s “resource path” as defined in Hawkular Inventory. See the <a href="http://www.hawkular.org/docs/rest/rest-inventory.html" target="_blank">Hawkular Inventory REST API documentation</a> for more details on how to obtain inventory data such as these resource paths. <br />
<h4>
Receiving a Response Message</h4>
The clients can receive messages in the same format as it sent them. In other words, the JSON content received by the client will be prefixed with the JSON schema name that is to be used to properly parse the JSON content received.<br />
<br />
For example, when a client sends a request message destined to an agent, the server will immediately send a GenericSuccessResponse back to the client. This means the request was received by the server and has been successfully forward to the agent (note: it does not mean the request was successfully processed by the agent - the agent will send its own success or failure response message back once it processes the original request). Such a response message received by the client would look like this:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">GenericSuccessResponse={“message”:”The request has been forwarded to the feed [foo]”}</span>
</blockquote>
<h4>
Sending Binary Content</h4>
Sometimes a client needs to send raw binary data as part of a request (e.g. when a client wants to deploy a web application, it sends a DeployApplicationRequest along with the application’s .war file). To do this, the client sends the JSON message as usual (i.e. in the form “<span style="font-family: "courier new" , "courier" , monospace;">json-schema={json content}</span>”) but immediately following the final curly brace of the JSON content, the client should stream the binary content.John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-14257265748611520142016-01-01T07:51:00.000-08:002016-01-02T18:02:41.022-08:00Hawkular WildFly Agent API For Your Own Inventory and MetricsThe <a href="https://github.com/hawkular/hawkular-agent" target="_blank">Hawkular WildFly Agent</a> is well into development and is coming along nicely. It provides a way to monitor one or more <a href="http://wildfly.org/" target="_blank">WildFly</a> or <a href="http://www.jboss.org/products/eap/overview/" target="_blank">EAP</a> application servers (including the one it is deployed into). It communicates with a <a href="http://www.hawkular.org/" target="_blank">Hawkular Server</a> where the agent stores its inventory and metric data via the <a href="https://github.com/hawkular/hawkular-inventory" target="_blank">Hawkular Inventory</a> and <a href="https://github.com/hawkular/hawkular-metrics" target="_blank">Hawkular Metrics</a> components. You can use the Hawkular GUI to interact with your managed application servers: view historical graphs of metric data, deploy and undeploy applications, etc.<br />
<br />
But the Hawkular WildFly Agent provides a hidden gem that might be useful to those developers that want to store metrics in a metric storage facility for later reporting and graphing but don't want to take the time to implement that storage facility. This hidden gem also provides a way for developers to store their own managed resource definitions in an inventory storage facility but, again, don't want to implement all of the backend required for such a thing.<br />
<br />
The Hawkular WildFly Agent already integrates with Hawkular Inventory and Hawkular Metrics - this is to enable the agent to be able to store its own inventory and metrics. It makes sense to open that up for applications to use so they, too, can store inventory and metrics. To allow for this, Hawkular WildFly Agent stores a helpful object in JNDI called <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-agent/src/main/java/org/hawkular/agent/monitor/api/HawkularWildFlyAgentContext.java" target="_blank">HawkularWildFlyAgentContext</a> under the name "java:global/hawkular/agent/api" (side note: the agent can be told to bind this object to a different JNDI name if you want; it can also be told to not bind this hidden gem in JNDI at all if you do not wish to expose this feature).<br />
<br />
The API this exposes is very simple - you can <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-agent/src/main/java/org/hawkular/agent/monitor/api/InventoryListener.java" target="_blank">store or remove resources from inventory</a>, and you can <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-agent/src/main/java/org/hawkular/agent/monitor/api/MetricStorage.java" target="_blank">store metric data</a> and <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-agent/src/main/java/org/hawkular/agent/monitor/api/AvailStorage.java" target="_blank">availability data</a> (availability data is just a "special" kind of metric that allows you to store "UP", "DOWN" or "UNKNOWN" availability states).<br />
<br />
Any application that is deployed in the same WildFly or EAP application server as the agent can use this API by obtaining the agent context object via JNDI. A simple example of how you can obtain this agent context object from JNDI can be seen in a test war that is used in the agent integration tests - its a singleton EJB that gets this context object injected via @Resource. See the <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-agent-itest-parent/hawkular-wildfly-agent-example-jndi/src/main/java/org/hawkular/agent/example/HawkularWildFlyAgentProvider.java" target="_blank">HawkularWildFlyAgentProvider</a> class.<br />
<br />
Once that HawkularWildFlyAgentContext object is obtained, your application can use it to store inventory, metric, and availability data. Note that you do not have to use all of these. For example, if you just want to store metrics in a historical time-based data store, just use the Metric Storage API that you get from the agent context object. This will send your data for storage to Hawkular Metrics. You could then do whatever you want with your data later - Hawkular Metrics provides a REST interface and some clients that you can use to query and report on your metric data.<br />
<br />
The example test war can show you how the API is used to do these things - see the test <a href="https://github.com/hawkular/hawkular-agent/blob/master/hawkular-wildfly-agent-itest-parent/hawkular-wildfly-agent-example-jndi/src/main/java/org/hawkular/agent/example/MyAppServlet.java" target="_blank">MyAppServlet.java</a>. This is just for integration testing, so it doesn't do anything earth-shattering, but it does show you how the API is used to create and remove managed resources from inventory, store metric data, and store availability data.<br />
<br />
This feature of the Hawkular WildFly Agent is a relatively minor feature considering all the other main requirements that the agent must fulfill, but since it is rather hidden I decided to talk about it here.John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-54206997735920817472015-01-15T09:33:00.000-08:002015-01-15T09:33:29.125-08:00Hawkular BusWork under the new <a href="https://github.com/hawkular">Hawkular</a> umbrella has begun. More information about this new project will trickle out more and more from the team members in the coming weeks and months ahead. There's already a <a href="https://twitter.com/hawkular_org">Twitter</a> feed and a <a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev">developer mailing list</a>.<br />
<br />
Here I'll briefly discuss the <a href="https://github.com/hawkular/hawkular-bus">hawkular-bus</a> work I've done.<br />
<br />
The idea is that the hawkular-bus provides a framework to hook into a messaging bus including a Wildfly container that provides all the messaging infrastructure setup for you as well as a convienence API that frees you from the annoying JMS boilerplate code that you typically have to write when it comes to developing bus applications and EJB MDBs in Java.<br />
<br />
hawkular-bus also has a REST API that allows any client (Java or non-Java) to produce messages on the bus or consume messages off the bus.<br />
<br />
The container that you can use to deploy your bus-based components (be they WARs or EARs or Wildfly module extensions) is called a <a href="https://github.com/hawkular/hawkular-bus/tree/master/hawkular-nest">hawkular-nest</a>. It is simply a pre-configured Wildfly server - it provides the necessary module extensions that give you the messaging infrastructure such that all you need to do is deploy WARs and EARs to it and they can immediately take advantage of the hawkular framework. You can deploy your WARs and EARs directly in the nest module extension (modules/system/layers/base/org/hawkular/nest/main/deployments), thus immediately allowing your applications to be patchable using Wildfly's patching mechanism. In the future, we may want to use this deployment to be able to deploy some kind of "Hawkular Plugin" but nothing has been fleshed out. Right now, WAR and EAR deployments make it easy for you to plug into the hawkular message bus. There is even a <a href="https://github.com/hawkular/hawkular-bus/tree/master/hawkular-bus-mdb">hawkular-bus-mdb</a> API that provides an API to allow your MDBs to easily process RPC-like messages (that is, your MDB can not only listen for messages on the bus but then immediately return a response back to the client that sent the message).<br />
<br />
I then prototyped a <a href="https://github.com/jmazzitelli/hawkular-audit">hawkular-audit</a> component that uses hawkular-bus. Hawkular-audit is a server-side component that accepts "audit records" over the bus and places them in a backend storage (today, its just an in-memory H2 database - the one that comes with Wildfly). For example, if you have something that you want to track via an audit trail, that something can put its audit records on the bus and have hawkular-audit store it for you. Your client, if it is Java, can use <a href="https://github.com/hawkular/hawkular-bus/tree/master/hawkular-bus-common/src/main/java/org/hawkular/bus/common">hawkular-bus API</a> to send the message, or if your client is not Java you can have it use the hawkular-bus REST API to put the message on the bus (for example, using curl).<br />
<br />
To try out all this hawkular stuff:<br />
<br />
1. Clone both <a href="https://github.com/hawkular/hawkular-bus">hawkular-bus</a> and <a href="https://github.com/jmazzitelli/hawkular-audit">hawkular-audit</a> repos<br />
2. Build them by just executing "mvn install" from each repo's root directory.<br />
3. Unzip the distribution from hawkular-audit/hawkular-audit-distro/target and run the "standalone.sh" from the Wildfly's bin/ directory.<br />
4. When the hawkular-audit server starts, it will log its own audit event to record when it started. To see the audit records, point your browser to "http://localhost:8080/hawkular-audit/AuditQueryServlet" and you'll see a simple HTML table with the data.<br />
5. To add your own audit records to the audit trail, run curl to send your audit records over the REST API. For example, try something like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> curl -d "body={"subsystem":{"MyComponent":"MySubsystem"},"message":"My audit record message","details":{"key2":"value2","key1":"value1"}}" http://localhost:8080/hawkular-bus/message/AuditQueue?type=queue</span><br />
<br />
Refresh your browser using the same URL mentioned in step 4 above and you'll see your new records.<br />
<br />
Hawkular-audit is just a prototype that shows how you can build your own server components to utilize the hawkular-bus and hawkular-nest in order to plug into the bus framework and be able to receive (and send) messages off the bus. In the future, we'll want to cluster the bus so multiple components can talk to each other over one main bus and have clients hook into the bus to send messages to one or more of these server components.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-75033741990755466932014-11-12T15:38:00.000-08:002014-11-12T15:44:14.531-08:00Messaging Infrastructure using ActiveMQ<br/>
<h2>What is rhq-msg?</h2>
<p>Before I get into the news about the new rhq-msg repository, let me first take a step back and summarize what rhq-msg is and what I'm trying to accomplish with it.</p>
<p>rhq-msg is a simple messaging API built on top of ActiveMQ and JMS. However, I wanted to isolate the user of the API from as much ActiveMQ and JMS specific classes and code as possible. I wanted a simpler API that provides basic messaging functionality without requiring the user from having to create and manage lots of little specific classes (like JMS Connections, Sessions, Destinations, Consumers, Producers, etc) and without having to know very many specific ActiveMQ details.</p>
<p>I also wanted to ensure that non-Java clients and servers can interact with rhq-msg clients and servers. So the messages that are sent and received from the rhq-msg API are JSON-encoded and thus can be sent to and received by other non-Java endpoints so long as they can handle JSON-encoded messages (ActiveMQ supports non-Java clients, I just wanted to make sure the messages rhq-msg handles can easily flow to/from those non-Java clients as well).</p>
<p>So the point is I have small, simple, easier-to-use messaging API that can talk to non-Java endpoints, but yet still retain all the nice functionality (like guaranteed delivery and things like that) that JMS and ActiveMQ provides.<p>
<h2>New Repository for rhq-msg</h2>
<p>My past few blog posts were about the prototyping work I'm doing with respect to rhq-msg and <a href="https://github.com/rhq-project/rhq-audit">rhq-audit</a>.</p>
<p>I just split out rhq-msg and put it in <a href="https://github.com/rhq-project/rhq-msg">its own rhq-msg repository</a> since it really does belong as a separate project.</p>
<p>I also added a nifty feature to it - you can now use it for request-response workflows. Before, the rhq-msg API really only supported fire-and-forget async messaging. You had a message and you sent it to an endpoint asynchronously and that was that. The problem is sometimes you want a response back, and many times you want to wait for that response to come back in an RPC-like fashion (as opposed to accepting the response asynchrously). I call this a request-response workflow.</p>
<p>Well, the API now supports this. In fact, it supports receiving the response both synchronously (via a Java Future implementation) and asynchronously (via a message listener implementation). More on this below.</p>
<h2>A Quick Overview of the API</h2>
<p>The main purpose of rhq-msg is to provide a simpler API to do messaging (simpler than, say, JMS). If I can't describe how to send and receive messages via rhq-msg in a couple paragraphs of a blog, then I think I failed :) So let me give a quick overview of what the API calls would look like if you want to send and receive messages. I will cover sending fire-and-forget messages, request-response messaging, and listening for incoming messages.</p>
The main rhq-msg API is found in the <a href="https://github.com/rhq-project/rhq-msg/tree/master/rhq-msg-common">rhq-msg-common module</a>.
<p><b>* Common Code For Both Producers and Consumers</b></p>
<p>Before I talk about how to send and receive messages, let me introduce the few classes both senders (aka producers) and receivers (aka consumers) use.
<p>Each message that flows through rhq-msg is a <a href="https://github.com/rhq-project/rhq-msg/blob/master/rhq-msg-common/src/main/java/org/rhq/msg/common/BasicMessage.java">BasicMessage</a>. You can subclass that to create your own message types, but all messages must derive from BasicMessage. BasicMessage provides the JSON functionality needed to serialize the messages over the wire.</p>
<p><a href="https://github.com/rhq-project/rhq-msg/blob/master/rhq-msg-common/src/main/java/org/rhq/msg/common/ConnectionContextFactory.java">ConnectionContextFactory</a> connects to your rhq-msg broker and creates contexts for both producers and consumers that are then passed to the MessageProcessor so it knows where to send/receive its messages from.</p>
<p><a href="https://github.com/rhq-project/rhq-msg/blob/master/rhq-msg-common/src/main/java/org/rhq/msg/common/MessageProcessor.java">MessageProcessor</a> provides the API to send and listen for messages.</p>
<p>OK, with that out of the way, let's talk about how to send and listen for messages. For these examples, let's assume you have a broker running on the local machine (127.0.0.1) listening to port 17173, you have a BasicMessage you want to send (or you want to listen for a BasicMessage) and the messages are to be found on a queue named "Foo".</p>
<p><b>* Sending Fire-and-Forget Messages</b></p>
<p>Fire-and-forget means you send a message off to an endpoint and forget about it - you don't need or expect a response back. You assume the message broker will deliver the message and the recipient will process the message properly.</p>
<pre>
Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
ProducerConnectionContext context = factory.createProducerConnectionContext(endpoint);
MessageProcessor processor = new MessageProcessor();
processor.send(context, basicMessage);
factory.close();
</pre>
<p>Here you create your factory so it connects to your broker. Using the factory, create a producer context. With that new producer context and your basic message, just tell a message processor to send it. That's it. Message is away. You can keep your factory around to create additional contexts which share the connection. But once you are done sending messages, close the factory so it cleans up all resources and closes its connection to the broker. Note that every class you see is a rhq-msg object. No ActiveMQ or JMS classes are used to code this up (though, obviously, under the covers, ActiveMQ and JMS is used heavily).</p>
<p><b>* Sending Request-Response Messages</b></p>
<p>Many times when you send a message request, you want a response back. In this example I will show how you can have an RPC-like request-response workflow. I will assume that the remote endpoint will process my BasicMessage and send back to me its own BasicMessage response. Note that, as I mentioned earlier, I could use custom message types (I don't have to use BasicMessage) but those custom message types must always derive from BasicMessage.</p>
<pre>
Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
ProducerConnectionContext context = factory.createProducerConnectionContext(endpoint);
MessageProcessor processor = new MessageProcessor();
Future<BasicMessage> future = processor.sendRPC(context, basicMessage, BasicMessage.class);
BasicMessage response = future.get(30, TimeUnit.SECONDS);
</pre>
<p>The first several lines are identical with the fire-and-forget example above. The difference starts with the API call made on the processor - here you call sendRPC() passing in the same context and your outgoing basicMessage request object as before, but you also now pass in the message type of the expected response message. In return, you get a Future object which you can use to retrieve the response when it is received. In this example, it blocks for 30 seconds waiting for the response.</p>
<p>There is another API I won't talk about in detail here, but suffice it to say there is another request-response API you can call on the MessageProcessor (as opposed to sendRPC()) and that is "sendAndListen()". sendAndListen() also allows you to send a request and listen for a response, but this API allows you to give your own message listener so it can wait for and receive the response, rather than go through a Future object. It seems more intuitive and easier to use Future, but in case you want to write your own listener object and listen for the response that way, this is doable. I'll explain the listener API below - it would be the same thing you would need to pass to sendAndListen().<p>
<p><b>* Listening for Incoming Messages</b></p>
<p>This last example shows how you implement consumers via rhq-msg API. You implement these via listeners. These listeners are your server-side code - they listen for and accept incoming messages from producers and process those messages. There are two general types: those that do not send responses back, and those that do. I'll cover both in that order.</p>
<p>To listen for "fire-and-forget" messages (that is, messages that are sent that do not require a response sent back to the sender) you implement a BasicMessageListener and hand that listener off to the message processor:</p>
<pre>
Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
ConsumerConnectionContext context = factory.createConsumerConnectionContext(endpoint);
MessageProcessor processor = new MessageProcessor();
processor.listen(context, listener);
</pre>
<p>where "listener" is a subclass implementation of BasicMessageListener. An example is:</p>
<pre>
public class MyCustomListener extends BasicMessageListener<BasicMessage> {
@Override
protected void onBasicMessage(BasicMessage receivedMessage) {
// Process the received message.
}
}
</pre>
<p>Notice that we still create a ConnectionContextFactory (because we still need a connection to the message broker) but we ask that factory to create for us a consumer context this time. We call "listen()" on the message processor, passing to it that consumer context and our custom listener. That listener is now listening for messages to arrive on the "Foo" queue and will process them.</p>
<p>What about request-response processing? If your listener needs to send data back to the sender, it needs to implement a RPCBasicMessageListener. Other than that difference, the main code is still the same:</p>
<pre>
Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
ConsumerConnectionContext context = factory.createConsumerConnectionContext(endpoint);
MessageProcessor processor = new MessageProcessor();
processor.listen(context, listener);
</pre>
<p>but this time "listener" is a subclass implementation of RPCBasicMessageListener. An example is:</p>
<pre>
public class MyCustomRPCListener extends RPCBasicMessageListener<BasicMessage, AnotherMessage> {
@Override
protected AnotherMessage onBasicMessage(BasicMessage receivedMessage) {
// Process the received message.
AnotherMessage response = ...create your response message object...;
return response;
}
}
</pre>
<p>Note the onBasicMessage() method now can return a non-void type - the response message type. This return type (the specific response message type) is declared in the generic type definition found in your class definition. The first generic type is that of the expected incoming message type, the second generic type is that of the response message type.</p>
<h2>How to Build rhq-msg?</h2>
<p>rhq-msg is composed of a set of Maven modules. From the parent root module directory, just run "mvn install" and everything will build, tests will run, and artifacts will be packaged.</p>
<p>There are also Eclipse project files that allow you to work with rhq-msg via <a href="http://www.eclipse.org/m2e/">M2E</a> (the Eclipse Maven plugin).</p>
<h2>How to Run a rhq-msg Broker</h2>
<p>If you want to run your own code that utilizes the rhq-msg API, you will need to run a rhq-msg broker (which is really just an ActiveMQ broker). You can run a rhq-msg broker easily in a couple ways. You can run the EmbeddedBroker from the Maven command line or you can install the EmbeddedBroker in a WildFly 8 installation. This EmbeddedBroker code is found in the rhq-msg-broker Maven module.</p>
<p><b>* Running EmbeddedBroker from the command line</b></p>
<p>Starting a broker as a standalone process is relatively painless. You can use Maven to do it:</p>
<pre> mvn -Prun-test-broker install</pre>
<p>Right now the rhq-msg broker is not packaged with all its dependencies, but if you do it yourself (that is, get all the dependency jars and start Java with the appropriate classpath) you can run straight from the Java command line:</p>
<pre> java -cp <the-appropriate-classpath> -jar rhq-msg-broker-0.1.jar -c broker-config.xml</pre>
<p>where "broker-config.xml" is an actual <a href="http://activemq.apache.org/xml-configuration.html">ActiveMQ XML configuration file</a> (you can also pass in a simpler ActiveMQ .properties configuration file if you wish). Examples are <a href="https://github.com/rhq-project/rhq-msg/blob/master/rhq-msg-broker/src/test/resources/test-broker.xml">test-broker.xml</a> and <a href="https://github.com/rhq-project/rhq-msg/blob/master/rhq-msg-broker/src/test/resources/test-broker.properties">test-broker.properties</a>.</p>
<p><b>* Installing EmbeddedBroker in WildFly 8</b></p>
<p>One of the artifacts within rhq-msg is a WildFly Extension Module that provides a broker. Once installed in WildFly, you will have an rhq-msg broker subsystem running within WildFly itself. It is a handy way to have your own broker running in your own WildFly instance. So this means you can have, for example, an rhq-msg client within a web application running in WildFly and provide your own broker for that client running in the same WildFly instance. Technically, this is useful for any ActiveMQ or JMS client - since this broker is nothing more than an ActiveMQ broker and can serve any client from anywhere, not just those using the rhq-msg API.</p>
<p>First, <a href="http://wildfly.org/downloads/">download and unzip the WildFly 8 app server</a> (I have not tested on WildFly 9+). I'll use <wildfly-install-dir> to indicate where you installed WildFly.<p>
<p>Now install the custom <a href="https://github.com/rhq-project/rhq-msg/tree/master/rhq-msg-broker-wf-extension">rhq-msg broker WildFly Extension Module</a> into WildFly. You can use the <a href="http://lzoubek.github.io/wildfly-extension-maven-plugin/">new Maven plugin "wildfly-extension-maven-plugin"</a> to do this - the rhq-msg-broker-wf-extension Maven module has integrated that Maven plugin. I talk about this in <a href="/2014/11/using-new-maven-wildfly-extension.html">a previous blog post</a>. So, simply running this Maven command will install your rhq-msg broker WildFly Extension Module for you:</p>
<pre> mvn -Dorg.rhq.msg.broker.wildfly.home=<wildfly-install-dir> wildfly-extension:deploy</pre>
<p>Since this is nothing more than a normal subsystem like all the other subsystems in WildFly, you can use the JBoss CLI to look at its configuration. To poke around, run the JBoss CLI in GUI mode ("jboss-cli.sh --gui") and look for the "org.rhq.msg.broker" subsystem.</p>
<h2>In Closing</h2>
<p>This is a prototype and was developed just to see how a messaging API around JMS and ActiveMQ can be made simpler and easier to use. I am sure I am missing some pieces of functionality (one such missing piece is security - this currently doesn't handle logging into a secured broker). If you see anything missing, let me know. Feel free to suggest corrections or enhancements. All the code is in github, so play around with it and see how useful it is.</p>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-61971194343180405442014-11-05T14:05:00.002-08:002014-11-05T14:05:41.526-08:00Using a Message Broker to Send Audit Messages<p>One thing I've been working on on the side is a little project that uses a message broker to send and receive "auditing" events. Just briefly, the idea is that any piece of software might want to emit audit messages to log certain events that have occurred so that those events are captured for future reporting purposes. This project (call it rhq-audit for now) can provide a message broker if you need one, along with a high-level API that can be used to both emit and process these audit messages. For example, maybe you want your sales application to record each attempted login by a remote client, or maybe you want to record each sales order that has been submitted. Using the rhq-audit API, audit messages can be fired off on the message bus for later consumption by an rhq-audit consumer whose job it will be to store the audit record in a persistence store that can be used by reporting tools when audit reports need to be generated.</p>
<p>RHQ-Audit doesn't provide any reporting tools, but does provide a compact <a href="https://github.com/rhq-project/rhq-audit/tree/master/rhq-audit/rhq-audit-api">API</a> that allows you to emit and store audit messages. You can persist your audit messages to a simple log file (see <a href="https://github.com/rhq-project/rhq-audit/blob/master/rhq-audit/rhq-audit-api/src/main/java/org/rhq/audit/consumer/LoggerConsumer.java">LoggerConsumer</a>) or to a relationship database (see <a href="https://github.com/rhq-project/rhq-audit/blob/master/rhq-audit/rhq-audit-api/src/main/java/org/rhq/audit/consumer/DataSourceConsumer.java">DataSourceConsumer</a>), or to any custom backend (you would just need to implement your own <a href="https://github.com/rhq-project/rhq-audit/blob/master/rhq-msg/rhq-msg-common/src/main/java/org/rhq/msg/common/consumer/BasicMessageListener.java">BasicMessageListener</a> and have the <a href="https://github.com/rhq-project/rhq-audit/blob/master/rhq-audit/rhq-audit-api/src/main/java/org/rhq/audit/common/AuditRecordProcessor.java">AuditRecordProcessor</a> use it as its listener).</p>
<p>I talk a bit more detail on this type of thing in my previous <a href="/2014/07/starting-activemq-project-with-maven.html">blog on starting out with ActiveMQ</a>. The code now in the <a href="https://github.com/rhq-project/rhq-audit">rhq-audit repo</a> is a bit different, but the concepts are still the same.</p>
<p>I separated out the message broker stuff (along with a generic and extensible consumer/producer API) into <a href="https://github.com/rhq-project/rhq-audit/tree/master/rhq-msg">rhq-msg</a>. rhq-audit extends rhq-msg to handle audit events. But the idea is rhq-msg can be used to provide a message bus along with an API for any type of messaging app you want to build.</p>
<p>In addition to the <a href="https://github.com/rhq-project/rhq-audit/tree/master/rhq-msg/rhq-msg-common">API</a>, an <a href="https://github.com/rhq-project/rhq-audit/tree/master/rhq-msg/rhq-msg-broker">embedded broker</a> that you can use to embed in your VM or run standalone, and an extensible <a href="https://github.com/rhq-project/rhq-audit/tree/master/rhq-msg/rhq-msg-test-common">test API</a>, there is also a <a href="https://github.com/rhq-project/rhq-audit/tree/master/rhq-msg/rhq-msg-broker-wf-extension">custom Wildfly module extension</a> that can be used to easily deploy a message broker inside your application server via Maven. See <a href="/2014/11/using-new-maven-wildfly-extension.html">my previous blog on how to install that Wildfly extension via a mvn plugin</a>.</p>
<p>To build and run the tests for rhq-audit and rhq-msg, all you need to do is run "mvn install" from the top level root directory.</p>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-11332584652848642702014-11-05T09:54:00.003-08:002014-11-05T10:02:05.824-08:00Using the new Maven Wildfly Extension plugin to deploy module extensions<a href="http://lzoubek.blogspot.com/2014/10/wildfly-extension-installer-maven-plugin.html">Libor is putting together a nice maven plugin</a> to help you deploy your custom Wildfly module extensions to an existing Wildfly or EAP installation.<br />
<br />
I decided to integrate it with my rhq-msg-broker-wf-extension project (<a href="/2014/07/starting-activemq-project-with-maven.html">I talked about my prototype of the rhq-msg work here</a>). What this now allows us to do is deploy a message broker directly in an existing Wildfly or EAP installation via a simple mvn command line.<br />
<br />
I won't repeat all of Libor's instructions - just go to his blog I linked above for the full details of how to integrate his mvn plugin in your mvn project - but in short all I had to do was create a snippet of .xml which includes the configuration I want to deploy in the Wildfly or EAP instance's standalone.xml (which includes the subsystem and the socket-binding) and then in the pom just define some settings for the plugin to let it know where to put my extension code and that XML configuration.<br />
<br />
In the pom.xml, I have this to define the mvn plugin configuration - notice I allow the user to tell us where the app server is installed via a system property "org.rhq.msg.broker.wildfly.home":<br />
<br /><blockquote><pre>
<plugin>
<groupId>org.jboss.plugins</groupId>
<artifactId>wildfly-extension-maven-plugin</artifactId>
<configuration>
<moduleZip>${project.build.directory}/${project.build.finalName}-module.zip</moduleZip>
<jbossHome>${org.rhq.msg.broker.wildfly.home}</jbossHome>
<modulesHome>modules/system/layers/base</modulesHome>
<serverConfig>standalone/configuration/standalone.xml</serverConfig>
<subsystem>${basedir}/src/main/scripts/standalone-subsystem.xml</subsystem>
<profiles>
<profile>default</profile>
</profiles>
<socketBinding>${basedir}/src/main/scripts/socket-binding.xml</socketBinding>
<socketBindingGroups>
<socketBindingGroup>standard-sockets</socketBindingGroup>
</socketBindingGroups>
</configuration>
</plugin>
</pre></blockquote><br/>
To publish the rhq-msg-broker-wf-extension custom module extension to an already-installed EAP instance, simply run this command (this sets the system property to point to a EAP install directory, and it uses the goal "wildfly-extension:deploy" which tells the mvn plugin to deploy the extension to that EAP installation):<br />
<div>
<br /></div>
<span style="font-family: 'Courier New', Courier, monospace;">mvn -Dorg.rhq.msg.broker.wildfly.home=/opt/jboss-eap-6.3 wildfly-extension:deploy</span><br />
<div>
<br /></div>
<div>
At this point, the standalone.xml will have been modified to include the custom module extension's subsystem configuration and the module code will have been copied to the EAP modules directory. At this point, once the app server is started, the module extension will be deployed and running.<br />
<br /></div>John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-14830658148111086132014-07-29T08:35:00.000-07:002014-07-29T08:35:34.602-07:00Starting an ActiveMQ Project with Maven and EclipseI'm currently researching and prototyping a new subproject under the <a href="http://rhq.jboss.org/">RHQ</a> umbrella that will be a subsystem that can perform the emission and storage of audit messages (we're tentatively calling it "rhq-audit").<br />
<br />
I decided to start the prototype with <a href="http://activemq.apache.org/">ActiveMQ</a>. But one problem I had was I could not find a "starter" project that used ActiveMQ. I was looking for something with basic, skeleton Maven poms and Eclipse project files and some stub code that I could take and begin fleshing out to build a prototype. So I decided to publish my basic prototype to fill that void. If you are looking to start an ActiveMQ project, or just want to play with ActiveMQ and want a simple project to experiment with, then this might be a good starting point for you. This is specifically using ActiveMQ 5.10.<br />
<br />
The code is in Github located at <a href="https://github.com/jmazzitelli/activemq-start">https://github.com/jmazzitelli/activemq-start</a><br />
<br />
Once you clone it, you can run "mvn install" to compile everything and run the unit tests. Each maven module has an associated Eclipse project and can be directly imported into Eclipse as-is. If you have the <a href="https://www.eclipse.org/m2e/">Eclipse M2E plugin</a>, these can be imported using that Eclipse Maven integration.<br />
<br />
Here's a quick overview of the Maven modules and a quick description of some of the major parts of the code:<br />
<br />
<ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/pom.xml">/pom.xml</a></li>
<ul>
<li>This is the root Maven module's pom. The name of this parent module is rhq-audit-parent and is the container for all child modules. This root pom.xml file contains the dependency information for the project (e.g. dependency versions and the repositories where they can be found) and identifies the child modules that are built for the entire project.</li>
</ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/tree/master/rhq-audit-common">rhq-audit-common</a></li>
<ul>
<li>This Maven module contains some core code that is to be shared across all other modules in the project. The main purpose of this module is to provide code that is shared between consumer and producer (specifically, the message types that will flow from sender to receiver).</li>
<ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-common/src/main/java/org/rhq/audit/common/AuditRecord.java">AuditRecord.java</a> is the main message type the prototype project plans to have its producers emit and its consumers listen for. It provides JSON encoding and decoding so it can be sent and received as JSON strings.</li>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-common/src/main/java/org/rhq/audit/common/AuditRecordProcessor.java">AuditRecordProcessor.java</a> is an abstract superclass that will wrap producers and consumers. This provides basic functionality such as connecting to an ActiveMQ broker and creating JMS sessions and destinations.</li>
</ul>
</ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/tree/master/rhq-audit-broker">rhq-audit-broker</a></li>
<ul>
<li>This provides the ability to start an ActiveMQ broker. It has a main() method to allow you to run it on the command line, as well as the ability to instantiate it in your own Java code or unit tests.</li>
<ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-broker/src/main/java/org/rhq/audit/broker/EmbeddedBroker.java">EmbeddedBroker.java</a> is the class that provides the functionality to embed an ActiveMQ broker in your JVM. It can be configured using either an ActiveMQ .properties configuration file or an ActiveMQ <a href="http://activemq.apache.org/xml-reference.html">.xml configuration file</a>.</li>
</ul>
</ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/tree/master/rhq-audit-test-common">rhq-audit-test-common</a></li>
<ul>
<li>The thinking with this module is that there is probably going to be common test code that is going to be needed between producer and consumer. This module is to support this. The intent is for other Maven modules in this project to list this module as a dependency with a scope of "test". For example, some common code will be needed to start a broker in unit tests - including this module as a test dependency will give unit tests that common code.</li>
<ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-test-common/src/main/java/org/rhq/audit/common/test/TCPEmbeddedBrokerWrapper.java">TCPEmbeddedBrokerWrapper.java</a> provides a simple ActiveMQ broker <a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-test-common/src/main/resources/simple-activemq.xml">configured very simply</a> and will use a free TCP port to listen to. Tests can have their consumers and producers communicate with this broker.</li>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-test-common/src/main/java/org/rhq/audit/common/test/VMEmbeddedBrokerWrapper.java">VMEmbeddedBrokerWrapper.java</a> provides a simple ActiveMQ that is only for intra-VM messaging. It too is <a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-test-common/src/main/resources/simple-activemq.properties">very simply configured</a> and is useful, again, for unit or integration tests that need a broker for message processing.</li>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-test-common/src/test/java/org/rhq/audit/common/test/EmbeddedBrokerTest.java">EmbeddedBrokerTest.java</a> exercises the above test brokers by using a test consumer and producer (which are also available for use by other Maven modules since they are also part of <a href="https://github.com/jmazzitelli/activemq-start/tree/master/rhq-audit-test-common/src/main/java/org/rhq/audit/common/test">the common code provided</a> by this rhq-audit-test-common module).</li>
</ul>
</ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/tree/master/rhq-audit-producer">rhq-audit-producer</a></li>
<ul>
<li>This provides the producer-side functionality of the project. The intent here is to flesh out the API further. This will become rhq-audit's producer API.</li>
<ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-producer/src/main/java/org/rhq/audit/producer/AuditRecordProducer.java">AuditRecordProducer.java</a> provides a simple API that allows a caller to connect the producer to the broker and send messages. The caller need not worry about working with the JMS API as that is taken care of under the covers.</li>
</ul>
</ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/tree/master/rhq-audit-consumer">rhq-audit-consumer</a></li>
<ul>
<li>This provides the consumer-side functionality of the project. The intent here is to flesh out the client-side API further. This will become the rhq-audit's consumer API.</li>
<ul>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-consumer/src/main/java/org/rhq/audit/consumer/AuditRecordConsumer.java">AuditRecordConsumer.java</a> provides a simple API that allows a caller to connect the consumer to the broker and attach listeners so they can process incoming messages.</li>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-consumer/src/main/java/org/rhq/audit/consumer/AuditRecordListener.java">AuditRecordListener.java</a> provides the abstract listener class that is to be extended in order to process received audit records. The idea here is that subclasses can process audit records in different ways - perhaps one can store the audit records in a backend data store, and another can log the audit messages in rsyslog.</li>
<li><a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-consumer/src/test/java/org/rhq/audit/consumer/AuditRecordConsumerTest.java">AuditRecordConsumerTest.java</a> provides a simple end-to-end unit test that uses the embedded broker to pass messages between a producer and consumer.</li>
</ul>
</ul>
</ul>
<div>
Taking a look at <a href="https://github.com/jmazzitelli/activemq-start/blob/master/rhq-audit-consumer/src/test/java/org/rhq/audit/consumer/AuditRecordConsumerTest.java">AuditRecordConsumerTest</a> shows how this initial prototype can be tested and shows how audit records can be sent and received through an ActiveMQ broker:</div>
<div>
<br /></div>
<div>
1. Create and start the embedded broker:</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">VMEmbeddedBrokerWrapper broker = new VMEmbeddedBrokerWrapper();<br />broker.start();<br />String brokerURL = broker.getBrokerURL();</span></blockquote>
3. Connect the producer and consumer to the test broker:<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">producer = new AuditRecordProducer(brokerURL);<br />consumer = new AuditRecordConsumer(brokerURL);</span></blockquote>
</div>
<div>
2. Prepare to listen for audit record messages:</div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">consumer.listen(Subsystem.MISCELLANEOUS, listener);</span></blockquote>
<div>
3. Produce audit record messages:</div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">producer.sendAuditRecord(auditRecord);</span></blockquote>
At this point, the messages are flowing and the test code will ensure that all the messages were received successfully and had the data expected.<div>
<br /><div>
A lot of the code in this prototype is generic enough to provide functionality for most messaging projects; but of course there are rhq-audit specific types such as AuditRecord involved. The idea is to now flesh out this generic prototype to further provide the requirements of the rhq-audit project. More on that will be discussed in the future. But for now, perhaps this could help others come up to speed quickly with an AcitveMQ project without having to start from scratch.</div>
</div>
John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-53746440207595107572014-04-15T14:10:00.001-07:002014-04-15T14:10:21.954-07:00Completed Remote Agent InstallMy <a href="http://management-platform.blogspot.com/2014/03/remote-install-of-jon-agent.html">previous blog post</a> talked about work being done on implementing <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1070242">an enhancement request</a> which asked for the ability to remotely install an RHQ Agent. That feature has been finished and checked into the master branch and will be in the next release.<br />
<br />
I created <a href="http://mazz.fedorapeople.org/demos/remote-agent-install/remote-agent-install.ogv">a quick 11-minute demo</a> showing the UI (which is slightly differently than what the prototype looked like) and demonstrates the install, start, stop, and uninstall capabilities of this new feature.<br />
<br />
I can already think of at least two more enhancements that can be added to this in the future. One would be to support SSH keys rather than passwords (so you don't have to require passwords to make the remote SSH connection) and the other would be to allow the user to upload a custom rhq-agent-env.sh file so that file can be used to override the default agent environment (in other words, it would be used instead of the default rhq-agent-env.sh that comes with the agent distribution).John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-25021740286332377882014-03-20T09:11:00.004-07:002014-03-21T05:33:01.100-07:00Remote Install of JON AgentA <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1070242" target="_blank">new feature request</a> has been added to JBoss Operations Network. JON users will now be able to install agents on remote boxes from the UI as long as the remote box is accessible via SSH.<br />
<br />
All you need is the hostname of the machine you want to install the agent on, its SSH port that it's listening to (default is 22) and the credentials of the user who will install (and run) the JON agent.<br />
<br />
You can install, start, and stop the JON agent from this UI mechanism. You can also use it to get the status of any JON agent that might be installed as well and even attempt to find if and where an agent may be installed on the remote box.<br />
<br />
Here's a <a href="https://drive.google.com/file/d/0B8qtap4jaGQeYml5d0NrbUJWUWM/edit?usp=sharing" target="_blank">snapshot</a> of the UI page after I just successfully remote installed a JON agent:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/dm6gZiUvj0q3qj0SFvax1nrj4Mqtp6UtgVRbD3ZOcEepk6ErCtOrCitdvhUGqrLlgw=s400" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/dm6gZiUvj0q3qj0SFvax1nrj4Mqtp6UtgVRbD3ZOcEepk6ErCtOrCitdvhUGqrLlgw=s400" height="200" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com1tag:blogger.com,1999:blog-6944050187568241059.post-30373884256814950832013-09-12T13:47:00.001-07:002013-09-12T13:47:09.124-07:00Availability Updates in RHQ GUIIn older versions, the RHQ GUI showed you the availability status of resources but if you were viewing the resource in the GUI, it did not update the icons unless you manually refreshed the screen.<br />
<br />
In RHQ 4.9, this has changed. If you are currently viewing a resource and its availability status changes (say, it goes down, or it comes back up), the screen will quickly reflect the new availability status by changing the availabilty icon and by changing the tree node icons.<br />
<br />
To see what I mean, take a look at this quick 3-minute demo to see the feature in action (view this in full-screen mode if you want to get a better look at the icons and tree node badges):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<object class="BLOGGER-youtube-video" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0" data-thumbnail-src="https://lh4.googleusercontent.com/SEUDZ3WVAl8EldJ271wdY7GTvcsDBuNECwDx4NJLwGrjT0ItkoXd0WOPl4a6IN9LEQ=s0" height="344" width="512"><param name="movie" value="https://video.google.com/get_player?ps=docs&partnerid=30&docid=0B8qtap4jaGQeSWRLWkw0bUtBWWc&BASE_URL=https://docs.google.com/" /><param name="bgcolor" value="#FFFFFF" /><param name="allowFullScreen" value="true" /><embed width="512" height="344" src="https://video.google.com/get_player?ps=docs&partnerid=30&docid=0B8qtap4jaGQeSWRLWkw0bUtBWWc&BASE_URL=https://docs.google.com/" type="application/x-shockwave-flash" allowfullscreen="true"></embed></object></div>
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-17866958394681661402013-09-11T12:59:00.000-07:002013-09-11T12:59:12.907-07:00Fine-Grained Security Permissions In Bundle ProvisioningRHQ allows one to bundle up content and provision that bundle to remote machines managed by RHQ Agents. This is what we call the "Bundle" subsystem, the documentation actually titles it the "<a href="https://docs.jboss.org/author/display/RHQ/Provisioning" target="_blank">Provisioning</a>" subsystem. I've blogged about it <a href="http://management-platform.blogspot.com/2011/01/bundle-provisioning-via-rhq.html" target="_blank">here</a> and <a href="http://management-platform.blogspot.com/2011/06/deploying-rhq-bundles-to-non-platform.html" target="_blank">here</a> if you want to read more about it.<br />
<br />
<a href="http://pilhuhn.blogspot.com/2013/09/rhq-49-released.html" target="_blank">RHQ 4.9 has just been released</a> and with it comes a new feature in the Bundle subsystem. RHQ can now allow your admins to give users fine-grained security constraints around the Bundle subsystem.<br />
<br />
In the older RHQ versions, it was an all-or-nothing prospect - a user either could do nothing with respect to bundles or could do everything.<br />
<br />
Now, users can be granted certain permissions surrounding bundle functionality. For example, a user could be given the permission to create and delete bundles, but that user could be denied permission to deploy those bundles anywhere. A user could be restriced in such a way to allow him to deploy bundles only to a certain group of resources but not others.<br />
<br />
Along with the new permissions, RHQ has now introduced the concept of "bundle groups." Now you can organize your bundles into separate groups, while providing security constraints around those bundles so only a select set of users can access, manipulate, and deploy bundles in certain bundle groups.<br />
<br />
If you want all the gory details, you can read the <a href="https://docs.jboss.org/author/display/RHQ/Security+Model+for+Bundle+Provisioning" target="_blank">wiki documentation</a> on this new security model for bundles. <br />
<br />
I put together <a href="https://docs.jboss.org/author/display/RHQ/Demo-BundlePermissions" target="_blank">a quick, 15-minute demo</a> that illustrates this fine-grained security model. It demonstrates the use of the bundle permissions to implement a typical use-case that demarcates workflows to provision different applications to different environments:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/fJ24_NY4czqWAJwe7sdsYztJoETyomt9216RrVHCBfVP025_0ZdCjL8x9icIWSFeJQ=s1600" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/fJ24_NY4czqWAJwe7sdsYztJoETyomt9216RrVHCBfVP025_0ZdCjL8x9icIWSFeJQ=s1600" height="246" width="640" /></a></div>
Watch the demo to see how this can be done. The demo will illustrate how the user "HR Developer" will only be allowed to create bundles and put them in the "HR Applications" bundle group and the user "HR Deployer" will only be allowed to deploy those "HR Applications" bundles to the "HR Environment" resource group.<br />
<br />
Again, read the wiki for more information. The <a href="https://docs.jboss.org/author/display/RHQ/Release+Notes+4.9.0" target="_blank">RHQ 4.9 release notes</a> also has information you'll want to read about this. John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com0tag:blogger.com,1999:blog-6944050187568241059.post-54674891080547689552013-08-12T13:14:00.000-07:002013-08-12T13:14:00.160-07:00Moving from Eclipse to IntelliJWell, the second shoe dropped. The final straw was placed on the camel's back and the camel's back broke. I tried one more time and, once again, Eclipse still doesn't have a good Maven integration - at least for such a large project as RHQ.<br />
<br />
Now, for some history, I've been using Eclipse for at least a decade. I like it. I know it. I'm comfortable with it. While I can't claim to know how to use everything in it, I can navigate around it pretty good and can pump out some code using it.<br />
<br />
However, the Maven integration is just really bad from my experience. I've tried, I really have. In fact, it has been an annual ritual of mine to install the latest Maven plugin and see if it finally "just works" for me. I've done this for at least the last three years if not longer. So it is not without a lack of trying. Every year I keep hearing "try it again, it got better." (I really have heard this over the span of years). But every time I install it and load in the RHQ project, it doesn't "just work". I tried it again a few weeks ago and nothing has changed. What I expect is to import my root Maven module and have Eclipse load it in and let me just go back to doing my work. Alas, it has never worked.<br />
<br />
I hate to leave Eclipse because, like I said, I have at least a decade invested in using it. But I need a good Maven integration. I don't want to have tons of Eclipse projects in my workspace - but then again, if the Eclipse Maven plugin needs to create one project per Maven module so it "just works", so be it. I can deal with it (after all, IntelliJ has tons of modules, even if it places them under one main project). But I can't even get that far.<br />
<br />
So, after hearing all the IntelliJ fanboys denigrate Eclipse and tell me that I should move to IntelliJ because "it's better", I finally decided to at least try it.<br />
<br />
Well, I can at least report that IntelliJ's Maven integration actually does seem to "just work" - but that isn't to say I didn't have to spend 15 minutes or so figuring out some things to get it to work (I had to make sure I imported it properly and I had to make sure to set some options). But spending 15 minutes and getting it to work is by far better than what I've gone through with Eclipse (which is, spending lots more time and never getting it to work over the years). So, yes, I can confirm that the IntelliJ folks are correct that Maven integration "just works" - with that small caveat. It actually is very nice.<br />
<br />
In addition, I really like IntelliJ's git integration - it works out of box and has some really nice features.<br />
<br />
I also found that IntelliJ provides an Eclipse keymap - so, while I may not like all the keystrokes required to unlock all the features in IntelliJ (more on that below), I do like how I can use many of the Eclipse keystrokes I know and have it work in IntelliJ.<br />
<br />
As I was typing up this blog, I was about to rail on IntelliJ about its "auto-save" feature. Reading their <a href="http://www.jetbrains.com/idea/documentation/migration_faq.html" target="_blank">Migration FAQ</a> they make it sound like you can't turn off that auto-save feature (where, as soon as you type, it saves the file). I really hate that feature. But, I just found out, to my surprise, you can kinda turn that off. It still maintains the changes though, in what I suppose is a cache of changed files. So if I close the editor with the changed file, and open it back up again, my changes are still there. That's kinda annoying (but yet, I can see this might be useful, too!). But at least it doesn't change the source file. I'll presume there is a way to throw away these cached changes - at least I can do a git revert and that appears to do it.<br />
<br />
<br />
However, with all that said, as I use IntelliJ (and really, it's only been about week), I'm seeing on the edges of it things that I do not like where Eclipse is better. If you are an IntelliJ user and know how to do the following, feel free to point out my errors. Note: I'm using the community version of IntelliJ v12.14.<br />
<br />
For one thing, where's the Problems View that Eclipse has? I mean, in Eclipse, I have a single view with all the compile errors within my project. I do not see anywhere in IntelliJ a single view that tells me about problems project-wise. Now, I was told that this is because Eclipse has its own compiler and IntelliJ does not. That's an issue for me. I like being able to change some code in a class, and watch the Problems View report all the breakages that that change causes. I see in the Project view, you can limit the scope to problem files. That gets you kinda there - but I want to see it as a list (not a tree) and I want to see the error messages themselves, not just what files have errors in them.<br />
<br />
Second, the Run/Debug Configuration feature doesn't appear to be as nice as Eclipse. For example, I have some tool configurations in Eclipse that, when selected, prompt the user for parameter values, but apparently, <a href="http://stackoverflow.com/questions/7675705/does-intellij-support-variables-in-launch-configurations" target="_blank">IntelliJ doesn't support this</a>. In fact, Eclipse supports lots of parameter replacement variables (${x}) whereas it doesn't look like IntelliJ supports any.<br /><br />
Third, one nice feature in Eclipse is the ability to have the source code for a particular method to popup in a small window when you hover over a method call while holding down, say, the ALT key (this is configurable in Eclipse). But, I can't see how this is done in IntelliJ. I can see that View->QuickDefinition does what I want, but I just want to hold down, say, ALT or SHIFT and have the quick definition popup where I hover. I have a feeling you can tell IntelliJ to do this, I just don't know how.<br />
<br />
Another thing I am missing is an equivalent to Eclipse's "scrapbook" feature. This was something I use(d) all the time. In any scrapbook page, you can add and highlight any Java snippet and execute it. The Console View shows the output of the Java snippet. This is an excellent way to quickly run some small code snippet you want to try out to make sure you go it right (I can't tell you how many times I've used it to test regex's). The only way it appears you can do this in IntelliJ is if you are debugging something and you are at a breakpoint. From there, you can execute random code snippets. But Eclipse has this too (the Display view). I want a way to run a Java snippet right from my editor without setting up a debug session.<br />
<br />
I also don't want to see this "TODO" or "JetGradle" or other views that IntelliJ seems to insist I want. You can't remove them from the UI entirely.<br />
<br />
Finally, IntelliJ seems to be really keen on keyboard control. I am one of those developers that hates relying on keystrokes to do things. I am using a GUI IDE, I want to use the GUI :-) I like mouse/menu control over keystrokes. I just can't remember all the many different key combinations to do things, plus my fingers can't consistently reach all the F# function keys, but I can usually remember where in the menu structure a feature is. I'm sure as I use IntelliJ more that I'll remember more. And most everything does seem to have a main menu or popup-menu equivalent. So, this is probably just a gripe that I have to spend time on a learning curve to learn a new tool - can't really blame IntelliJ for that (and with the Eclipse keymap, lots of Eclipse keystrokes now map in IntelliJ). I guess I have to blame Eclipse for that since it's forcing me to make this move in the first place.<br />
<br />
Some of those are nit-picky, others not. And I'm sure I'll run into more things that either IntelliJ doesn't have or is hiding from me. Maybe as I use IntelliJ more, and my ignorance of it recedes a bit, I'll post another blog entry to indicate my progress.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com8tag:blogger.com,1999:blog-6944050187568241059.post-63902041442867781612013-05-08T19:38:00.000-07:002013-05-08T19:38:08.901-07:00Creating Https Connection Without javax.net.ssl.trustStore PropertyQuestion: How can you use the simple <span style="font-family: "Courier New",Courier,monospace;">Java API call java.net.URL.openConnection()</span> to obtain a secure HTTP connection without having to set or use the
global system property "javax.net.ssl.trustStore"? How can you make a secure HTTP connection and
not even need a truststore?<br />
<br />
I will show you how you can do both below.<br />
<br />
First, some background. Java has a basic API to make a simple HTTP connection to any URL via <span style="font-family: "Courier New",Courier,monospace;">URL.openConnection()</span>. If your URL uses the "http" protocol, it is very simple to use this to make basic HTTP connections.<br />
<br />
Problems creep in when you want a secure connection over SSL (via the "https" protocol). You can still use that API - <span style="font-family: "Courier New",Courier,monospace;">URL.openConnection()</span> will return a <span style="font-family: "Courier New",Courier,monospace;">HttpsURLConnection</span> if the URL uses the https protocol - however, you must ensure your JVM can find and access your truststore in order to authenticate the remote server's certificate.<br />
<br />
<i>[note: I won't discuss how you get your trusted certificates and how you put them in your truststore - I'll assume you know, or can find out, how to do this.]</i><br />
<br />
You tell your JVM where your truststore is by setting the system property "javax.net.ssl.trustStore"
and you tell your JVM how to access your truststore by giving your JVM the password via the system property "javax.net.ssl.trustStorePassword".<br />
<br />
The problem is these are global settings (you often see instructions telling you to set these values via the -D command line arguments when starting your Java process) so everything running in your JVM must use that truststore. And you can't alter those system properties during runtime and expect those changes to take effect. Once you ask the JVM to make a secure connection, those system property values appear to be cached in the JVM and are used thereafter for the life of the JVM (I don't know exactly where in the JRE code these values are cached, but my experience shows me that they are). Changing those system properties later on in the lifetime of the JVM has no effect; the original values are forever used.<br />
<br />
Another problem that some people run into is having the need for a truststore in the first place. Sometimes you don't have a requirement to authenticate the server endpoint; however, you would still like to send your data encrypted over the wire. You can't do this readily since the connection you obtain from <span style="font-family: "Courier New",Courier,monospace;">URL.openConnection()</span> will, by default, expect to use your truststore located at the path pointed to by the system property javax.net.ssl.trustStore.<br />
<br />
To allow me to use different truststores for different connections, or to allow me to encrypt a connection but not authenticate the endpoint, I wrote a Java utility object that allows you to do just this.<br />
<br />
The main constructor is this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">public SecureConnector(String secureSocketProtocol,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> File truststoreFile,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> String truststorePassword,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> String truststoreType,</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> String truststoreAlgorithm)</span>
<br />
<br />
You pass it a secure socket protocol (such as "TLS") and your truststore file location. If the truststore file is null, the SecureConnector object will assume you do not want to authenticate the remote server endpoint and you only want to encrypt your over-the-wire traffic. If you do provide a truststore file, you need to provide its password, its type (e.g. "JKS"), and its algorithm (e.g. "SunX509") - if you pass in null for type and/or algorithm, the JVM defaults are used.<br />
<br />
Once you create the object, just obtain a secure connection to any URL via a call to <span style="font-family: "Courier New",Courier,monospace;">SecureConnector.openSecureConnection(URL)</span>. This expects your URL to have a protocol of "https". If successful, an <span style="font-family: "Courier New",Courier,monospace;">HttpsURLConnection</span> object is returned and you can use it like any other connection object. You do not need to set javax.net.ssl.trustStore (or any other javax.net.ssl system property) and, as explained above, you don't even need to provide a truststore at all (assuming you don't need to do any authentication).<br />
<br />
The code for this is found inside of RHQ's agent - <a href="https://git.fedorahosted.org/cgit/rhq/rhq.git/tree/modules/enterprise/agent/src/main/java/org/rhq/enterprise/agent/SecureConnector.java#n37">you can read its javadoc and look through SecureConnector code here</a>.<br />
<br />
The core code is found in <span style="font-family: "Courier New",Courier,monospace;">openSecureConnection</span> and looks like this, I'll break it down:<br />
<br />
First, it simply obtains the HTTPS connection object from the URL itself:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();</span></blockquote>
Then it prepares a custom <span style="font-family: "Courier New",Courier,monospace;">SSLContext</span> object using the given secure socket protocol:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">TrustManager[] trustManagers;<br />SSLContext sslContext = SSLContext.getInstance(getSecureSocketProtocol());</span></blockquote>
If no truststore file was provided, it will build its own "no-op" trust manager and "no-op" hostname verifier. What these "no-op" objects will do is always accept all certificates and hostnames thus they will always allow the SSL communications to flow. This is how the authentication is by-passed:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">if (getTruststoreFile() == null) {<br /> // configured to not care about authenticating server, encrypt but don't worry about certificates<br /> trustManagers = new TrustManager[] { NO_OP_TRUST_MANAGER };<br /> connection.setHostnameVerifier(NO_OP_HOSTNAME_VERIFIER);</span></blockquote>
If a truststore file was provided, then it will be loaded in memory and stored in a <span style="font-family: "Courier New",Courier,monospace;">KeyStore</span> instance:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">} else {<br /> // need to configure SSL connection with truststore so we can authenticate the server.<br /> // First, create a KeyStore, but load it with our truststore entries.<br /> KeyStore keyStore = KeyStore.getInstance(getTruststoreType());<br /> keyStore.load(new FileInputStream(getTruststoreFile()), getTruststorePassword().toCharArray());</span></blockquote>
The truststore file's content (now stored in a <span style="font-family: "Courier New",Courier,monospace;">KeyStore</span> object) is used to initialize a trust manager. Unlike the "no-op" trust manager that was created above (if a truststore file was not provided), this trust manager really does perform authentication and it uses the provided truststore's certificates to authorize the server being communicated with. This is why we no longer need to worry about the system properties "javax.net.ssl.trustStore" and "javax.net.ssl.trustStorePassword" - this builds its own trust manager using the data provided by the caller:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"> // create truststore manager and initialize it with KeyStore we created with all truststore entries<br /> TrustManagerFactory tmf = TrustManagerFactory.getInstance(getTruststoreAlgorithm());<br /> tmf.init(keyStore);<br /> trustManagers = tmf.getTrustManagers();<br />}</span></blockquote>
Finally, the SSL context is initialized with the trust manager that was created earlier (either the "no-op" trust manager, or the trust manager that was initialized with the truststore's certificates). That SSL context is handed off to the SSL connection so the connection can use the context when it needs to perform authentication:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">sslContext.init(null, trustManagers, null);<br />connection.setSSLSocketFactory(sslContext.getSocketFactory());</span></blockquote>
The connection is finally returned to the caller, fully configured and ready to be used.<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">return connection;</span></blockquote>
This is helpful for certain use cases. First, it is helpful when you have multiple truststores that you need to choose from when connecting to different servers as well as being able to switch truststores at runtime (remember, the system property values of javax.net.ssl.trustStore, et. al. are fixed for the lifetime of the JVM - this helps bypass that restriction). This is also helpful in local testing, debugging and demo scenarios when you don't really need or care about setting up truststores and certificates but you do want to connect over https.<br />
<br />John Mazzhttp://www.blogger.com/profile/11415685873835789040noreply@blogger.com2