Tuesday, July 12, 2016

Collecting Prometheus Data and Storing in Hawkular

My last blog entry talked about how to collect JMX data via Jolokia and store that data in Hawkular. Another relatively unknown feature similar to that, which I will now describe in this blog entry, is the ability to collect Prometheus metric data and store that in Hawkular.

Prometheus is itself a metric collection and storage system. However, Prometheus data endpoints (that is, endpoints that emit Prometheus metric data) follow a specific format for the emitted metric data. The Hawkular WildFly Agent has the ability to parse this Prometheus metric data format and push that metric data for storage into Hawkular which can then, of course, be used by Hawkular and its clients (for the purposes of graphing the metric data, alerting on the metric data, etc.).

I will explain how you can quickly get Hawkular to collect metric data from any Prometheus endpoint and store that data into Hawkular.

First, you need a Prometheus endpoint that emits metric data! For this simple demo, I simply ran the Prometheus Server which itself emits metrics. I did this via docker:

1. Make sure your docker daemon is running: sudo docker daemon
2. Run the Prometheus Server docker image: sudo docker run -p 9090:9090 prom/prometheus

That's it! You now have a Prometheus Server running and listening on port 9090. To see the metrics it, itself, emits, go to http://localhost:9090/metrics. We'll be asking the Hawkular WildFly Agent to collect these metrics and push them into a Hawkular Server.

Second, you need to run a Hawkular Server. I won't go into details here on how to do that. Suffice it to say, either build or download a Hawkular Server distribution and run it (if it is not a developer build, make sure you run your external Cassandra Server prior to starting your Hawkular Server - e.g. sudo docker run cassandra -p 9042:9042).

Now you want to run a Hawkular WildFly Agent to collect some of that Prometheus metric data and store it in the Hawkular Server. In this demo, I'll be running the Swarm Agent, which is simply a Hawkular WildFly Agent packaged in a single jar that you can run as a standalone app. However, its agent subsystem configuration is the same as if you were running the agent in a full WildFly Server so the configuration I'll be describing can be used no matter how you have deployed your agent.

Rather than rely on the default configuration file that comes with the Swarm Agent I extracted the default configuration file and edited it as I describe below.

I deleted all the "-dmr" related configuration settings (metric-dmr, resource-type-dmr, remote-dmr, etc). I want my agent to only collect data from my Prometheus endpoint, so no need to define all these DMR metadata. (NOTE: the Swarm Agent configuration file does already, out-of-box, support Prometheus. I will skip that for this demo - I want to explicitly explain the parts of the agent configuration that is needed to be in the configuration file).

The Prometheus portion of the agent configuration is very small. Here it is:
<managed-servers>
  <remote-prometheus name="My Prometheus Endpoint"
                     enabled="true"
                     interval="30"
                     time-units="seconds"
                     metric-tags="feed=%FeedId"
                     metric-id-template="%FeedId_%MetricName"
                     url="http://127.0.0.1:9090/metrics"/>
</managed-servers>
That's it. <remote-prometheus> tells the agent where the Prometheus endpoint is and how often to pull the metric data from it. Every metric emitted by that Prometheus endpoint will be collected and stored in Hawkular.

Notice I can associate my managed server with metric tags (remote-prometheus is one type of "managed server"). For every metric that is collected for this remote-prometheus managed server, those tags will be added to those metrics in the Hawkular Server (specifically in the Hawkular Metrics component). All metrics will have these same tags. In addition, any labels associated with the emitted Prometheus metrics (Prometheus metric data can have name/value pairs associated with them - Prometheus calls these labels) will be added as Hawkular tags. Similarly, the ID used to store the metrics in the Hawkular Metric component can also be customized. Both metric-tags and metric-id-template are optional. You can also place those attributes on individual metric definitions (which I describe below) which is most useful if you have specific tags you want to add only to metrics of a specific metric type but not on all of the metrics collected for your managed server.

If you wish to have the agent only collect a subset of the metrics emitted by that endpoint, then you must tell the agent which metrics you want collected. You do this via metric sets:
<metric-set-prometheus name="My Prometheus Metrics">
  <metric-prometheus name="http_requests_total" />
  <metric-prometheus name="go_memstats_heap_alloc_bytes" />
  <metric-prometheus name="process_resident_memory_bytes" />
</metric-set-prometheus>
Once you defined your metric-set-prometheus entries, you specify them in the metric-sets attribute on your <remote-prometheus> element (e.g. metric-sets="My Prometheus Metrics").

OK, now that I've got my Swarm Agent configuration in place (call it "agent.xml" for simplicity), I can run the agent and point it to my configuration file: 
java -jar hawkular-swarm-agent-dist-*-swarm.jar agent.xml
At this point I have my agent running along side of my Hawkular Server and the Prometheus Server which is the source of our metric data. The agent is collecting information from the Prometheus endpoint and pushing the collected data to Hawkular Server.

In order to visualize this collected data, I'm using the experimental developer tool HawkFX. This is simply a browser that let's you see what data is in Hawkular Inventory as well as Hawkular Metrics. When I log in, I can see all the metric data that comes directly from the Prometheus endpoint.



I can select a metric to see its graph:



If I were to have configured my agent to only collect a subset of metrics (as I had shown earlier), I would see only those metrics that I asked to collect - all the other metrics emitted by the Prometheus endpoint are  ignored:

 



What this all shows is that you can use Hawkular WildFly Agent to collect metric data from a Prometheus endpoint and store that data inside Hawkular.

Collecting JMX Data and Storing in Hawkular

The Hawkular WildFly Agent has the ability to not only monitor WildFly Servers but also JMX MBean Servers via Jolokia (and also Prometheus endpoints, but that's for another blog - let's focus on Jolokia-enabled JMX MBean Servers for now).

What this means is if you have a Jolokia-enabled server, you can collect JMX data from it and store that data in Hawkular. This includes both metric data as well as resource information.

This blog will attempt to quickly show how this is done.

First, you need a Jolokia-enabled server! For this demo, here's the quick steps I did to get this running on my box:

1. I downloaded WildFly 10.0.0.Final and unzipped it.
2. I downloaded the latest Jolokia .war file and copied it to my new WildFly Server's standalone/deployments directory
3. I started my WildFly Server and bound it to some IP address that I dedicated to it (in this case, it was simply a loopback IP that I dedicated to this WildFly Server):

bin/standalone.sh -b 127.0.0.5 -bmanagement=127.0.0.5

At this point, I now have a server with some JMX data exposed over the Jolokia endpoint.

Second, I need to run a Hawkular Server. I won't go into details here on how to do that. Suffice it to say, either build or download a Hawkular Server distribution and run it (if it is not a developer build, make sure you run your external Cassandra Server prior to starting your Hawkular Server - e.g. sudo docker run cassandra -p 9042:9042).

Now I want to run a Hawkular WildFly Agent to collect some of that JMX data from my WildFly Server and store it in Hawkular Server. In this demo, I'll be running the Swarm Agent, which is simply a Hawkular WildFly Agent packaged in a single jar that you can run as a standalone app. However, its agent subsystem configuration is the same as if you were running the agent in a full WildFly Server so the configuration I'll be describing can be used no matter how you have deployed your agent.

Rather than rely on the default configuration file that comes with the Swarm Agent (which is designed to collect and store data from a WildFly Server's DMR management endpoint, not Jolokia) I extracted the default configuration file and edited it as I describe below.

I deleted all the "-dmr" related configuration settings (metric-dmr, resource-type-dmr, remote-dmr, etc). I want my agent to only collect data from my Jolokia endpoint, so no need to define all these DMR metadata.

I then added metadata that describes the JMX data I want to collect. For example, I collect availability metrics (to tell me if an MBean is available or not) and gauge metrics (to graph things like used memory). I also collect resource properties that some MBeans expose as JMX attributes. I assign these to different resources by defining resource metadata which point to specific JMX MBean ObjectNames. I then define the details of my Jolokia-enabled WildFly Server in a <remote-jmx> so my agent knows where my Jolokia-enabled WildFly Server is.

Some example configuration is:
<avail-jmx name="Memory Pool Avail"
           interval="30"
           time-units="seconds"
           attribute="Valid"
           up-regex="[tT].*"/>
This defines an availability metric that says for the attribute "Valid" if its value matches the regex "[tT].*" consider its availability UP (note this regex matches the string "true", case-insensitive), otherwise it is DOWN. We will attach this availability metric to a resource below.

Here is an example of a gauge metric:
<metric-jmx name="Pool Used Memory"
            interval="1"
            time-units="minutes"
            metric-units="bytes"
            attribute="Usage#used"/>
Notice the data can come from a sub-attribute of a composite value (the "used" value within the composite attribute "Usage").

You group availability metrics and numeric metrics in metric sets (avail-set-jmx and metric-set-jmx respectively) and then associate those metric sets to specific resource types. Resource types define resources that you want to monitor (resources in JMX are simply MBeans identified with ObjectNames). For example, in my demo, I want to monitor my Memory Pools. So I create a resource type definition that describe the Memory Pools:
<resource-type-jmx name="Memory Pool MBean"
         parents="Runtime MBean"
         resource-name-template="%type% %name%"
         object-name="java.lang:type=MemoryPool,name=*"
         metric-sets="MemoryPoolMetricsJMX"
         avail-sets="MemoryPoolAvailsJMX"
  <resource-config-jmx name="Type"

                       attribute="Type"/>
</resource-type-jmx>
Here you can see my resource type "Memory Pool Bean" refers to all resources that match the JMX query "java.lang:type=MemoryPool,name=*". For all the resources that match that query, I associate with them the availability and numeric metrics defined in the sets mentioned in the metric-sets and avail-sets attributes. I also want a resource configuration property collected for each resource - "Type". Each Memory Pool MBean has a Type attribute that we want to collect and store. Notice also that all of these resources are to be considered children of the parent resource whose resource type name is "Runtime MBean" (which I defined elsewhere in my configuration).

Once all of my metadata is configured (I've configured the agent to collect all the configuration properties, availability and numeric metrics of all the JMX MBeans I want), I now configure the agent with the Jolokie-enabled endpoint. This tells the agent how to connect to the Jolokia endpoint and what I want the agent to monitor in that endpoint:
<remote-jmx name="My Remote JMX"
  enabled="true"
  resource-type-sets="MainJMX,MemoryPoolJMX"
  metric-tags="server=%ManagedServerName,host=127.0.0.5"
  metric-id-template="%ResourceName_%MetricTypeName"
  url="http://127.0.0.5:8080/jolokia-war-1.3.3"/>
Here I configure the URL endpoint of my WildFly Server's Jolokia war. I then tell it what resource types I want to monitor in that Jolokia endpoint (I've grouped all my resource types into two different resource type sets called MainJMX and MemoryPoolJMX). The grouping is all up to you - if you want one big resource type set, that's fine. For metrics, you can have one big availability metric set and one big numeric metric set - it doesn't matter how you organize your sets.

One final note before I run the agent. Notice I can associate my managed server with metric tags (remote-jmx is one type of "managed server"). For every metric that is collected for this remote-jmx managed server, those tags will be added to those metrics in the Hawkular Server (specifically in the Hawkular Metrics component). So for the "Pool Used Memory" metric I defined earlier, when that metric is stored in the Hawkular Server, it will be tagged with "server" and "host" where the values of those tags are the name of my managed server (in this case, %ManagedServerName is replaced with "My Remote JMX") and "127.0.0.5" respectively. All metrics will have the same tags. Similarly, the ID used to store the metrics in the Hawkular Metric component can also be customized though this is a rarely used feature and you probably will never need it. Both metric-tags and metric-id-template are optional. You can also place those attributes on individual metrics which is most useful if you have specific tags you want to add only to metrics of a specific metric type but not on all of the metrics collected for your managed server.

OK, now that I've got my Swarm Agent configuration in place (call it "agent.xml" for simplicity), I can run the agent and point it to my configuration file:
java -jar hawkular-swarm-agent-dist-*-swarm.jar agent.xml
At this point I have my agent running along side of my Hawkular Server and WildFly Server enabled with Jolokia. The agent is collecting information from Jolokia and pushing the collected data to Hawkular Server.

In order to visualize this collected data, I'm using the experimental developer tool HawkFX. This is simply a browser that let's you see what data is in Hawkular Inventory as well as Hawkular Metrics. When I log in, I can see all the resources stored in Hawkular that comes directly from Jolokia - these resources represent the different JMX MBeans we asked the agent to monitor:



You can see "My Remote JMX Runtime MBean" is my parent resource, it has one availability metric "VM Avail", three numeric metrics and six child resources (those are the Memory Pool resources described above when we added them to the configuration).

You can drill down and see the metrics associated with the children as well. For example, the Memory Pool for the PS Old Gen has a metric "Pool Used Memory" that we can graph (the metric data was pulled from Jolokia, stored in Hawkular Metrics, which is then graphed by HawkFx as you see here):



Finally, you can use HawkFx to confirm that the resource configuration properties were successfully collected from Jolokia and stored into Hawkular. For example, here you can see the "Type" property we configured earlier - the type of this memory pool is "HEAP".

 

What this all shows is that you can use Hawkular WildFly Agent to collect resource information and metric data from JMX over a Jolokia endpoint and store that data inside Hawkular.

Monday, April 18, 2016

Prometheus Metric Endpoint Parser for Java

I have a need for a Java-based parser that can parse metric data from any Prometheus endpoint.

Prometheus has two main data formats - a binary format and a text format. You can read about those formats here. That document says that "Clients must support at least one of these two alternate formats." So I needed a Java-based parser that can parse both.

The Prometheus team has published parsers for several different languages (e.g. C++, Go, Python, Ruby, and Java). Some only support the binary formats, Java being one of those with only binary support. In addition, the Prometheus team may delete the Java parser entirely since it is relatively unused by the community. As of this writing, the latest release of the Prometheus Java parser is version 0.0.2 from July 2013 which also doesn't support histograms (though version 0.0.3-SNAPSHOT in the master branch does support it - so if/when 0.0.3 is released, histogram support will be avaialble).

So I needed to write my own Java-based parser for the text format to ensure I could read any Prometheus metric endpoint (even though the documentation says clients must support one or the other, in practice it seems all endpoints support the text format and only some (mainly Go endpoints) support the binary format). So even if the Java-based binary parser support goes away, having a text parser should still be able to read all Prometheus endpoints (in other words, those endpoints with binary-format support should also have text-format support as well).

Here is my Java-based Prometheus Metrics Scraper code. There is a README for a quick synopsis. It supports both binary and text formats and utilizes content negotiation with the URL endpoint to determine what format to expect. You can also programatically process files as opposed to URL endpoints.

This Prometheus Metrics Scraper comes with a CLI that you can run via a simple Java command:
java -jar prometheus-scraper*-cli.jar [--simple | --xml | --json] {url}
It can output any URL endpoint's metric data in several formats (JSON and XML being the two more interesting ones). If you'd like to try it out, grab the latest release from here and run it. For example, you can download the 0.17.1Final CLI jar here.

Programmatically you use this by simply passing a URL (or File) to PrometheusScraper and calling its scrape() method. This will return a list of MetricFamily objects, which contain all the metric data found in the endpoint URL.

See the code's Javadoc for more complete documentation.

There are a few things still missing that would be nice to enhance for the future.

First is histogram support for binary formatted data (but once the jar artifact "io.prometheus.client:model" version 0.0.3 is released by the Prometheus team, it would just be a matter of uncommenting one block of code for my Java-based parser to begin supporting it). Of course, histograms are fully supported in the text parser.

Secondly, the URL endpoint is assumed to be unsecured. If SSL certificates or authentication is required to access the metric data over the given URL, the scraper will fail to process the data.

Thursday, January 21, 2016

Hawkular Command Gateway Clients

This document is to briefly explain how clients can use the command gateway to send requests to the Hawkular Server and receive responses. The typical use case (though certainly not the only way to use this feature) is for a browser to send requests to Hawkular WildFly Agents routed through the Hawkular Server, with the agent’s responses routed back to the browser (back through the server intermediary) that sent the request.

Brief Summary

The typical workflow is the following:
  1. Client makes a websocket connection to the server.
  2. Client immediately receives a WelcomeResponse JSON message from the server notifying the client what its session ID is.
  3. Client can send JSON requests over the Web Socket connection.
  4. Client can receive JSON responses over the Web Socket connection. These messages are received asynchronous from its requests and may not even be tied to a specific request.
  5. Client can keep the connection open as long as it wants, and can disconnect when it wants.

Make the Connection

Clients first need to make a WebSocket connection to the Hawkular Server. That is done by connecting to the URL: "ws://:8080/ui/ws".

Note that a secure connection via SSL can be made by connecting to “wss://:8443” but this requires the server to be configured properly with a certificate and a security realm defined. Such details are out of scope for this document. For more, see http://www.hawkular.org/docs/user/secure-comm.html

Receiving the Welcome Message

For each client websocket connection that is made, the server sends an initial WelcomeResponse message immediately to it. This message contains the client’s session ID. This is the session ID that the server will use to identify that client. This session ID is actually not needed by the client today, but it is available for future functionality. For now, clients can actually ignore this session ID.

An example WelcomeResponse message that a client could receive is:

WelcomeResponse={“sessionId”:”abc:123:xyz:789”}

The WelcomeResponse JSON schema is defined here.

Sending a Request Message

Once connected, clients can immediately begin sending any valid request over the web socket connection. Because clients are able to send in different kinds of requests, the client must ensure the request message’s JSON content is prefixed with the JSON schema name followed by “=” . Valid JSON schemas can be found here.

Within the JSON content, there must be an authentication node containing the credentials of the client. The authentication node’s JSON schema is defined here.

So, for example, if you want to execute the “Undeploy” operation on a web deployment resource, the request sent over the web socket connection will look something like this:

ExecuteOperationRequest={“authentication”:{“username”:”joe”,”password”:”pw”},“operationName”:”Undeploy”,”resourcePath”:”/t;tenant-id/e;env-id/r;resource-id-of-the-deployment-resource”}

Note that in order for a client to request something related to a specific resource in inventory, that client must know the resource’s “resource path” as defined in Hawkular Inventory. See the Hawkular Inventory REST API documentation for more details on how to obtain inventory data such as these resource paths.

Receiving a Response Message

The clients can receive messages in the same format as it sent them. In other words, the JSON content received by the client will be prefixed with the JSON schema name that is to be used to properly parse the JSON content received.

For example, when a client sends a request message destined to an agent, the server will immediately send a GenericSuccessResponse back to the client. This means the request was received by the server and has been successfully forward to the agent (note: it does not mean the request was successfully processed by the agent - the agent will send its own success or failure response message back once it processes the original request). Such a response message received by the client would look like this:

GenericSuccessResponse={“message”:”The request has been forwarded to the feed [foo]”}

Sending Binary Content

Sometimes a client needs to send raw binary data as part of a request (e.g. when a client wants to deploy a web application, it sends a DeployApplicationRequest along with the application’s .war file). To do this, the client sends the JSON message as usual (i.e. in the form “json-schema={json content}”) but immediately following the final curly brace of the JSON content, the client should stream the binary content.

Friday, January 1, 2016

Hawkular WildFly Agent API For Your Own Inventory and Metrics

The Hawkular WildFly Agent is well into development and is coming along nicely. It provides a way to monitor one or more WildFly or EAP application servers (including the one it is deployed into). It communicates with a Hawkular Server where the agent stores its inventory and metric data via the Hawkular Inventory and Hawkular Metrics components. You can use the Hawkular GUI to interact with your managed application servers: view historical graphs of metric data, deploy and undeploy applications, etc.

But the Hawkular WildFly Agent provides a hidden gem that might be useful to those developers that want to store metrics in a metric storage facility for later reporting and graphing but don't want to take the time to implement that storage facility. This hidden gem also provides a way for developers to store their own managed resource definitions in an inventory storage facility but, again, don't want to implement all of the backend required for such a thing.

The Hawkular WildFly Agent already integrates with Hawkular Inventory and Hawkular Metrics - this is to enable the agent to be able to store its own inventory and metrics. It makes sense to open that up for applications to use so they, too, can store inventory and metrics. To allow for this, Hawkular WildFly Agent stores a helpful object in JNDI called HawkularWildFlyAgentContext under the name "java:global/hawkular/agent/api" (side note: the agent can be told to bind this object to a different JNDI name if you want; it can also be told to not bind this hidden gem in JNDI at all if you do not wish to expose this feature).

The API this exposes is very simple - you can store or remove resources from inventory, and you can store metric data and availability data (availability data is just a "special" kind of metric that allows you to store "UP", "DOWN" or "UNKNOWN" availability states).

Any application that is deployed in the same WildFly or EAP application server as the agent can use this API by obtaining the agent context object via JNDI. A simple example of how you can obtain this agent context object from JNDI can be seen in a test war that is used in the agent integration tests - its a singleton EJB that gets this context object injected via @Resource. See the HawkularWildFlyAgentProvider class.

Once that HawkularWildFlyAgentContext object is obtained, your application can use it to store inventory, metric, and availability data. Note that you do not have to use all of these. For example, if you just want to store metrics in a historical time-based data store, just use the Metric Storage API that you get from the agent context object. This will send your data for storage to Hawkular Metrics. You could then do whatever you want with your data later - Hawkular Metrics provides a REST interface and some clients that you can use to query and report on your metric data.

The example test war can show you how the API is used to do these things - see the test MyAppServlet.java. This is just for integration testing, so it doesn't do anything earth-shattering, but it does show you how the API is used to create and remove managed resources from inventory, store metric data, and store availability data.

This feature of the Hawkular WildFly Agent is a relatively minor feature considering all the other main requirements that the agent must fulfill, but since it is rather hidden I decided to talk about it here.

Thursday, January 15, 2015

Hawkular Bus

Work under the new Hawkular umbrella has begun. More information about this new project will trickle out more and more from the team members in the coming weeks and months ahead. There's already a Twitter feed and a developer mailing list.

Here I'll briefly discuss the hawkular-bus work I've done.

The idea is that the hawkular-bus provides a framework to hook into a messaging bus including a Wildfly container that provides all the messaging infrastructure setup for you as well as a convienence API that frees you from the annoying JMS boilerplate code that you typically have to write when it comes to developing bus applications and EJB MDBs in Java.

hawkular-bus also has a REST API that allows any client (Java or non-Java) to produce messages on the bus or consume messages off the bus.

The container that you can use to deploy your bus-based components (be they WARs or EARs or Wildfly module extensions) is called a hawkular-nest. It is simply a pre-configured Wildfly server - it provides the necessary module extensions that give you the messaging infrastructure such that all you need to do is deploy WARs and EARs to it and they can immediately take advantage of the hawkular framework. You can deploy your WARs and EARs directly in the nest module extension (modules/system/layers/base/org/hawkular/nest/main/deployments), thus immediately allowing your applications to be patchable using Wildfly's patching mechanism. In the future, we may want to use this deployment to be able to deploy some kind of "Hawkular Plugin" but nothing has been fleshed out. Right now, WAR and EAR deployments make it easy for you to plug into the hawkular message bus. There is even a hawkular-bus-mdb API that provides an API to allow your MDBs to easily process RPC-like messages (that is, your MDB can not only listen for messages on the bus but then immediately return a response back to the client that sent the message).

I then prototyped a hawkular-audit component that uses hawkular-bus. Hawkular-audit is a server-side component that accepts "audit records" over the bus and places them in a backend storage (today, its just an in-memory H2 database - the one that comes with Wildfly). For example, if you have something that you want to track via an audit trail, that something can put its audit records on the bus and have hawkular-audit store it for you. Your client, if it is Java, can use hawkular-bus API to send the message, or if your client is not Java you can have it use the hawkular-bus REST API to put the message on the bus (for example, using curl).

To try out all this hawkular stuff:

1. Clone both hawkular-bus and hawkular-audit repos
2. Build them by just executing "mvn install" from each repo's root directory.
3. Unzip the distribution from hawkular-audit/hawkular-audit-distro/target and run the "standalone.sh" from the Wildfly's bin/ directory.
4. When the hawkular-audit server starts, it will log its own audit event to record when it started. To see the audit records, point your browser to "http://localhost:8080/hawkular-audit/AuditQueryServlet" and you'll see a simple HTML table with the data.
5. To add your own audit records to the audit trail, run curl to send your audit records over the REST API. For example, try something like this:

    curl -d "body={"subsystem":{"MyComponent":"MySubsystem"},"message":"My audit record message","details":{"key2":"value2","key1":"value1"}}" http://localhost:8080/hawkular-bus/message/AuditQueue?type=queue

Refresh your browser using the same URL mentioned in step 4 above and you'll see your new records.

Hawkular-audit is just a prototype that shows how you can build your own server components to utilize the hawkular-bus and hawkular-nest in order to plug into the bus framework and be able to receive (and send) messages off the bus. In the future, we'll want to cluster the bus so multiple components can talk to each other over one main bus and have clients hook into the bus to send messages to one or more of these server components.

Wednesday, November 12, 2014

Messaging Infrastructure using ActiveMQ


What is rhq-msg?

Before I get into the news about the new rhq-msg repository, let me first take a step back and summarize what rhq-msg is and what I'm trying to accomplish with it.

rhq-msg is a simple messaging API built on top of ActiveMQ and JMS. However, I wanted to isolate the user of the API from as much ActiveMQ and JMS specific classes and code as possible. I wanted a simpler API that provides basic messaging functionality without requiring the user from having to create and manage lots of little specific classes (like JMS Connections, Sessions, Destinations, Consumers, Producers, etc) and without having to know very many specific ActiveMQ details.

I also wanted to ensure that non-Java clients and servers can interact with rhq-msg clients and servers. So the messages that are sent and received from the rhq-msg API are JSON-encoded and thus can be sent to and received by other non-Java endpoints so long as they can handle JSON-encoded messages (ActiveMQ supports non-Java clients, I just wanted to make sure the messages rhq-msg handles can easily flow to/from those non-Java clients as well).

So the point is I have small, simple, easier-to-use messaging API that can talk to non-Java endpoints, but yet still retain all the nice functionality (like guaranteed delivery and things like that) that JMS and ActiveMQ provides.

New Repository for rhq-msg

My past few blog posts were about the prototyping work I'm doing with respect to rhq-msg and rhq-audit.

I just split out rhq-msg and put it in its own rhq-msg repository since it really does belong as a separate project.

I also added a nifty feature to it - you can now use it for request-response workflows. Before, the rhq-msg API really only supported fire-and-forget async messaging. You had a message and you sent it to an endpoint asynchronously and that was that. The problem is sometimes you want a response back, and many times you want to wait for that response to come back in an RPC-like fashion (as opposed to accepting the response asynchrously). I call this a request-response workflow.

Well, the API now supports this. In fact, it supports receiving the response both synchronously (via a Java Future implementation) and asynchronously (via a message listener implementation). More on this below.

A Quick Overview of the API

The main purpose of rhq-msg is to provide a simpler API to do messaging (simpler than, say, JMS). If I can't describe how to send and receive messages via rhq-msg in a couple paragraphs of a blog, then I think I failed :) So let me give a quick overview of what the API calls would look like if you want to send and receive messages. I will cover sending fire-and-forget messages, request-response messaging, and listening for incoming messages.

The main rhq-msg API is found in the rhq-msg-common module.

* Common Code For Both Producers and Consumers

Before I talk about how to send and receive messages, let me introduce the few classes both senders (aka producers) and receivers (aka consumers) use.

Each message that flows through rhq-msg is a BasicMessage. You can subclass that to create your own message types, but all messages must derive from BasicMessage. BasicMessage provides the JSON functionality needed to serialize the messages over the wire.

ConnectionContextFactory connects to your rhq-msg broker and creates contexts for both producers and consumers that are then passed to the MessageProcessor so it knows where to send/receive its messages from.

MessageProcessor provides the API to send and listen for messages.

OK, with that out of the way, let's talk about how to send and listen for messages. For these examples, let's assume you have a broker running on the local machine (127.0.0.1) listening to port 17173, you have a BasicMessage you want to send (or you want to listen for a BasicMessage) and the messages are to be found on a queue named "Foo".

* Sending Fire-and-Forget Messages

Fire-and-forget means you send a message off to an endpoint and forget about it - you don't need or expect a response back. You assume the message broker will deliver the message and the recipient will process the message properly.

   Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
   ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
   ProducerConnectionContext context = factory.createProducerConnectionContext(endpoint);
   MessageProcessor processor = new MessageProcessor();
   processor.send(context, basicMessage);
   factory.close();

Here you create your factory so it connects to your broker. Using the factory, create a producer context. With that new producer context and your basic message, just tell a message processor to send it. That's it. Message is away. You can keep your factory around to create additional contexts which share the connection. But once you are done sending messages, close the factory so it cleans up all resources and closes its connection to the broker. Note that every class you see is a rhq-msg object. No ActiveMQ or JMS classes are used to code this up (though, obviously, under the covers, ActiveMQ and JMS is used heavily).

* Sending Request-Response Messages

Many times when you send a message request, you want a response back. In this example I will show how you can have an RPC-like request-response workflow. I will assume that the remote endpoint will process my BasicMessage and send back to me its own BasicMessage response. Note that, as I mentioned earlier, I could use custom message types (I don't have to use BasicMessage) but those custom message types must always derive from BasicMessage.

   Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
   ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
   ProducerConnectionContext context = factory.createProducerConnectionContext(endpoint);
   MessageProcessor processor = new MessageProcessor();
   Future<BasicMessage> future = processor.sendRPC(context, basicMessage, BasicMessage.class);
   BasicMessage response = future.get(30, TimeUnit.SECONDS);

The first several lines are identical with the fire-and-forget example above. The difference starts with the API call made on the processor - here you call sendRPC() passing in the same context and your outgoing basicMessage request object as before, but you also now pass in the message type of the expected response message. In return, you get a Future object which you can use to retrieve the response when it is received. In this example, it blocks for 30 seconds waiting for the response.

There is another API I won't talk about in detail here, but suffice it to say there is another request-response API you can call on the MessageProcessor (as opposed to sendRPC()) and that is "sendAndListen()". sendAndListen() also allows you to send a request and listen for a response, but this API allows you to give your own message listener so it can wait for and receive the response, rather than go through a Future object. It seems more intuitive and easier to use Future, but in case you want to write your own listener object and listen for the response that way, this is doable. I'll explain the listener API below - it would be the same thing you would need to pass to sendAndListen().

* Listening for Incoming Messages

This last example shows how you implement consumers via rhq-msg API. You implement these via listeners. These listeners are your server-side code - they listen for and accept incoming messages from producers and process those messages. There are two general types: those that do not send responses back, and those that do. I'll cover both in that order.

To listen for "fire-and-forget" messages (that is, messages that are sent that do not require a response sent back to the sender) you implement a BasicMessageListener and hand that listener off to the message processor:

   Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
   ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
   ConsumerConnectionContext context = factory.createConsumerConnectionContext(endpoint);
   MessageProcessor processor = new MessageProcessor();
   processor.listen(context, listener);

where "listener" is a subclass implementation of BasicMessageListener. An example is:

   public class MyCustomListener extends BasicMessageListener<BasicMessage> {
       @Override
       protected void onBasicMessage(BasicMessage receivedMessage) {
          // Process the received message.
       }
   }

Notice that we still create a ConnectionContextFactory (because we still need a connection to the message broker) but we ask that factory to create for us a consumer context this time. We call "listen()" on the message processor, passing to it that consumer context and our custom listener. That listener is now listening for messages to arrive on the "Foo" queue and will process them.

What about request-response processing? If your listener needs to send data back to the sender, it needs to implement a RPCBasicMessageListener. Other than that difference, the main code is still the same:

   Endpoint endpoint = new Endpoint(Endpoint.Type.QUEUE, "Foo");
   ConnectionContextFactory factory = new ConnectionContextFactory("tcp://localhost:17173");
   ConsumerConnectionContext context = factory.createConsumerConnectionContext(endpoint);
   MessageProcessor processor = new MessageProcessor();
   processor.listen(context, listener);

but this time "listener" is a subclass implementation of RPCBasicMessageListener. An example is:

   public class MyCustomRPCListener extends RPCBasicMessageListener<BasicMessage, AnotherMessage> {
       @Override
       protected AnotherMessage onBasicMessage(BasicMessage receivedMessage) {
          // Process the received message.
          AnotherMessage response = ...create your response message object...;
          return response;
       }
   }

Note the onBasicMessage() method now can return a non-void type - the response message type. This return type (the specific response message type) is declared in the generic type definition found in your class definition. The first generic type is that of the expected incoming message type, the second generic type is that of the response message type.

How to Build rhq-msg?

rhq-msg is composed of a set of Maven modules. From the parent root module directory, just run "mvn install" and everything will build, tests will run, and artifacts will be packaged.

There are also Eclipse project files that allow you to work with rhq-msg via M2E (the Eclipse Maven plugin).

How to Run a rhq-msg Broker

If you want to run your own code that utilizes the rhq-msg API, you will need to run a rhq-msg broker (which is really just an ActiveMQ broker). You can run a rhq-msg broker easily in a couple ways. You can run the EmbeddedBroker from the Maven command line or you can install the EmbeddedBroker in a WildFly 8 installation. This EmbeddedBroker code is found in the rhq-msg-broker Maven module.

* Running EmbeddedBroker from the command line

Starting a broker as a standalone process is relatively painless. You can use Maven to do it:

   mvn -Prun-test-broker install

Right now the rhq-msg broker is not packaged with all its dependencies, but if you do it yourself (that is, get all the dependency jars and start Java with the appropriate classpath) you can run straight from the Java command line:

   java -cp <the-appropriate-classpath> -jar rhq-msg-broker-0.1.jar -c broker-config.xml

where "broker-config.xml" is an actual ActiveMQ XML configuration file (you can also pass in a simpler ActiveMQ .properties configuration file if you wish). Examples are test-broker.xml and test-broker.properties.

* Installing EmbeddedBroker in WildFly 8

One of the artifacts within rhq-msg is a WildFly Extension Module that provides a broker. Once installed in WildFly, you will have an rhq-msg broker subsystem running within WildFly itself. It is a handy way to have your own broker running in your own WildFly instance. So this means you can have, for example, an rhq-msg client within a web application running in WildFly and provide your own broker for that client running in the same WildFly instance. Technically, this is useful for any ActiveMQ or JMS client - since this broker is nothing more than an ActiveMQ broker and can serve any client from anywhere, not just those using the rhq-msg API.

First, download and unzip the WildFly 8 app server (I have not tested on WildFly 9+). I'll use <wildfly-install-dir> to indicate where you installed WildFly.

Now install the custom rhq-msg broker WildFly Extension Module into WildFly. You can use the new Maven plugin "wildfly-extension-maven-plugin" to do this - the rhq-msg-broker-wf-extension Maven module has integrated that Maven plugin. I talk about this in a previous blog post. So, simply running this Maven command will install your rhq-msg broker WildFly Extension Module for you:

   mvn -Dorg.rhq.msg.broker.wildfly.home=<wildfly-install-dir> wildfly-extension:deploy

Since this is nothing more than a normal subsystem like all the other subsystems in WildFly, you can use the JBoss CLI to look at its configuration. To poke around, run the JBoss CLI in GUI mode ("jboss-cli.sh --gui") and look for the "org.rhq.msg.broker" subsystem.

In Closing

This is a prototype and was developed just to see how a messaging API around JMS and ActiveMQ can be made simpler and easier to use. I am sure I am missing some pieces of functionality (one such missing piece is security - this currently doesn't handle logging into a secured broker). If you see anything missing, let me know. Feel free to suggest corrections or enhancements. All the code is in github, so play around with it and see how useful it is.