JMS and Database Integration with Apache ServiceMix, Camel and ActiveMQ

I've been asked to create a simple project to demonstrate how Apache ActiveMQ, Camel and ServiceMix could be used in the same integration context. Then, for the specific customer I was talking to, I decided to create something related to the world they were used to.

This example will process messages as they are delivered on an input queue and I thought that XML would be a good format to also demonstrate how to parse the payload and generate an SQL statement  before hitting the database.

In summary, for each message going to the input queue, we're going to create a record in the database table and then generate a response message on the output queue.

I'm assuming you're already have Apache ServiceMix downloaded and installed in your machine. If not, it's time to download it… I recommend you to just go to http://fusesource.com/downloads/ and download FUSE ESB.

I'm also assuming that you already have Maven installed but if not feel free to download and install it from maven.apache.org

Here are the steps to create this short and simple demonstration:

1) Download and install MySQL database from http://dev.mysql.com/downloads/
You can use the default test database that's shipped with MySQL and the only thing you have to do is to create a table to insert test records. To do that, connect to the database  and run the following SQL statement:

CREATE table PARTNER_METRIC(partner_id int, time_occurred DATE, status_code int, perf_time long);


2) Install the following features in ServiceMix:

features:install camel-jdbc
features:install camel-jms


3) Install commons-dbcp bundle:

osgi:install -s mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.commons-dbcp/1.2.2_6


4) When you do this you will receive a bundle id, e.g. 229.  Enable dynamic imports on this bundle so that it can pick up the appropriate database driver.

dev:dynamic-import 229


5) Using ServiceMix hot deploy feature to deploy the MySQL JDBC driver:

cp /Applications/MySQL/mysql-connector-java-5.1.18/mysql-connector-java-5.1.18-bin.jar  /Users/mjabali/Fuse/apache-servicemix-4.4.1-fuse-00-08/deploy/


Then you can verify that the driver gets deployed correctly with the following command:

karaf@root> list |grep -i mysql


and you should get a message similar to the following:

[ 227] [Active     ] [            ] [       ] [   60] Sun Microsystems' JDBC Driver for MySQL (5.1.18)


6) Download the source files of this sample and extract them on the directory of your choice. So, let's review the JMS and Database connection settings  in the following resource files:

$PROJECT_HOME/src/test/resources/sample/RiderAutoPartsPartnerTest.xml
$PROJECT_HOME/src/main/resources/META-INF/spring/beans.xml file

These are the same file albeit named differently. One is used by the bundle and the other by our test case.

We'll be also using the default Apache ActiveMQ instance for the messaging aspect of this tutorial.
To check whether the ActiveMQ is installed, up and running you can run the following command:

karaf@root> osgi:list |grep activemq-broker


and you should see an output similar to this

[  66] [Active     ] [Created     ] [       ] [   50] activemq-broker.xml (0.0.0)


You can see the default broker's definition within the activemq-broker.xml file under $SERVICEMIX_HOME/etc/activemq-broker.xml

The database source definition is on the beans.xml file and you may want to adjust it to reflect the settings on your environment. Here is a copy of the definitions I have on my machine:

<bean id="myDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">


    <property name="driverClassName" value="com.mysql.jdbc.Driver"/>

    <property name="url" value="jdbc:mysql://localhost/test"/>

    <property name="username" value="root"/>

    <property name="password" value="XXXX"/>

</bean>

7) Let's compile the code then… Run the following command on the root directory of the project

mvn clean install -Dmaven.test.skip=true


If the code builds up successfully then you're ready to deploy on ServiceMix.

8) To deploy the bundle into ServiceMix, run the following command:

karaf@root>osgi:install mvn:com.fusesource.fusebyexample/camel-jms-dbase/1.0-SNAPSHOT


A confirmation that the bundle was deployed successfully should be returned to you and a message similar to the follow one should be displayed:

Bundle ID: 228


Then, you are ready to start the bundle and verify if has started correctly. Follow the instructions below to do that:

To start the bundle:

karaf@root>osgi:start <bundle ID>


To verify the bundle has started correctly:

karaf@root>osgi:list |grep <bundle ID>


and then a message like the following should be displayed:

[ 228] [Active     ] [            ] [Started] [   60] Camel JMS Database Example (1.0.0.SNAPSHOT)


You can also immediately turn on the trace logging capability for Camel and get additional output in the ServiceMix log facility. To enable the trace logging just run:

karaf@root>set TRACE org.apache.camel


and then to see the ServiceMix log, run:

karaf@root>log:display


You should be able to see messages with the TRACE identifier similar to the below:

11:46:17,095 | TRACE | tnerRequestQueue | JmsMessageListenerContainer      | ?                                   ? | 94 - org.springframework.jms - 3.0.5.RELEASE | Consumer [ActiveMQMessageConsumer { value=ID:titan.local-54477-1319240594120-6:1:1:1, started=true }] of session [PooledSession { ActiveMQSession {id=ID:titan.local-54477-1319240594120-6:1:1,started=true} }] did not receive a message


9) To test this sample project all you have to do is to send a sample message (there is one included in the $PROJECT_HOME/src/test/resources/sample directory) to the ActiveMQ queue that the Camel route is listening for. You can use any approach you like but writing a JMS Test client is pretty simple and there are tons of examples on the Internet and also available in the ActiveMQ sample directory. You can also use tools like HermesJMS which adds a little UI for you so you don't need to create a test program.

Either way, after sending a message to the partnerRequestQueue (default queue name used on this tutorial) you should see a couple of messages in the ServiceMix console like those below:

karaf@root>  **** toSql returning: INSERT INTO PARTNER_METRIC (partner_id, time_occurred, status_code, perf_time) VALUES ('123', '200911150815', '200', '9876')

 **** fromSql returning: Sample message to be returned on reply queue


You can verify that the values were inserted into the MySQL database running a simple query like:

select * from PARTNER_METRIC;


It is also recommended to check the ServiceMix log to make sure there are no exceptions there.

The sample code for this tutorial is available on GitHub: https://github.com/mjabali/JMS-DB-Sample

I also would like to thank my colleague Susan Javurek for putting an extra effort on this and helping out with the sample code.

UPDATE: I also would like to say a big thanks to Claus Ibsen (@davsclaus) for reminding me that all the manual steps related to OSGi bundles installation could be automated using the feature capability that consists in deploy multiple related bundles into a larger unit of deployment.

Enjoy the ride...

Using Apache Camel for Application Notification

I was watching a forum thread (http://fusesource.com/forums/thread.jspa?threadID=3433) this week where the use case was not to consume the file content but rather the file name. That seems to be an interesting use of Apache Camel because most of people used the Camel File Component to consume the contents of the file and then do some kind of processing on top of that (routing, web services invocation, ftp, bean execution, database interaction, etc).

What brought my attention to this specific use case is that the important information here was the file name and not the content of the file. Given that the file itself was pretty large (around ~1GB) it would be a nightmare to handle all of that in memory on a Camel route or any other middleware available out there. But, this use case is really about notifying other systems or applications that a large file is available to be processed outside the route (think about a file being available for download for example) and the reason to only care about the file name is that information could be part of the URL generated during the processing of the Camel route.

So, basically the Camel route was watching the file system for files generated, then grabbed the file name and then notify another system or application (or even the end user i.e. e-mail) that the file was ready to be consumed. I've recreated below the Camel route generated by default in the sample project to illustrate how to do that.

<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xmlns:camel="http://camel.apache.org/schema/spring"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd

       http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

<camel:camelContext xmlns="http://camel.apache.org/schema/spring">

    <camel:package>com.fusesource.fusebyexample</camel:package>

 <camel:route>

      <!-- Consume Files from the File System -->

      <camel:from uri="file:src/data/?noop=true"/>

      <!-- Set the Message Payload with the File Name --> 

      <camel:setBody>

          <camel:simple>header.CamelFileName</camel:simple>

      </camel:setBody>

      <camel:log message="The message contains ${body}"/>

      <!-- Perform simple Content-Based Routing -->

      <camel:choice>

      <camel:when>

          <camel:simple>${body} == 'message1.xml'</camel:simple>

          <camel:log message="Going to UK message"/>

          <camel:to uri="file:target/messages/uk"/>

        </camel:when>

        <camel:otherwise>

          <camel:log message="Going to Other message"/>

          <camel:to uri="file:target/messages/others"/>

        </camel:otherwise>

      </camel:choice>

    </camel:route>

  </camel:camelContext>

</beans>

Of course, if you want to try that out, the project is available at: https://github.com/mjabali/FileNameBasedRouter

Enjoy the ride!

FuseSource Community Day in San Francisco, CA - November 17th

Thirsty for information about Apache Camel? Hungry to learn more about Apache ActiveMQ in production environments? Keen to know more about the Business Impact of Open Source Software...

Don't miss the opportunity to hear from Rob Davies (CTO of FuseSource, PMC Member, Founder and Committer on Apache ServiceMix and ActiveMQ and also the co-author of ActiveMQ in Action) and James Strachan (Creator of Groovy, member of the Apache Software Foundation and co-founder of Apache Camel). You'll be also hearing from architects, consultants and developers using those technologies in real world applications and what problems they were able to solve.

Not enough for you?!?! The FuseSource Community Day will be held at a really nice venue in downtown San Francisco called Thirsty Bear (661 Howard St., San Francisco, CA 94103).

For more information, visit the event web page: http://form.fusesource.com/sanfrancisco2011

I hope to see you all there...

Using HTTP-based endpoints with Apache Camel

On a recent discussion, a partner was trying to use Apache ActiveMQ HTTP Transport Connectors to receive HTTP requests from a non-JMS Web application and asked me what I would think it could be a good approach to the use-case. Analyzing the use case, which needs a synchronous multi-step execution I suggested that they took a look in the Apache Camel Jetty component instead of Apache ActiveMQ and that was surprisingly easier to setup and maintain then the original approach.


While there is nothing wrong with the approach they thought about I think that Apache Camel would give them a more powerful setup to what they were looking for.


So, here is a sample configuration, that I created as a demo project, using Apache Camel Jetty component enabling HTTP endpoints.


In our Apache Camel route definition (using the Spring XML approach in this case) we'll have to define the Jetty endpoint similar to this:

<route id="Jetty_Sample">

        <from uri="jetty:http://localhost:8888/myBookService"/>

where we basically specify the address we'll be consuming HTTP requests. There are many options we can use to fine tuning the Jetty component but we'll keep it with only the required information for simplicity.

Then, the next step is going to be a regular approach to any other route we have developed so far. In this example, we're going to use a LOG component and then a custom Process pointing to a simple Java bean where we perform the processing of every request.

<log logName="HTTP LOG" loggingLevel="INFO" message="HTTP REQUEST: ${in.header.bookid}"/>

<process ref="myBookService"/>

The log component above will be displaying a header name called 'bookid' and then the Java bean (listed below) will be taking care of the request:

package com.fusesource.fusebyexample;

import org.apache.camel.Processor;

import org.apache.camel.Exchange;

public class myBookService implements Processor {

    public void process(Exchange exchange) throws Exception {

        // Grab the booked header value

        String bookId = (String) exchange.getIn().getHeader("bookid");

        // send a html response

        exchange.getOut().setBody("<html><body>Book " + bookId + " is Camel in Action.</body></html>");

    }

} 

The last thing we need to add to our Apache Camel context file is the bean definition which could be simply like the line below:

<bean class="com.fusesource.fusebyexample.myBookService" id="myBookService"/>

That's all we need... To test the integration use case described above, we can simply run the Camel route, open any web browser and hit the following URL:

http://localhost:8888/myBookService?bookid=91942

where the Camel route will log the following in the console:

HTTP LOG                       INFO  HTTP REQUEST: 91942

and the web browser will display the result of the Java bean processing with the following:

If you want to use a simple HTML form, here is something to easily test the Camel route...

<html>

<body>

<form action="http://localhost:8888/myBookService">

Book ID: <input text name="bookId"/>

<input type="submit" value="Submit">

</form>

</body>

</html>

I've also included the test HTML form on the source code package of this configuration available here:

https://github.com/mjabali/Jetty-Sample

Enjoy the ride!

FuseSource Training in Brazil

Since the FuseSource Brazilian tour a couple of weeks ago I've been getting more and more questions about what's next for the region. Well, here is the next step... We're going to deliver an Apache ActiveMQ training class on Nov, 16-17 in São Paulo and an Apache ServiceMix with Camel training class on November, 18-19 also in São Paulo. We are still working to get a nice venue for the training in case you wonder where this is going to happen.

I've personally been involved in the FuseSource training and I can guarantee you that's a lot of fun and tons of good information.

Here is the URL with information about the training classes: http://fusesource.com/enterprise-support/open-enrollment/ but if you have any questions please don't hesitate to contact me through this blog, Twitter (@mjabali) or e-mail (marcelo@fusesource.com).

Hope to see you there...

Apache ActiveMQ Enhancements to JMS

The JMS (Java Message Service) specification is very well-known for those working with messaging platforms but it has been around for quite some time and honestly not getting updates lately which gives a lot of room for enhancements and extensions.

As you can imagine Apache ActiveMQ goes beyond the JMS spec and implement lots of cool things where you can gain more in functionality, performance and scalability.

Starting from the top ConnectionFactory object, ActiveMQ has its own called ActiveMQConnectionFactory, as other JMS providers also create their own specific factories, ActiveMQ enhances the JMS with some settings that you can activate right away with little configuration and definitely see gains in performance.  Of course, when you optimize for performance, most of the time, you have the trade-off of more CPU utilization but I think CPU cycles are getting really cheap when comparing with network bandwidth and disk I/O.

Here is a list of things you can use to optimize performance extending the JMS specification with Apache ActiveMQ:

disableTimeStampsByDefault - it sets whether or not you're going to include the Time Stamp in the message header. The story behind this is that when you create a JMS message (actually when you call the send() method) we have to make a call to the system kernel to get the time stamp. This setting could save you a trip to that level and save you some time optimizing performance if the time stamp is not required in your messaging application.

copyMessageOnSend - The Java Message Service specification says that inside the send method we have to make a deep copy of the message object before sending it from the client to the broker. Setting the copyMessageOnSend property to false (default is set to true) will save you a cycle as the client won't make that copy of the message saving local resources.

useCompression - I think the property name here is self explanatory... but the ActiveMQ client will compress the message payload before it goes to the broker. So, the question is why not to use compression all the time and send smaller messages over the wire? Well, in order to compress the message payload you're going to spend CPU time and then the point is CPU vs. I/O. It's a trade-off that you may have to consider... in other words, you'll have to consider spending more CPU to compress the message or more bandwidth sending larger messages. Personally, I believe (and have seem) that if you have the chance to use compression you definitely should.

objectMessageSerializationDefered - Imagine that you're sending an Object Message and you're using an embedded broker for example (client and broker running on the same JVM). What this feature really enables you to do is that you tell the client to delay the object serialization as long as it can until it needs to hit the wire or store the message on disk. Basically, everything inside the same JVM should not be serialized as marshalling and unmarshalling objects can also be expensive.

useAsyncSend - Let's say you're sending NON_PERSISTENT messages to the broker which is equal to non-blocking call with no immediate acknowledgement and in fact it's ver fast but still not reliable messaging.
On the other hand, you can also use PERSISTENT messages which is going to be a blocking call from the client to the broker. So, one of the things you can do is to set useAsyncSend to true (default is false) and then change the behavior of PERSISTENT delivery mode to be asynchronous and no acknowledgement would be expected. In reality,  the send() method will return immediately giving you more performance because it reduces the time that the message producer waits for the message to be delivered. Keep in mind that this setting can possibly cause message loss but that's another trade-off that you have to consider when designing your messaging infrastructure.
This is a good approach if you can tolerate some message loss... typically on systems where the most updated information (last message) is always what you care about or when you have a reconciliation process in place to catch those conditions.

useRetroactiveConsumer - This feature allows you to work with limited window of memory topics where you can configure a policy to deliver let's say for example the last 10 messages sent to the topic, the last 5MB of data that went through to that topic or simply the last message sent to the topic... and all of that without the need of durable subscribers :-)
This is actually far more lightweight than using durable subscribers and it's really easy to configure and very easy to use.
Finally, since this is not JMS standard, of course, it's not set by default.

For a complete list of ActiveMQ enhancements to the JMS specification see the ActiveMQ product documentation.

I hope this helps you to push the go-faster button in your messaging infrastructure.

FuseSource Tour in Brazil

I had the pleasure to spend a week in Brazil doing a FuseSource Tour in two of the major cities... Sao Paulo and Rio de Janeiro. It's pretty clear that the Open Source community is growing really fast and the integration and messaging opportunities are hot.

Unfortunately, I couldn't visit all of the prospective partners and customers that had contacted us because we couldn't find a single empty spot on the agenda but I think the tour was an amazing experience and I'm pretty I'll back in the near future for more fun with the techies.

Some of the things I realized while traveling throughout Brazil is that the traffic is really crazy and made me remember some other cities like Los Angeles or Mexico City... You have to be really creative in terms of logistics to be able to meet more than two customers in the same day especially in Sao Paulo.

Food was a big surprise and it was pretty good either in Sao Paulo or Rio and I met nice and friendly people all over the place... Gotta keep up on the workout front to shed some extra pounds!

On the business side, I've met mostly customers in the financial sector and partners that already use Apache technologies like ActiveMQ, Camel, CXF and ServiceMix and I was able to have a good tech talk with them while discussing some architectural aspects as well as development techniques.

While talking to a couple of big banks in Sao Paulo, it sounded to me that companies are realizing the value and maturity of, for example, Apache ActiveMQ where their are not only reducing licensing costs but also exploring features that other messaging products don't offer at this point.

Of course, the reliability and extensibility as well as high performance are key factors for all of these organizations and ActiveMQ is/is going to be part of their architecture.

Sao Paulo Subway System

Then, on the second part of the week (more precisely in Rio de Janeiro), I visited a telecom company who's using ServiceMix as part of their core engineering systems and I was happy to discuss patterns and features they are using and also some things they can do to improve it in the next rollout.

Rio de Janeiro's Sugar Loaf

Interesting enough, all of the developers and architects I had the pleasure to talk to were really amazed with Apache Camel and there are tons of use cases where they see possibilities of writing a couple lines using either Java DSL or Spring XML Configuration rather than several lines of code where they need to manage everything from the connection to the business logic. What everybody really liked during the Camel demos is the abstraction layer that Camel provides where you concentrate on the business logic and not on the details of the implementation. Gotta a comment that we're now "High-tech plumbers" connecting all of the enterprise applications with the simplicity and power of Camel, ActiveMQ, CXF and ServiceMix. 

But, the best moment was when I did an Apache ActiveMQ, Camel, CXF and ServiceMix overview presentation in one of the largest companies in the country for 100 developers and technical folks. I was really impressed on how excited they got after the talk and Q&A and they are really looking forward to apply the discussed topics on their projects right away. The same presentation should be the core of a webinar that I'm planning to do in December... a quick overview about the technology and how FuseSource has been helping customer/partners to successfully work with those Apache projects from conception to deployment.

To celebrate the successful talk, next morning I went for a run at Ipanema's beach where I ended up with the view below... Simply amazing...

Ipanema's Beach

Well, I'm really looking forward to continue to work with organizations in Latin America and come back soon...