Cloud Event Processing on CloudBees RUN@Cloud PaaS

I did some prototyping of a Cloud Event Processor around a year ago that combined the open-source Esper CEP engine with Amazon's SNS and some code I wrote to create a loosely coupled event processor in the cloud.

Now, when I did this I used SNS to serve as the Cloud based loosely coupled messaging, but I did have the event processor itself running on a server on my network.  I noted then that because of using SNS and HTTP as the protocol between my server and SNS that the event processor could be deployed anywhere, but until today I hadn't actually moved it somewhere else.

What did I move it to?  I could have just found a hosting provider or gone the IaaS route and used AWS's free micro instance and configured the OS and installed a Web container, but with just a standard WAR file what I really wanted was a Java Platform as a Service (PaaS) offering I could just deploy to.  Doing this would save me the time and effort of setting everything up myself and having to maintain/upgrade it over time.  As much as the geek in me likes to play around with infrastructure and platform components, there isn't really much value in me spending time doing that in a situation like this.

So, I took a look at CloudBees and their RUN@Cloud offering.  It advertised the ability to develop like I'm used to and then instantly deploy to the Cloud.  Does it live up to this promise?  Based on my simple application and a very basic deployment, the answer is yes.

I signed up for a free CloudBees account which was very straightforward, and in a matter of minutes I had access and could create a new RUN@Cloud application.  Once created, I could then upload and deploy my WAR file and voila, I was deployed in the Cloud.  I didn't have to change a line of code or create/modify any descriptors or XML, it couldn't have been simpler.  I simply had to change my client and/or SNS URLs to point to the new server and it just worked.  And my initial goal of not having to deal with installing/configuring the platform was realized.

On top of it just working, I get some nice monitoring and administration tools that show me session, request, memory, and load statistics.  Very nice.

If you are looking for an easy to use Java PaaS, give CloudBees a look.

Event Processing in the Cloud - Combining Esper with AWS SNS

The folks at Amazon Web Services released their Simple Notification Service in beta a little over a month ago, and I used it to create a loosely coupled weather notification system by publishing weather events to an SNS topic.  It is working great, but I started to think of other types of situations I'd want to monitor and have notifications for, and what better way to process these weather events than with a Complex Event Processing (CEP) engine?  While I could keep extending my shell script, that would be a bit of a hack and I wanted something cleaner that would more easily last the test of time.

Event processing is in many ways a natural extension of messaging infrastructure as the latter is typically used for the passing of events between systems and so having an easy way to plug in or use an event processor makes a lot of sense.  In my case, I wanted to be able to feed periodic weather observations, the raw events, into the messaging infrastructure, then be able to dynamically define the processing rules for those events, and then be able to publish any notifications or "complex events" back into the messaging infrastructure where I could then subscribe or receive them for e-mail/SMS notifications or to kick off other processes or logic.

So, how to put this all together?  There are CEP engines from many of the major middleware vendors, but on my shoestring budget I wasn't about to go spend 5-6 figures on software just to process events for my weather station!  At Sun, as part of the OpenESB project we had developed the Intelligent Event Processor in open-source so it was a natural choice, but it requires the complete ESB infrastructure and can't run standalone.  As I was already using SNS for my messaging and was aiming to make what I created Cloud friendly (see below), I didn't want to bring along an entire ESB, so I needed something else.  Another open-source CEP engine I was familiar with is Esper, and after a quick refresher on its capabilities, I decided it would be perfect.

Esper provides a simple to use but very powerful Java API for configuring the events, queries, patterns, etc. and so I set about creating the interface between SNS and Esper.  What I ended up building has the following capabilities:

• A simple (HTTP/JSON) API for dynamically configuring the events and queries/statements in the engine.
• Receives events from an SNS topic via HTTP and feeds them into Esper.  The publisher of the raw events just publishes to the SNS topic like it was before and I just create a new subscriber for the CEP engine, thus leveraging the beauty of a loosely coupled messaging system.
• Esper processes the events per the configured queries.  As the queries generate results, the results are published to a specified SNS topic which can result in a notification e-mail or kicking off some other process or logic, again leveraging the decoupled nature of SNS.

Now, Esper provides a variety of ways to configure the engine, XML being one of them, but you'll note above I chose to create an HTTP/JSON API to serve as the interface to Esper's Java API for doing this configuration.  This was to be both Cloud friendly and to make the configuration dynamic.  This allows the idea of an event processor in the Cloud that is just available to be configured dynamically, receive events, and publish results, all with no installation or maintenance of the engine or other infrastructure.  One never even needs to log in to a machine to mess with any configuration files.

With this created, I was then able to easily configure the following:

1. I had an existing script called by cron every 10 minutes that sent the weather observation to the Weather Underground.  I added a few lines of code to this script to publish the observation to an SNS topic as well.
2. I created a subscription to the SNS topic that notifies the event processor using HTTP/JSON of each event.
3. I configured my "Cloud CEP" with the following JSON to define the events it would be receiving:

{     "Type": "EventConfiguration",     "Name": "WeatherEvent",     "Fields": [["temp", "double"], ["humidity", "double"], ["dewpoint", "double"]] }

4. I configured a query to use for processing the events, this one telling me the high and low for the past 24 hours every 12 hours at midnight and noon, and sending the result to the specified SNS topic.  The statement is Esper's SQL like syntax for specifying queries:

{     "Type": "ListenerConfiguration",     "Name": "Every 12 hours high and low",     "Statement": "select max(temp) as High, min(temp) as Low from WeatherEvent.win:time(24 hours) output at (1, */12, *, *, *)",     "ActionType": "SNS",     "SNSTopicArn": "arn:aws:sns:us-east-1:444520459559:WeatherEventOutput" }

5. I then created a subscription to the WeatherEventOutput topic to send myself an e-mail.

I now get an e-mail at midnight and noon each day telling me what the high and low were for the past 24 hours.  If I want to know any time the temperature changes more than a certain amount in a given period, or if the temperature shows a trend of getting closer to the dewpoint (i.e. it is about to rain), I can add this to the processing with just a simple HTTP/JSON configuration request.  And with the power of Esper, those scenarios and a lot more are possible for my weather data or any other event data one might have.

The further beauty of this is that while I happen to have this running on my server at home, because of the way it is built using SNS and HTTP, it could be located anywhere on the internet whether hosted on EC2 or your favorite provider, or a "Cloud CEP" service available to all.

Note also that what I've written is by no means tied to SNS either.  It just happens to be Cloud based messaging infrastructure that is convenient to use and gives the benefits of loose coupling.  The input to my Cloud CEP is just an HTTP request and I've written an e-mail and HTTP handler so that generated events can go direct as well instead of going to SNS.

So what do you think?  Do you have scenarios where a Cloud CEP would be useful?  Would you like to try out what I've built thus far?  Feel free to leave a comment or contact me at kschmidt at techrunning dot com.

links for 2010-02-02: Google Web Server; Flash required?; BEP/CEP

• Google mystery server runs 13% of active websites - Pretty amazing considering no one outside of Google runs the Google Web Server.  A lot of folks are using Google to host their sites/blogs/etc. (this blog being one of them).
• Is Flash required for a rich web experience? - I have to admit that the lack of Flash on the iPhone is annoying at times, so I'd have to think it would be a lot more often with the iPad.
• Business event processing and SOA: Joined at the hip - Sun had a cool CEP engine in OpenESB, not sure what will happen to it now.
• Windows Azure too expense for small apps - If true, missing out on an opportunity to get customers when they are small and growing.