Event Processing in the Cloud - Combining Esper with AWS SNS

The folks at Amazon Web Services released their Simple Notification Service in beta a little over a month ago, and I used it to create a loosely coupled weather notification system by publishing weather events to an SNS topic.  It is working great, but I started to think of other types of situations I'd want to monitor and have notifications for, and what better way to process these weather events than with a Complex Event Processing (CEP) engine?  While I could keep extending my shell script, that would be a bit of a hack and I wanted something cleaner that would more easily last the test of time.

Event processing is in many ways a natural extension of messaging infrastructure as the latter is typically used for the passing of events between systems and so having an easy way to plug in or use an event processor makes a lot of sense.  In my case, I wanted to be able to feed periodic weather observations, the raw events, into the messaging infrastructure, then be able to dynamically define the processing rules for those events, and then be able to publish any notifications or "complex events" back into the messaging infrastructure where I could then subscribe or receive them for e-mail/SMS notifications or to kick off other processes or logic.

So, how to put this all together?  There are CEP engines from many of the major middleware vendors, but on my shoestring budget I wasn't about to go spend 5-6 figures on software just to process events for my weather station!  At Sun, as part of the OpenESB project we had developed the Intelligent Event Processor in open-source so it was a natural choice, but it requires the complete ESB infrastructure and can't run standalone.  As I was already using SNS for my messaging and was aiming to make what I created Cloud friendly (see below), I didn't want to bring along an entire ESB, so I needed something else.  Another open-source CEP engine I was familiar with is Esper, and after a quick refresher on its capabilities, I decided it would be perfect.

Esper provides a simple to use but very powerful Java API for configuring the events, queries, patterns, etc. and so I set about creating the interface between SNS and Esper.  What I ended up building has the following capabilities:

• A simple (HTTP/JSON) API for dynamically configuring the events and queries/statements in the engine.
• Receives events from an SNS topic via HTTP and feeds them into Esper.  The publisher of the raw events just publishes to the SNS topic like it was before and I just create a new subscriber for the CEP engine, thus leveraging the beauty of a loosely coupled messaging system.
• Esper processes the events per the configured queries.  As the queries generate results, the results are published to a specified SNS topic which can result in a notification e-mail or kicking off some other process or logic, again leveraging the decoupled nature of SNS.

Now, Esper provides a variety of ways to configure the engine, XML being one of them, but you'll note above I chose to create an HTTP/JSON API to serve as the interface to Esper's Java API for doing this configuration.  This was to be both Cloud friendly and to make the configuration dynamic.  This allows the idea of an event processor in the Cloud that is just available to be configured dynamically, receive events, and publish results, all with no installation or maintenance of the engine or other infrastructure.  One never even needs to log in to a machine to mess with any configuration files.

With this created, I was then able to easily configure the following:

1. I had an existing script called by cron every 10 minutes that sent the weather observation to the Weather Underground.  I added a few lines of code to this script to publish the observation to an SNS topic as well.
2. I created a subscription to the SNS topic that notifies the event processor using HTTP/JSON of each event.
3. I configured my "Cloud CEP" with the following JSON to define the events it would be receiving:

{     "Type": "EventConfiguration",     "Name": "WeatherEvent",     "Fields": [["temp", "double"], ["humidity", "double"], ["dewpoint", "double"]] }

4. I configured a query to use for processing the events, this one telling me the high and low for the past 24 hours every 12 hours at midnight and noon, and sending the result to the specified SNS topic.  The statement is Esper's SQL like syntax for specifying queries:

{     "Type": "ListenerConfiguration",     "Name": "Every 12 hours high and low",     "Statement": "select max(temp) as High, min(temp) as Low from WeatherEvent.win:time(24 hours) output at (1, */12, *, *, *)",     "ActionType": "SNS",     "SNSTopicArn": "arn:aws:sns:us-east-1:444520459559:WeatherEventOutput" }

5. I then created a subscription to the WeatherEventOutput topic to send myself an e-mail.

I now get an e-mail at midnight and noon each day telling me what the high and low were for the past 24 hours.  If I want to know any time the temperature changes more than a certain amount in a given period, or if the temperature shows a trend of getting closer to the dewpoint (i.e. it is about to rain), I can add this to the processing with just a simple HTTP/JSON configuration request.  And with the power of Esper, those scenarios and a lot more are possible for my weather data or any other event data one might have.

The further beauty of this is that while I happen to have this running on my server at home, because of the way it is built using SNS and HTTP, it could be located anywhere on the internet whether hosted on EC2 or your favorite provider, or a "Cloud CEP" service available to all.

Note also that what I've written is by no means tied to SNS either.  It just happens to be Cloud based messaging infrastructure that is convenient to use and gives the benefits of loose coupling.  The input to my Cloud CEP is just an HTTP request and I've written an e-mail and HTTP handler so that generated events can go direct as well instead of going to SNS.

So what do you think?  Do you have scenarios where a Cloud CEP would be useful?  Would you like to try out what I've built thus far?  Feel free to leave a comment or contact me at kschmidt at techrunning dot com.