Cloud Event Processing on CloudBees RUN@Cloud PaaS

I did some prototyping of a Cloud Event Processor around a year ago that combined the open-source Esper CEP engine with Amazon's SNS and some code I wrote to create a loosely coupled event processor in the cloud.

Now, when I did this I used SNS to serve as the Cloud based loosely coupled messaging, but I did have the event processor itself running on a server on my network.  I noted then that because of using SNS and HTTP as the protocol between my server and SNS that the event processor could be deployed anywhere, but until today I hadn't actually moved it somewhere else.

What did I move it to?  I could have just found a hosting provider or gone the IaaS route and used AWS's free micro instance and configured the OS and installed a Web container, but with just a standard WAR file what I really wanted was a Java Platform as a Service (PaaS) offering I could just deploy to.  Doing this would save me the time and effort of setting everything up myself and having to maintain/upgrade it over time.  As much as the geek in me likes to play around with infrastructure and platform components, there isn't really much value in me spending time doing that in a situation like this.

So, I took a look at CloudBees and their RUN@Cloud offering.  It advertised the ability to develop like I'm used to and then instantly deploy to the Cloud.  Does it live up to this promise?  Based on my simple application and a very basic deployment, the answer is yes.

I signed up for a free CloudBees account which was very straightforward, and in a matter of minutes I had access and could create a new RUN@Cloud application.  Once created, I could then upload and deploy my WAR file and voila, I was deployed in the Cloud.  I didn't have to change a line of code or create/modify any descriptors or XML, it couldn't have been simpler.  I simply had to change my client and/or SNS URLs to point to the new server and it just worked.  And my initial goal of not having to deal with installing/configuring the platform was realized.

On top of it just working, I get some nice monitoring and administration tools that show me session, request, memory, and load statistics.  Very nice.

If you are looking for an easy to use Java PaaS, give CloudBees a look.

links for 2011-01-26: AWS Simple Email Service; Marc Fleury on Oracle and Open Source; No screenshots on Windows Phone 7

• Introducing the Amazon Simple Email Service - Another in the line of "simple" services.
• Oracle and Open Source. NOT GOOD - Marc's view on how Oracle is handling open source, based largely on what has happened with OpenOffice and Hudson.
• Capturing screenshots in Windows Phone 7's future? - Hiding behind DRM seems like a poor excuse.  And no one has ever asked for it?

links for 2011-01-21: AWS Elastic Beanstalk; Googled copied Sun's Java code

• AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud - AWS continues to avoid going up stack to compete with partners, but still offer new services and value.
  
• Oops: Android contains directly copied Java code, strengthening Oracle's case - That would be a big ooops!
  
• Oops: No copied Java code or weapons of mass destruction found in Android - Or maybe not.
  

links for 2010-12-17: Invisible open-source; Oracle Apps on EC2; Pirate or DRM; Netflix on AWS

• 2010: The year open-source went invisible - It has gone mainstream and is even a pre-requisite in some markets.
• Oracle Apps on EC2 - You could do it on your own before, easier now.
• Pirate or buy DRM? - Not a fan of DRM it would seem.
• Netflix lessons from using AWS - Also read this for why they switched.

links for 2010-12-14: IBM's Watson to play Jeopardy; FreeBSD on EC2; Data on Hudson/Kohsuke; Open data

• IBM’s “Watson” Computing System to Challenge All Time Greatest Jeopardy! Champions - This should be entertaining and interesting to watch.
  
• FreeBSD on Amazon EC2 - I'm somewhat surprised it has taken this long!
• Hudson's Secret: Kohsuke - Eduardo offers some insight into a fantastic developer.  But also shows a great example of visualizing data.  Love it!
• Make Data Dance - Another great article on generating value and intelligence from data.  And kudos to the Bill Gates Foundation for requiring that data from projects they fund be open.

links for 2010-10-27: Netflix moves to AWS; RedMonk Analytics; Google Docs Charting; Apple alienating their users

• Netflix transitions to HA storage in the Cloud - The value of moving to the Cloud isn't just for startups.  Netflix has their own data center, but elected to transition to the Cloud to not burden their IT staff with mundane data center build out.
• RedMonk Analytics: What Operating Systems are Developers Using? - It depends on the perspective and which developer audience is looked at!
  
• Using new visualizations to tell your story - Improving the charting Google Docs can deliver.
  
• Apple Won't Push Flash Player Updates - Apple's alienation of developers and consumers continues.

links for 2010-10-22: AWS free for a year; PostgreSQL vs MySQL; Stacking the JCP election; Java on OS X; What Steve could have said; Adivce to Oracle regarding Java

• Amazon Web Services: Try us free for a year - Something worth looking in to.
• PostgreSQL vs. MySQL: How to Select the Right Open-Source Database - A balanced well written article.
  
• Stacking the JCP election - An interesting read even if it is a bit biased (and I don't know if it is or isn't).
• Steve Jobs Comments on Apple's Java Discontinuation - Some interesting history from James.
  
• What Steve could have said - "We’ve been in the computer business for a long time and we’ve learned that 80% of people just aren’t very interesting. They don’t need to create that much content, because when they do, most it isn’t very good. The iPad actually saves them from themselves because it keeps them from creating a bunch of useless stuff that no one else wants to read."
• Dear Oracle, Get a Clue - Some good advice.  Is Oracle listening?

links for 2010-10-19: Twitter stats; Windows Phone 7; AWS Management Console adds SNS support

• Replies and Retweets on Twitter - Some interesting stats, love the data.  Are you surprised the retweets/replies are that high or do you think it was higher?
• Windows Phone 7 - Seems like an uphill climb in the phone market but tablet market could be a place to grow share.
• AWS Management Console Now Supports Amazon SNS - I didn't mind using the command line tools, in fact they are preferred in many cases, but adding support to the Management Console GUI is a nice addition.
  

links for 2010-10-10: If Oracle and MySQL had a child; AWS weakness a strength?; JavaOne content will cost you; Android revolution; Short-term focus hurts R&D

• If Oracle and MySQL had a child - Why not get the best of both worlds with PostgreSQL?
• Amazon Web Services: an instance of weakness as strength - Most people like simple evolution when making change in the enterprise.
  
• JavaOne 2010 content costs money? - In the past, the content from JavaOne was generally made available for free to the community, but the link seems to indicate it now costs $245 or you have to have been an attendee.  Past content on SDN only includes 2008 and 2009.  To be fair, there is still a lot of free content but this seems to be a change.
• The five factors powering the Android revolution
  
• R&D: Short-term focus, cynicism hampers us - Who else will innovate just for the sake of doing so?
  

links for 2010-09-30: Gosling on JavaOne; MySQL prices going up; Gmail capitulates; MySQL vs PostgreSQL performance; PHP SDK for AWS

• JavaOne was fun - Some interesting, albeit likely biased, observations.
• Oracle raising prices on MySQL - Rationalizing the pricing to Oracle's way is to be expected, but the removal of Basic and Silver is important and what the prices actually are will be doubly so.
• Gmail unthreads messages - I prefer threaded views and find search works much better than sort.
• MySQL vs PostgreSQL performance - Simply confirms what I think we all knew: PostgreSQL is much faster for inserts, MySQL slightly faster for selects.
• New AWS SDK for PHP - Adds to the existing Java and .NET SDKs.

links for 2010-09-21: Amazon Hiding in Plain Sight; PostgreSQL 9.0 released; iPad good for two things; Oracle and HP make up; Unbreakable Enterprise Linux kernel performance doubts

• Hiding in Plain Sight: The Rise of Amazon Web Services - It really is surprising AWS doesn't get credit or press for much of what they are doing.  When they announce something they have it working and available, often with a free/beta tier.  When most other vendors announce something, it isn't available yet and when it is, you have to pay for it, get a machine, install it, etc.
  
• EnterpriseDB Announces Support for PostgreSQL 9.0 - 9.0 has some great new replication features that make it even more of a natural cheaper alternative to Oracle.
• The iPad is good for only two things - Admittedly, the first thing is pretty big and broad.
• Oracle and HP settle spat - Having the issue prolonged would not have been good for HP.
• Mirror, Mirror On The Wall, Which Is The Best Open Source DB Of Them All? - Will PostgreSQL fill the perceived void created by Oracle's ownership of MySQL?
  
• Oracle announces Unbreakable Enterprise Linux kernel - This story seems to doubt the performance claims.
  

links for 2010-09-16: Android benchmarks; Android market share up; AWS Linux AMI; Password resuse

• Benchmark showdown: High-end Android smartphones - Unfortunately, there isn't one "fastest" phone, but several do very well in multiple but different categories
• Android market share up, everyone else flat or down - Interestingly Palm is flat, RIM, Apple, Microsoft all down
• Amazon Linux AMI - Need Linux in the Cloud?  This should save you a little time
• Password Reuse - Hmmm, I'll be changing some passwords now thank you

Amazon further integrates Simple Notification Service with AWS

I've always been interested in messaging systems and had some fun learning about AWS's SNS and then building a simple Cloud Event Processor that utilized it earlier this year.  Amazon had already provided out of the box integration of SNS with the Simple Queue Service (SQS) but has now introduced integration with S3 so that you can get notifications for certain S3 Reduced Redundancy Storage events.

Asynchronous programming at work.  Good stuff!

links for 2010-07-01: iPhone to Verizon rumor (again); Brown University goes Google; SQS Updates; Open Source Licensing; Steve Jobs on the Enterprise; Database Migrations Easy?

• Verizon to Start Selling iPhone in January [REPORT] - This report seems to be a bit more credible than ones of the past, so will it really happen?
  
• Brown University has gone Google - "Our students were really the ones that led us down the Google path. They knew these tools would work because they already used them in their non-school lives."
• New: Free Tier and Increased Limits for Amazon Simple Queue Service - Some incremental updates making SQS a little more valuable.
  
• Open Core the new Dual License - He still advocates pure open source and using telemetry and data as ways to generate incremental revenue.
• What Steve Jobs hates about the enterprise - So Apple will just rely on the consumerization of IT as their way into the enterprise?  Is this an opportunity for Microsoft to capitalize on RIM's weakness?
  
• Database Migrations Are Finally Becoming Simpler - Compatibility layers make sense.  I knew EnterpriseDB had been doing this for some time, I didn't know IBM was too.  Oracle, your move.
  

links for 2010-06-28: iPhone 4 doubles 3G bandwith speeds; iPhone wish list = Android; Google Voice goes GA; AWS in the enterprise

• iPhone 4 upgrades to HSDPA and HSUPA can double 3G bandwidth speeds - I wonder why these upgrades to HSDPA and HSUPA weren't advertised?  This is probably the benefit some are most interested in.
  
• An iPhone wish list looks more like an Android feature list - A survey says iPhone users want choice of wireless networks, a 4G network, 8 megapixel camera, and FM radio tuner, all features available on some Android devices.
  
• Google Voice: Will the masses adopt it? - It's been in private beta for a long time and is now GA for the masses.  I've indicated before I've a big fan and user and use it as my primary number for business.
  
• 10 things: Using Amazon Web Services in the enterprise - Looking to leverage the cloud?  Things to think about.
  

links for 2010-06-25: iOS4 vs Android Multitasking; Tiered Data Pricing; AWS SNS User Survey; eGate and Java CAPS future options

• Apple iOS 4 vs. Android Multitasking: Which Approach Is Better for Users? - A good highlighting of the pros and cons.  What it fails to note is that the con against Android (grace and vision?) can and will be addressed while the con against Apple will likely not be as it goes against Apple's approach of dumbing it down for the user.
• Report on AT&T's tiered data pricing forgets the innovation factor - It is interesting how the cycle seems to be going back to tiers and not unlimited.
  
• AWS SNS User Survey - An opportunity to give feedback to Amazon on their Simple Notification Service.  I'd advise not simply asking for all the features of your favorite JMS provider, think Cloud!
• NextGate webinar for Sun Integration customers - Good information on what the options are for future strategies around eGate, ICAN, SRE, and Java CAPS.  They are holding it again next week.

links for 2010-06-16: Scott McNealy on 25 years of .com; Oracle cutting more Sun jobs; NetBeans 6.9 released; Twitter outages; Linux share increases; JPL in the clouds

• Scott McNealy’s Top 15 Reasons He’s Surprised We’re Celebrating the 25th Anniversary of the Web - It is always great to see Scott on stage, especially when he has a top-10 (errr 15) list.
  
• Steven Strogatz on the Elements of Math - I blogged most of his individual entries earlier this year, now you can get them all in one place.  Good stuff for those curious about math.
  
• Oracle Cutting More Jobs Tied To Sun Deal; Taking Big Charge - Seems to be tied to LEC completing in European and Asian countries, but still sad to see.
  
• Oracle Delivers NetBeans IDE 6.9: Expanding Support for JavaFX 1.3 and OSGi - It is good to see that NetBeans doesn't seem to be withering away.  But the preferred IDE for Fusion is still JDeveloper and Eclipse.
  
• Twitter: Sorry, but outages come with bigger improvement plan - Interesting analogy to road construction.
  
• Linux inches up on desktop, holds steady on servers - The only OS to gain share in May.  I can understand Windows dipping but I'm surprised Mac OS dipped too.
  
• NASA JPL, robots and the AWS cloud - Some interesting applications of EC2, SimpleDB, and S3.
  

Event Processing in the Cloud - Combining Esper with AWS SNS

The folks at Amazon Web Services released their Simple Notification Service in beta a little over a month ago, and I used it to create a loosely coupled weather notification system by publishing weather events to an SNS topic.  It is working great, but I started to think of other types of situations I'd want to monitor and have notifications for, and what better way to process these weather events than with a Complex Event Processing (CEP) engine?  While I could keep extending my shell script, that would be a bit of a hack and I wanted something cleaner that would more easily last the test of time.

Event processing is in many ways a natural extension of messaging infrastructure as the latter is typically used for the passing of events between systems and so having an easy way to plug in or use an event processor makes a lot of sense.  In my case, I wanted to be able to feed periodic weather observations, the raw events, into the messaging infrastructure, then be able to dynamically define the processing rules for those events, and then be able to publish any notifications or "complex events" back into the messaging infrastructure where I could then subscribe or receive them for e-mail/SMS notifications or to kick off other processes or logic.

So, how to put this all together?  There are CEP engines from many of the major middleware vendors, but on my shoestring budget I wasn't about to go spend 5-6 figures on software just to process events for my weather station!  At Sun, as part of the OpenESB project we had developed the Intelligent Event Processor in open-source so it was a natural choice, but it requires the complete ESB infrastructure and can't run standalone.  As I was already using SNS for my messaging and was aiming to make what I created Cloud friendly (see below), I didn't want to bring along an entire ESB, so I needed something else.  Another open-source CEP engine I was familiar with is Esper, and after a quick refresher on its capabilities, I decided it would be perfect.

Esper provides a simple to use but very powerful Java API for configuring the events, queries, patterns, etc. and so I set about creating the interface between SNS and Esper.  What I ended up building has the following capabilities:

• A simple (HTTP/JSON) API for dynamically configuring the events and queries/statements in the engine.
• Receives events from an SNS topic via HTTP and feeds them into Esper.  The publisher of the raw events just publishes to the SNS topic like it was before and I just create a new subscriber for the CEP engine, thus leveraging the beauty of a loosely coupled messaging system.
• Esper processes the events per the configured queries.  As the queries generate results, the results are published to a specified SNS topic which can result in a notification e-mail or kicking off some other process or logic, again leveraging the decoupled nature of SNS.

Now, Esper provides a variety of ways to configure the engine, XML being one of them, but you'll note above I chose to create an HTTP/JSON API to serve as the interface to Esper's Java API for doing this configuration.  This was to be both Cloud friendly and to make the configuration dynamic.  This allows the idea of an event processor in the Cloud that is just available to be configured dynamically, receive events, and publish results, all with no installation or maintenance of the engine or other infrastructure.  One never even needs to log in to a machine to mess with any configuration files.

With this created, I was then able to easily configure the following:

1. I had an existing script called by cron every 10 minutes that sent the weather observation to the Weather Underground.  I added a few lines of code to this script to publish the observation to an SNS topic as well.
2. I created a subscription to the SNS topic that notifies the event processor using HTTP/JSON of each event.
3. I configured my "Cloud CEP" with the following JSON to define the events it would be receiving:

{     "Type": "EventConfiguration",     "Name": "WeatherEvent",     "Fields": [["temp", "double"], ["humidity", "double"], ["dewpoint", "double"]] }

4. I configured a query to use for processing the events, this one telling me the high and low for the past 24 hours every 12 hours at midnight and noon, and sending the result to the specified SNS topic.  The statement is Esper's SQL like syntax for specifying queries:

{     "Type": "ListenerConfiguration",     "Name": "Every 12 hours high and low",     "Statement": "select max(temp) as High, min(temp) as Low from WeatherEvent.win:time(24 hours) output at (1, */12, *, *, *)",     "ActionType": "SNS",     "SNSTopicArn": "arn:aws:sns:us-east-1:444520459559:WeatherEventOutput" }

5. I then created a subscription to the WeatherEventOutput topic to send myself an e-mail.

I now get an e-mail at midnight and noon each day telling me what the high and low were for the past 24 hours.  If I want to know any time the temperature changes more than a certain amount in a given period, or if the temperature shows a trend of getting closer to the dewpoint (i.e. it is about to rain), I can add this to the processing with just a simple HTTP/JSON configuration request.  And with the power of Esper, those scenarios and a lot more are possible for my weather data or any other event data one might have.

The further beauty of this is that while I happen to have this running on my server at home, because of the way it is built using SNS and HTTP, it could be located anywhere on the internet whether hosted on EC2 or your favorite provider, or a "Cloud CEP" service available to all.

Note also that what I've written is by no means tied to SNS either.  It just happens to be Cloud based messaging infrastructure that is convenient to use and gives the benefits of loose coupling.  The input to my Cloud CEP is just an HTTP request and I've written an e-mail and HTTP handler so that generated events can go direct as well instead of going to SNS.

So what do you think?  Do you have scenarios where a Cloud CEP would be useful?  Would you like to try out what I've built thus far?  Feel free to leave a comment or contact me at kschmidt at techrunning dot com.

links for 2010-05-07: Red Hat vs VMware; Cloud washing; Netflix selects AWS; Android in iPhone 3G

• VMware and Red Hat: The war for the data center - Increasingly overlapping portfolios force them to compete.  VMware's next acquisition SUSE Linux to get an OS?
• May the cloud washing begin: Enterprise software giants graze on cloud startups - "The burden of cloud computing proof rests with the large enterprise software folks. Your job will be to distinguish from mere cloud washing and real IT value."
  
• Netflix Selects Amazon Web Services to Power Mission-Critical Technology Infrastructure - They'd been using AWS for more than a year but are expanding their use.
  
• Android now running on the iPhone 3G - It is still a bit rough (no audio support, but coming in days the story says), but getting close to being able to really run Android on an iPhone 3G

links for 2010-04-29: AWS to be as big as Amazon retail?; Steve Jobs on Flash; Backlash against PowerPoint; Flash History

• Amazon Looks to Widen Lead in Cloud Computing - Opens a new data center in Singapore, but also says "Amazon Web Services can be as big as our retail business, in the fullness of time".
  
• Steve Jobs thoughts on Flash - He makes several good points, but calling Adobe closed and proprietary given Apple's tools target a single platform (vs many) and the closed nature of the App Store (vs anyone can create an app without approval) is a bit of the pot calling the kettle black.  And if third party software not taking advantage of new APIs and features is really so terrible, let the market determine that and decide.
• We Have Met the Enemy and He Is PowerPoint - "Commanders say that the slides impart less information than a five-page paper can hold, and that they relieve the briefer of the need to polish writing to convey an analytic, persuasive point. Imagine lawyers presenting arguments before the Supreme Court in slides instead of legal briefs."
• Flash History - A good reminder that Flash did fill a need, but he also had a zinger directly at Cupertino: "... a platform with a vendor who gets to decide what’s allowed to run is profoundly uninteresting to me anyhow."
• News Analysis: Salesforce.com and VMware Up The Ante In The Cloud Wars With VMforce - Good summary.  Pits the Java offerings vs Azure.