Tuesday, May 25, 2010

links for 2010-05-25: MPEG-LA sued; Stonebreaker's VoltDB; Apple playing catch-up?; Android to be "Windows" of phones?

Friday, May 21, 2010

links for 2010-05-21: Sayonara iPhone; Google and Adobe gang up on Apple

  • Sayonara, iPhone: Why I'm Switching to Android - "The Android OS is already outselling iPhone OS in the United States. Now it's blowing past Apple in terms of the technology it's delivering."  "What makes this even more insulting is that Jobs tries to dress up his selfishness as a kind of altruism. He says it's all about creating a beautiful experience, that while he may be selling you an intentionally crippled device, he's doing it for your own good."
  • Strange Bedfellows, Google And Adobe Gang Up On Apple - It was a bit of a change from day 1 to day 2.

Thursday, May 20, 2010

links for 2010-05-20: Android on the iPhone; Google IO; Conan visits Google

Tuesday, May 18, 2010

links for 2010-05-18: Oracle releases new version of MySQL Enterprise; Amazon stealing the Cloud; Book on Facebook; Future of Development; Flavors of Cloud Computing

Monday, May 17, 2010

links for 2010-05-17: Google I/O is the new JavaOne; Government using the Cloud; Caspio = Access + Cloud?; Jobs engages blogger

Thursday, May 13, 2010

Event Processing in the Cloud - Combining Esper with AWS SNS

The folks at Amazon Web Services released their Simple Notification Service in beta a little over a month ago, and I used it to create a loosely coupled weather notification system by publishing weather events to an SNS topic.  It is working great, but I started to think of other types of situations I'd want to monitor and have notifications for, and what better way to process these weather events than with a Complex Event Processing (CEP) engine?  While I could keep extending my shell script, that would be a bit of a hack and I wanted something cleaner that would more easily last the test of time.

Event processing is in many ways a natural extension of messaging infrastructure as the latter is typically used for the passing of events between systems and so having an easy way to plug in or use an event processor makes a lot of sense.  In my case, I wanted to be able to feed periodic weather observations, the raw events, into the messaging infrastructure, then be able to dynamically define the processing rules for those events, and then be able to publish any notifications or "complex events" back into the messaging infrastructure where I could then subscribe or receive them for e-mail/SMS notifications or to kick off other processes or logic.

So, how to put this all together?  There are CEP engines from many of the major middleware vendors, but on my shoestring budget I wasn't about to go spend 5-6 figures on software just to process events for my weather station!  At Sun, as part of the OpenESB project we had developed the Intelligent Event Processor in open-source so it was a natural choice, but it requires the complete ESB infrastructure and can't run standalone.  As I was already using SNS for my messaging and was aiming to make what I created Cloud friendly (see below), I didn't want to bring along an entire ESB, so I needed something else.  Another open-source CEP engine I was familiar with is Esper, and after a quick refresher on its capabilities, I decided it would be perfect.

Esper provides a simple to use but very powerful Java API for configuring the events, queries, patterns, etc. and so I set about creating the interface between SNS and Esper.  What I ended up building has the following capabilities:
  • A simple (HTTP/JSON) API for dynamically configuring the events and queries/statements in the engine.
  • Receives events from an SNS topic via HTTP and feeds them into Esper.  The publisher of the raw events just publishes to the SNS topic like it was before and I just create a new subscriber for the CEP engine, thus leveraging the beauty of a loosely coupled messaging system.
  • Esper processes the events per the configured queries.  As the queries generate results, the results are published to a specified SNS topic which can result in a notification e-mail or kicking off some other process or logic, again leveraging the decoupled nature of SNS.
Now, Esper provides a variety of ways to configure the engine, XML being one of them, but you'll note above I chose to create an HTTP/JSON API to serve as the interface to Esper's Java API for doing this configuration.  This was to be both Cloud friendly and to make the configuration dynamic.  This allows the idea of an event processor in the Cloud that is just available to be configured dynamically, receive events, and publish results, all with no installation or maintenance of the engine or other infrastructure.  One never even needs to log in to a machine to mess with any configuration files.

With this created, I was then able to easily configure the following:
  1. I had an existing script called by cron every 10 minutes that sent the weather observation to the Weather Underground.  I added a few lines of code to this script to publish the observation to an SNS topic as well.
  2. I created a subscription to the SNS topic that notifies the event processor using HTTP/JSON of each event.
  3. I configured my "Cloud CEP" with the following JSON to define the events it would be receiving:
  4. {     "Type": "EventConfiguration",     "Name": "WeatherEvent",     "Fields": [["temp", "double"], ["humidity", "double"], ["dewpoint", "double"]] }
  5. I configured a query to use for processing the events, this one telling me the high and low for the past 24 hours every 12 hours at midnight and noon, and sending the result to the specified SNS topic.  The statement is Esper's SQL like syntax for specifying queries:
  6. {     "Type": "ListenerConfiguration",     "Name": "Every 12 hours high and low",     "Statement": "select max(temp) as High, min(temp) as Low from WeatherEvent.win:time(24 hours) output at (1, */12, *, *, *)",     "ActionType": "SNS",     "SNSTopicArn": "arn:aws:sns:us-east-1:444520459559:WeatherEventOutput" }
  7. I then created a subscription to the WeatherEventOutput topic to send myself an e-mail.
I now get an e-mail at midnight and noon each day telling me what the high and low were for the past 24 hours.  If I want to know any time the temperature changes more than a certain amount in a given period, or if the temperature shows a trend of getting closer to the dewpoint (i.e. it is about to rain), I can add this to the processing with just a simple HTTP/JSON configuration request.  And with the power of Esper, those scenarios and a lot more are possible for my weather data or any other event data one might have.

The further beauty of this is that while I happen to have this running on my server at home, because of the way it is built using SNS and HTTP, it could be located anywhere on the internet whether hosted on EC2 or your favorite provider, or a "Cloud CEP" service available to all.

Note also that what I've written is by no means tied to SNS either.  It just happens to be Cloud based messaging infrastructure that is convenient to use and gives the benefits of loose coupling.  The input to my Cloud CEP is just an HTTP request and I've written an e-mail and HTTP handler so that generated events can go direct as well instead of going to SNS.

So what do you think?  Do you have scenarios where a Cloud CEP would be useful?  Would you like to try out what I've built thus far?  Feel free to leave a comment or contact me at kschmidt at techrunning dot com.

links for 2010-05-13: Ellison on fixing Sun; Fragmenting Linux a bad thing; Apple's cache waning?; Infinity; SAP acquires Sybase

Wednesday, May 12, 2010

links for 2010-05-12: Apple update for iPad WiFi woes; Flash not a CPU hog?; Microsoft responds regarding HTML5/H.264; Rich Green CTO at Nokia; Oracle Office; Adaptive PaaS

Friday, May 7, 2010

links for 2010-05-07: Red Hat vs VMware; Cloud washing; Netflix selects AWS; Android in iPhone 3G

Wednesday, May 5, 2010

links for 2010-05-05: Apple's control freakishness; Chrome update; SOA embraced; Android tablets; Solinsky AR splits

Tuesday, May 4, 2010

links for 2010-05-04: IBM acquires Cast Iron Systems; Scaling open-source; Browser market share; Software patents are good ... for lawyers; Google's cash; Open Data; Android on your TV

Monday, May 3, 2010

links for 2010-05-03: Apple/Adobe debate brings in Microsoft; More on HP/Palm; iPhone envy erased by Droid Incredible; iPad reaches 1M sold

Sunday, May 2, 2010

Is Flash costing the American public money in electricity costs?

There has been much furor about the lack of Flash on the iPhone, iPod Touch, and iPad with Apple espousing a number of reasons for that, a key one being performance and resource utilization.  I wrote about my experience with Flash and some minor improvements a beta of 10.1 seems to bring, but I've observed that even when not watching video, with the number of sites that use Flash for other things the CPU is actually quite busy when one would think it would be idle (i.e. I'm not actively browsing).

To test this out, on a Mac I opened up several windows, each with several tabs opened to sites I'll typically have open during the day including Gmail, Google Calendar, Blogger, ESPN.com, StatCounter, ZDNet, and a handful of others.  I also watched a Dodger game on MLB.com for a bit but closed that window and left the computer basically idle but with the tabs still open in my browser.

At this point the Shockwave plug-in was using about 2-3% of the CPU, not bad.  But 20 minutes later of just sitting otherwise idle, the plug-in was using 10% of the CPU.  During this test I did not wait significantly longer, but previously I've seen the plug-in using 15-20% of the CPU while the machine is seemingly idle.

So, even at a conservative 10% of the CPU being used unnecessarily by Flash on an idle machine, one has to then wonder what that is doing to the power consumption of the machine.  A little quick research revealed that moderate usage of a desktop uses 30-50 watts above idle and a laptop 10-15 watts above idle.  If moderate use is 20-30% CPU, then our 10% CPU is going to be using around 10 watts on average.

Let's keep doing more math to see where this takes us.  The current population of the US is just over 307 million and 76.2 computers per 100 people that is roughly 234 million computers.  If only 10%, or 23.4 million, are used on a daily basis and when they are used 10%, or 2.34 million of them visit sites each day for an hour that use Flash and have this CPU waste, that is 23,400 kilowatt hours each day, or 8.54 million per year.

To make that number more meaningful, at an electricity price of $0.10 per kilowatt hour, the use of Flash is costing the American public $854K per year.  And the 10%'s and single hour I used above are likely clearly on the low end of what the actuals are and my analysis ignores computers used at work. So the actual cost is likely well into the millions.  In the grand scheme of things, $854K isn't that much for the entire country, but it is still sobering to think about.

Now, is this all Adobe's fault?  If their software is indeed buggy and inefficient they do shoulder some of the blame, but I would argue that Flash is likely used in many situations it isn't needed and that it is poorly written Flash apps that is a big factor too.  Poorly written AJAX apps running in your browser could cause the exact same issues.

So what can one do about it?  Well, you can choose to do nothing as it is really only costing you at most pennies in extra electricity costs (although what about wear and tear on your computer from heat and the fan running?), but your other alternatives are to not install the Flash plug-in and forgo benefiting from sites that use Flash, or installing a Flash blocker.  I've done the latter and it blocks all Flash applets by default but allows you to white-list sites or selectively enable specific applets.

Using the blocker it is interesting to see what sites use Flash and the list includes Google Mail, ESPN.com, ZDNet, java.sys-con.com, StatCounter, and more.  Do each of these really need to use Flash?

Chris Solinsky shatters the 10000m American Record

An historic event occurred last night at Stanford as Chris Solinsky ran 26:59.6 to shatter the American Record that was 27:13.98.  It was a fabulous run but what was more surprising was that this was his debut at the distance and the race had been billed as Galen Rupp's attempt to break the record and Chris upstaged the runner from the sister training group.  Also, Chris becomes the first non-African born runner under 27 minutes.

The IAAF story above has a good recounting of the race and you can see Flotrack's finish of the race but a quick summary is the rabbits took it through mid-race on 27:10 or so pace and then Rupp took over the lead and pulled everyone along with Chelanga, Salel, and Solinsky close behind (Bairu may have been there too).  Rupp made a move with 3 or so to go, Solinsky went with him and passed him with about 2 to go and closed in about 1:56.  Rupp finished 4th in 27:10, breaking Meb's former AR time, but Salel and Chelanga beat him, Chelanga setting a new Collegiate Record.

Quite an impressive race, especially considering it was his debut, but clearly shows he is likely in sub-13 5000m shape which it appears is his goal for the year, or rather breaking Dathan Ritzenhein's AR of 12:56 is the goal.  He'll be running one in Oslo in early June and then another at the Pre-Classic in July.

It is shaping up to be a great summer for American distance running.  Look for Rupp to come back strong and Matt Tegenkamp who went sub-13 last year should have a another go at the AR.  Enjoy!