Benchmarking the iPhone, and lots of other devices too

Benchmarking has been around for years, and I started playing around with writing them back in college.  I recall an assembly language class were the assignment was to implement the Sieve of Eratosthenes in as few bytes as possible. I "won" the assignment by doing it in 24 (or was it 25, too many years ago!) bytes, but that got me started on benchmarking as I instrumented the code to time how long it took and then proceeded to run it on all the different machines I could to see which one was the fastest.

Clearly though, having a benchmark written in 8086 assembly language had limited usefulness, so in later years I rewrote it in C and added several other standard algorithms to the stable that were executed including calculating PI, a selection sort, and computing Ackermann's function to test recursive performance. I proceeded to run it on every machine I could from PCs running MS-DOS to workstations running Solaris to PPC machines running LynxOS and AIX. In later years, I ran the same benchmark on UltraSPARC, SGI MIPS, and DEC Alpha machines as well as today's dual-core Intel chips. Somewhere along the line I starting writing more in Java than C and I ported the benchmark to Java to see how it compared, and several years ago I began using BlackBerry's and wrote a Java ME version.

As you can imagine given the wide range of machines I've run it on in the past, having an iPhone was just begging to run it there too. While I have creating a proper iPhone app version on my list of things to do, thankfully a jailbroken iPhone makes it pretty easy to run the C and Java versions I already had. A sampling of results appears below, but the summary for the iPhone is that the native version is about 5 times faster than the Java one and ever so slightly faster than a 75 MHz Pentium 5 of yesteryear.

Machine	Language	Math	Array	Sort	Sieve	Ackermann	Total
Motorola L6	Java ME	0.11	0.19	0.23	0.13	0.19	0.17
BlackBerry Curve (OS 4.5)	Java ME	0.95	2.21	2.96	1.72	2.96	1.81
iPhone 3G	Java SE	12.4	4.79	4.64	3.28	5.01	4.95
Intel 486DX2-66 (Linux)	C	10	10	10	10	10	10
SPARC-20 (Solaris)	C	66.4	20.3	17.6	20.1	4.1	12.1
Pentium 5 75 MHz (Linux)	C	21.3	19.6	28.6	23.2	33.4	24.2
iPhone 3G	C	65.9	21	23.4	19.2	46.7	27.9
DEC Alpha 275 MHz (OSF/1)	C	111.4	93.6	63.1	41.2	76.1	68.6
Pentium 600 MHz (Linux)	Java SE	191.9	106.1	99.1	68.9	78.2	96.1
Pentium 600 MHz (Linux)	C	216.4	319.3	219.8	171.5	131.2	194.1
MacBook Pro 2.33 GHz	Java SE	655	1096	794	549	334	587
MacBook Pro 2.33 GHz	C	1089	2413	664	800	1356	1037

You may also view the full list that I have maintained over the years.  You'll notice that I did my initial calibration on a 66 MHz 486 DX-2 so that it scores 10 in each of the tests.

Now, I'll be the first to admit that my benchmark is pretty simplistic and thus shouldn't be be the end all be all testing to compare machines or platforms. The results can also be significantly affected by the quality of the C compiler or JVM being used. However, the results can provide a rough comparison and are at least fun to look at.

For example a SPARC-20 has terrible performance with deep recursion. And as noted above the JVM on the iPhone is about 5 times slower than a native app whereas on an x86 box the JVM is about half as fast as native.

While I've run the benchmark on quite a few machines, I'd love to add more to the list.  If you'd like to run it, here is the binaries in a variety of formats:

• Linux on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.


• OS X on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.


• OS X on ARM/iPhone - Download the binary, use ssh/scp to get it on your jailbroken iPhone, and execute with "./sci <iterations>". The default iterations should be fine.


• Solaris on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.


• Solaris on SPARC - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.


• AIX on RS/6000 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.


• Java SE - Download the jar and execute it on any machine you have a JVM with "java -jar SCIJ.jar <iterations>". On most of today's faster machines iterations should be at least 5,000.


• Java ME devices - Point your phone/device browser at the link and it should prompt to install the application. Run the app and if it executes in less than a few seconds change the iterations to a larger number to make it more accurate in calculating the benchmark.

If you do run the benchmark, please post a comment on this blog entry with the chipset, clock-speed, operating system, and results from each of the tests that the program displays. As soon as someone jailbreaks an iPhone 3G S, it would be great to add that result to the list.

Enjoy!

 

If You Buy a Mac

My recent "Why Buy a Mac" blog about sitting next to someone on a plane running Windows on a Mac received a number of comments so I thought I'd respond to some.

There were several OpenSolaris related comments and I'm all for using it. I've installed the 200805 release in a VirtualBox VM on my Mac as well as native on an old Dell I have. And I recently had an opportunity to get a release candidate for the next release and it is improved but I don't know that it quite has all the apps I want/need. I'm going to continue using/tracking it though and would gladly switch to it as my primary OS when it is "ready". But I don't know that I'd go out of my way to run it on a Mac and certainly wouldn't spend Apple money to buy a new machine to do so.

Related to the above, there was a comment about using Ubuntu. I've used it (or its variants like gOS and Linux Mint) both in VMs and native and feel it is still ahead of OpenSolaris as a "regular user" desktop, but similarly wouldn't buy Mac hardware to run it.

There was also the comment about the person being required to use Windows or that he worked from Microsoft. Actually could have been considering it was a flight leaving Seattle!

And while I'm on the subject of Mac's, see Michael Cote's writeup on tips for a new Mac user. A good read.

Track and Olympics links for 2008-8-12

• Track & Field News Predictions: Men's Top 10s - Bolt predicted to complete the double. Lagat to get the 1500 but 2nd in the 5000 to Bekele.


• Track & Field News Predictions: Women's Top 10s - U.S. Women in the 100, 200, and 400, Dibaba to get the 10000 but 2nd in 5000 to Defar. Jamaica picked in the 4x100!


• Gay removes flash from 100m dash - Kudos to Kobe for knowing to ask about Gay's leg.


• Powell propelled by personal tragedy as he prepares for greatest showdown on earth - Speaking of his WR in his first race after Worlds last year where he shut it down early, "I definitely would have gone under 9.7 but I slowed down because it was a heat."


• Edwin Moses: An Olympic message - We should all listen when one of our greatest speaks: "Without a doubt, there is absolutely no other event on the planet that has so much meaning and can have such an enormous effect on so many people."


• Where have all the spectators gone? - An interesting dynamic that having lower cost tickets could result in people being more willing to buy but then not show up. Some blame is put on corporations that don't use their tickets too.


• U.S. men gymnasts get unexpected bronze - Finishing on Pommel wasn't pretty, but it held up. Women's team final tonight (lineup).


• Doping control is bleeding Asafa Powell dry - I'm glad they are testing, but this seems a little extreme: "About two days ago, I got pretty upset, because since I've been here, they have tested me four times, and took blood, a lot of blood, ... I'm saying that they are taking so much blood that we are going to be very weak for the finals of the 100 meters."


• Mac doesn't want to play Olympic Games - Microsoft apparently doesn't care about Mac PPC users. They don't care about Linux users either, but that is no surprise.

MacBook Pro, VirtualBox, and OpenSolaris

OpenSolaris was a big part of CommunityOne this year and as I mentioned in blogging about it, I was installing it and would be blogging about it soon. Well, it isn't as soon as I wanted (something about Java CAPS 6 has kept me busy) but here is an update of where I've gotten.



I'm running on a Mac, so went the route of installing OpenSolaris 200805 in VirtualBox 1.6. Getting and installing VirtualBox was straightforward, and then installing OpenSolaris also went very well. This blog entry goes into a bunch of detail on an identical type of configuration so I won't attempt to capture it all here again.



After the install, I wanted to get the desktop to use the full screen real estate I have and so went about finding out how to do that. Followed the steps and voila I can run full screen and have it appear that I have running OpenSolaris natively on the Mac.



So far I've installed OpenOffice.org via IPS which went just fine, and I'll be looking to install more things soon.



My outstanding issue is that I don't have audio working yet. By default audio was disabled in VirtualBox so I enabled it and tried both options it presents there but OpenSolaris hasn't recognized either. If anyone has any tips, please let me know.



I have also started installing OpenSolaris on a spare Dell D600 I have. I had LinuxMint on it already (Ubuntu based distro) and when I was done installing I could no longer boot to Linux! I found this blog that explained how to put the Linux Grub back and have it provide for booting to OpenSolaris which worked fine. I haven't gotten wireless working on it yet so that is the next project (any tips welcome!)

Olympic Thoughts

I am a big sports fan and look forward to the Summer Olympics every 4 years and this year is no different. Track and field is my favorite event but I'll watch most any sport and gymnastics, volleyball (indoor and beach), and tennis are high on the list. August 8th thru 24th will find me glued to the TV at odd hours of the day/night.



Why do I say odd hours? I'm sure most of you are aware but for those that have been holed up in a cave somewhere, this years summer games will be in Beijing, China. That means "prime time" events occurring in the wee hours of the morning, and I'm sure we can depend on NBC to delay the majority of the coverage so we can play the game of avoiding results all day long so we can watch it in our prime time. What fun.



When the games were awarded to Beijing, there were immediate concerns given China's less than stellar record on human rights, openness, and pollution, but all the right people said all the right things and the concerns went away ... at least for a time. As we grow closer (124 days) to the games, those concerns are coming back.



The pollution problem, while better, doesn't appear to be fixed, there continue to be human rights concerns, and there is talk that media covering the Olympics will not be allowed to have full internet access and take pictures of or report everything they see. There is talk that some may boycott the opening ceremony (which would be far better than boycotting the whole games!).



A sampling of stories over just the past few weeks includes:

• It's time for Rogge to speak out - "IOC president remains silent while world leaders express their discontent about China as the Olympics approach."


• Waiting to inhale in Beijing - "After a 45-minute jog on some of the same streets where the marathons will be run, this reporter experienced minor discomfort and heaviness in my lungs and griminess on my skin."


• IOC flouting Olympic charter by ignoring Tibet issue: HRW - "It also asked the IOC to publicly assess the extent to which current human rights violations linked to the preparation of the Games were violating the commitments made by China at the time of its bid to host the Olympic Games, and to establish a standing mechanism to address human rights concerns."


• Stop blocking the internet, Olympics committee tells China - "China routinely blocks access to certain sites on the internet and can implement at will a blacklist of words that cannot be found and will crash a search engine." and "Wikipedia remained outside the firewall — apparently because of its many references to such sensitive issues as the 1989 Tiananmen Square crackdown, the Dalai Lama and Tibet. Many international blogs cannot be accessed because their host servers are blocked." (by writing this blog I may be getting Sun's server on a blacklist?)


• China may ban live broadcasts during Games - "Earlier this week, however, officials with the Beijing Olympics Organizing Committee, or BOCOG, told executives at BOB that the live shots [of Tiananmen Square] were canceled, ..."


• IOC inspectorate confirms that games will not be censored - "China routinely uses 30-second to one-minute delays to control broadcasts on state-run TV. The Olympic torch lighting ceremony last month in Greece was disrupted by a protester who ran up behind a top Chinese official giving a speech. The image was seen around the world but has not been shown on state TV in China."


• Gebrselassie pulls out of Olympic marathon due to pollution fears - "The pollution in China is a threat to my health and it would be difficult for me to run 42 kilometers in my current condition"

On the plus side, it appears NBC is going to expand their digital coverage but I'm sure it will be at a price.



So there seems to be some movement and continued promises, so I am hopeful that the games will occur without incident, but I have to say I'm not optimistic. But we'll see.

Is Open-Source "Freetarded"?

Is open-source "freetarded"? I just ran across an article by Christopher Keene pointing out that open-source has to be about more than free and that vendors have to start thinking that way. And he is absolutely correct!



If software is only free, you are addressing only part of having software be valuable and having a low TCO. As the article above points out, if you can't get the software installed and working, or you can't find talent to be able to work with the software, that "free" software starts to get pretty expensive.



In order to fully deliver on the promise of open-source, software must have three qualities:

1. Free and easy access - This is kind of assumed with open-source, but must be stated so we don't forget it. For most users it isn't about the source, but rather is about free and easy access to a binary or distribution. While I have to confess that I have downloaded source and built my own binaries for a variety of apps and even newer versions of Linux kernels (and I'm probably even proud of this fact), most users don't have the aptitude or desire to do that, so having binaries is a must.


2. Easy to use - This is extremely important and sometimes forgotten with open-source projects. If you download something and can't get it installed, or after installation can't get it configured or working, you can end up wasting a significant amount of time losing the "free" aspect of the software. Worse, if that initial experience is bad you may not come back again to try it and worse yet, word of mouth and hmmm, blogging, can spread the word about a product being difficult to use very quickly.


3. Vibrant community - This last piece is also very important. Without it you don't get the collaboration and feedback that help drive innovation and help make the software better. Furthermore, if there is no community, adoption suffers and that leads to small pools of talent resulting in higher costs for organizations wanting to use the software.

You only have to look as far as the Apache Web Server and Apache Tomcat, or more recently GlassFish to see examples of where these three qualities have led to successful open-source projects that have had and are gaining widespread adoption and use and have driven down TCO for organizations that are implementing the software. And while it isn't as large or mature as those mentioned, Open ESB is also applying the qualities to create a successful open-source project.



A related question that deserves its own blog entry but I'll mention here is, if software is free, how does a vendor make any money off it? See Mark's blog from some recent interaction with our customers to see their views on paying for "free" open-source software, and stay tuned for more on the subject.

Visible Source Software

I spent the last three days at Gartner's Application Architecture, Development, and Integration (AADI) Summit in Nashville. I had attended the AIWS conference in December and many of the themes were similar but there was some new information and predictions made that was great to hear.

I also had a chance to talk with several analysts, one of which, Mark Driver, has open source software as one of his focus areas. Something we talked about and he subsequently presented on is that many companies are positioning themselves as building open source software but that it isn't always the case that it is really open source. In fact, he says that Gartner will be changing their identification of some from open source to something else. This is because companies are using open source licenses that aren't truly open, either through not being able to use and extend the code or not being able to contribute back in any way.

Mark liked to think of this as accessible source software (doesn't result in a good acronym, although perhaps appropriate) and I suggested we talk about this kind of software as visible source software. In his presentation he talked about this as gated source software.

I like visible source as I think it properly distinguishes between those companies that are truly open, have a community, and opportunity for "outsiders" to join the community and contribute, and those that simply make their source available and try to benefit from the open source label but don't really allow for anyone to directly participate or become a contributor or committer. For all intents and purposes, the software is simply "visible" to others.

At Sun, we clearly believe in true open source software and this is evidenced by the contributors and committers in Project Open ESB. There are 7 companies listed as Community Partners, many of which are building components for the platform and several of those in open source with commit rights to the project, plus a growing number of individuals with commit rights that are building and contributing to components being developed in the project and community.

If you are interested in joining a truly open source community, come join Project Open ESB.

2016 Olympics

I was reading the Track and Field News website and saw a link to an article about whether the US should nominate Chicago or Los Angeles as the host city for the 2016 Olympics.



Now I wasn't expected an unbiased and rational story since it is in a Midwest newspaper, but the support for Chicago to be the host isn't even close and frankly isn't that strong. And not that I'm terribly unbiased either, but there are many reason Los Angeles would make a much better host city:

• Successfully hosting the games twice before with the 1984 games perhaps the most successful and model for future games since.


• Absence of heat and humidity; this is an athletic event after all. And to the rebuttal that the smog will kill the athletes, smog levels are way down from 20 years ago when they weren't nearly as bad as expected for the 1984 games.


• If you are concerned with the "experience", how can you beat sandy beaches and the Pacific Ocean?


• And as far as sports history goes, I don't even know why the article tried to justify Chicago beating Los Angeles in this regard. Just to name a few:




• The aforementioned Olympic games, including Carl Lewis' 4 golds in 1984.


• The Lakers 9 championships spanning many years and different superstars from Wilt to Kareem to Magic to Shaq to Kobe and more.


• The Dodgers 5 World Series including 1988 with Gibson's memorable home run. And many stars from Koufax to Fernando to Hershiser and more.


• The Angels of all teams winning the World Series in 2002.


• The dynasty that is UCLA with basketball during the Wooden era and beyond (11 championships and 2 straight trips to the final 4 the last 2 years) but across all sports with more national championships than any other NCAA institution with 99.


• The history of USC football and 7 Heisman trophy award winners from the school.

And when it comes right down to it, who really thinks the Cubs are going to win the World Series by 2016? :)



In the end, the right choice for the USOC would be to submit Los Angeles as the candidate, but the right choice isn't always made.

Routers in Space

Since Infoworld has gone online only, I was reading their site and one of the headlines was about putting routers in space. It certainly makes sense as a way to cut down on the traffic, but it makes one think, when will we ultimately put servers in space too? Wouldn't that cut down the traffic and improve performance even more?



It probably isn't going to happen, or at least be widespread, in the next couple of years but I don't doubt it will happen. Who will be the first Space Web/App Hosting Provider (SWAHP)?