Benchmarking has been around for years, and I started playing around with writing them back in college. I recall an assembly language class were the assignment was to implement the Sieve of Eratosthenes in as few bytes as possible. I "won" the assignment by doing it in 24 (or was it 25, too many years ago!) bytes, but that got me started on benchmarking as I instrumented the code to time how long it took and then proceeded to run it on all the different machines I could to see which one was the fastest.
Clearly though, having a benchmark written in 8086 assembly language had limited usefulness, so in later years I rewrote it in C and added several other standard algorithms to the stable that were executed including calculating PI, a selection sort, and computing Ackermann's function to test recursive performance. I proceeded to run it on every machine I could from PCs running MS-DOS to workstations running Solaris to PPC machines running LynxOS and AIX. In later years, I ran the same benchmark on UltraSPARC, SGI MIPS, and DEC Alpha machines as well as today's dual-core Intel chips. Somewhere along the line I starting writing more in Java than C and I ported the benchmark to Java to see how it compared, and several years ago I began using BlackBerry's and wrote a Java ME version.
As you can imagine given the wide range of machines I've run it on in the past, having an iPhone was just begging to run it there too. While I have creating a proper iPhone app version on my list of things to do, thankfully a jailbroken iPhone makes it pretty easy to run the C and Java versions I already had. A sampling of results appears below, but the summary for the iPhone is that the native version is about 5 times faster than the Java one and ever so slightly faster than a 75 MHz Pentium 5 of yesteryear.
| Machine | Language | Math | Array | Sort | Sieve | Ackermann | Total |
|---|---|---|---|---|---|---|---|
| Motorola L6 | Java ME | 0.11 | 0.19 | 0.23 | 0.13 | 0.19 | 0.17 |
| BlackBerry Curve (OS 4.5) | Java ME | 0.95 | 2.21 | 2.96 | 1.72 | 2.96 | 1.81 |
| iPhone 3G | Java SE | 12.4 | 4.79 | 4.64 | 3.28 | 5.01 | 4.95 |
| Intel 486DX2-66 (Linux) | C | 10 | 10 | 10 | 10 | 10 | 10 |
| SPARC-20 (Solaris) | C | 66.4 | 20.3 | 17.6 | 20.1 | 4.1 | 12.1 |
| Pentium 5 75 MHz (Linux) | C | 21.3 | 19.6 | 28.6 | 23.2 | 33.4 | 24.2 |
| iPhone 3G | C | 65.9 | 21 | 23.4 | 19.2 | 46.7 | 27.9 |
| DEC Alpha 275 MHz (OSF/1) | C | 111.4 | 93.6 | 63.1 | 41.2 | 76.1 | 68.6 |
| Pentium 600 MHz (Linux) | Java SE | 191.9 | 106.1 | 99.1 | 68.9 | 78.2 | 96.1 |
| Pentium 600 MHz (Linux) | C | 216.4 | 319.3 | 219.8 | 171.5 | 131.2 | 194.1 |
| MacBook Pro 2.33 GHz | Java SE | 655 | 1096 | 794 | 549 | 334 | 587 |
| MacBook Pro 2.33 GHz | C | 1089 | 2413 | 664 | 800 | 1356 | 1037 |
You may also view the full list that I have maintained over the years. You'll notice that I did my initial calibration on a 66 MHz 486 DX-2 so that it scores 10 in each of the tests.
Now, I'll be the first to admit that my benchmark is pretty simplistic and thus shouldn't be be the end all be all testing to compare machines or platforms. The results can also be significantly affected by the quality of the C compiler or JVM being used. However, the results can provide a rough comparison and are at least fun to look at.
For example a SPARC-20 has terrible performance with deep recursion. And as noted above the JVM on the iPhone is about 5 times slower than a native app whereas on an x86 box the JVM is about half as fast as native.
While I've run the benchmark on quite a few machines, I'd love to add more to the list. If you'd like to run it, here is the binaries in a variety of formats:
- Linux on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
- OS X on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
- OS X on ARM/iPhone - Download the binary, use ssh/scp to get it on your jailbroken iPhone, and execute with "./sci <iterations>". The default iterations should be fine.
- Solaris on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
- Solaris on SPARC - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
- AIX on RS/6000 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
- Java SE - Download the jar and execute it on any machine you have a JVM with "java -jar SCIJ.jar <iterations>". On most of today's faster machines iterations should be at least 5,000.
- Java ME devices - Point your phone/device browser at the link and it should prompt to install the application. Run the app and if it executes in less than a few seconds change the iterations to a larger number to make it more accurate in calculating the benchmark.
If you do run the benchmark, please post a comment on this blog entry with the chipset, clock-speed, operating system, and results from each of the tests that the program displays. As soon as someone jailbreaks an iPhone 3G S, it would be great to add that result to the list.
Enjoy!
