Do not run after misleading benchmarks

10 07 2007

I just had to find some time to write about this. This post is aimed to help anyone who feels elevated and impressed by the trendy benchmarks between programming languages and/or frameworks. Benchmarks are tools to help us judge. They have a purpose to serve. But using them to their lowest significance and publishing those results to the community does not help anybody. I’d rather say misleads the rookies. And let’s face it, there are more rookies in the industry right now than anyother time. High level MVC frameworks are starting to get in good stable shapes, enabling the developers do more with less. But the problem is doing more with knowing less is not good and neither is sustainable.

Let’s have a look at some posts that proves the point. Someone benchmarked Code Igniter, Cake PHP and Symfony in this blog. All the fuss is about, these three framework printing “Hello World”. This benchmark uses artillary to kill mosquitos, then honours and ranks the million dollar artillaries for doing that better that each others. And then (the catch) gets great appreciation too. What is the point of printing Hello World with RAD tools? Do you benchmark a sniper rifle, a rail gun and an AK-47 in an indoor fight and rank them which one is better? Aren’t they built for long range?

Some people did mention in the reaction that  a benchmarks like this should involve practical use. Like DB operations, ORM use, to find them perform what they are built to perform and compare them there. My intention is not to blame efforts. My point is, how many rookies gets distracted by these type of benchmarks? I say, many. Just read the comments. There are lot more examples. This one is at least,  amongst PHP frameworks. People are compairing cross language and cross purpose tools like that.

If you are either impressed or repelled by the arguments up to now, I suggest you to listen to this speech of brian d foy about benchmarking, given in Nordic Perl Workshop 2007 a few months ago. This is one of the most interesting speech I ever heard. He starts by saying, why you should never use benchmarking ever in your life. Well not ever, but not while your brain is off, as he later clarifies. And also advices to use profiling, which will help you more than benchmarking.

And about speed; speed is a very relative metric for judgement. It is sensible that RAD tools give away little bit of execution speed for developement speed. Symfony guys posted a good explanation in their blog to clear what is how and why. My final thoughts: there are lot of good frameworks to choose from, choose depending on your particular needs. But above all, choose with realistic expectations.

Blogged with Flock

Tags: , , , , , , , , , ,

uʍop ǝpısdn ‘ʎǝʞuoɯ ǝpoɔ ɐ ɯɐ ı

5 07 2007

.ʎǝʞuoɯ uʍop ǝpısdn uɐ ǝʞı1 buıpoɔ ɯɐ ı ǝɹǝɥʍ ǝbɐɔ ʎɯ oʇ ʞɔɐq ʇǝb sʇǝ1 .pɐq ooʇ .ʇɐɥʇ ʇnoqɐ uǝʌǝ ǝʇıɹʍ oʇ ʇobɹoɟ ı .ʎ1nظ pu2 uo ʎq pǝssɐd ʎɐpɥʇɹıq ʎɯ .ʎɐʍʎuɐ uʍop ǝpısdn buıbuɐɥ sı bo1q ʎɯ os .sʎɐp ʍǝɟ ɐ ɹoɟ ʎsnq ɯɐ ı .bo1q uʍop ǝpısdn uɐ buıʇıɹʍ ɯoɹɟ ɟ1ǝsʎɯ d1ǝɥ ʇou p1noɔ ı

Blogged with Flock

Tags: ,