A research on Web 2.0 webserver demographics

8 06 2007

Background: I had an interesting chat with one of my friends about what technology Web 2.0 uses and promotes. Does it break any trend or at least started to break any in the field of web servers? So for the welfare of science, technology and humanity, I decided to do a research. I can knock all the Web 2.0 web sites to learn about their web servers info and analyze to find out if there is really any significant visible trend or not. Alexa pays their researchers a lot of money to do this type of research. Anyway I am doing it for free this time.

Plan: So, I had to write a web crawler that makes a list of all the web 2.0 sites some how and knocks all of them at the head (pulls the http header only) to know what web server they use. I chose eConsultants.com to create the list. They have a list of all Web 2.0 sites like quite a few sites do, as you may know. And they are easier to crawl as they are least bloated than the others (and I cannot pull data from websites made with flash).

Research: So crawl i did. And found a list of 1269 sites in total. So you know how many Web 2.0 web sites are there. Hang on a minute! What makes them qualified as Web 2.0 sites? Lets leave that responsibility on eConsultants.com. But if you want to know what I think about Web 2.0, here you go, I found these comments in a Digg story:

Digger X: What exactly is Web 2.0? What kind of features can I expect? I keep hearing about this buzz, but I’m not sure exactly what it is.
Digger Y: web 2.0 is a new buzz word that will allow startups to get funding again if they can tag themselves as web 2.0 If your website has gradient colors and uses ajax you’re already web 2.0 baby!!!

Okey, so our mind is clear again. Here is the list of Web 2.0 websites. And another web crawler retrieved web server info from all these sites and created a Web 2.0 list with webserver info. Then a text analyzer program I wrote, made me a sorted list of all the web servers.

Result: Final list is not that big, so I can post it here:

725 (57.13%) ==> Apache
176 (13.87%) ==> Microsoft-IIS
173 (13.63%) ==> unknown
52 (4.10%) ==> Lighttpd
37 (2.92%) ==> Apache-Coyote
25 (1.97%) ==> Mongrel
10 (0.79%) ==> nginx
7 (0.55%) ==> Zope
7 (0.55%) ==> Jetty
6 (0.47%) ==> GFE/1.3
6 (0.47%) ==> LiteSpeed
6 (0.47%) ==> Resin
4 (0.32%) ==> Oversee Webserver v1.3.18
3 (0.24%) ==> GWS/2.1
2 (0.16%) ==> AOLserver/4.0.10
2 (0.16%) ==> Apache-AdvancedExtranetServer
2 (0.16%) ==> SWS
2 (0.16%) ==> Zeus
1 (0.08%) ==> Web Crossing(r)
1 (0.08%) ==> Juniper Networks NitroCache/v1.0
1 (0.08%) ==> Japache/2.2.4
1 (0.08%) ==> AZTK – dido
1 (0.08%) ==> Web Server
1 (0.08%) ==> TwistedWeb/2.2.0
1 (0.08%) ==> JoyWeb 1.0b1
1 (0.08%) ==> Server
1 (0.08%) ==> LuMriX
1 (0.08%) ==> JWS 1.2
1 (0.08%) ==> Lotus-Domino
1 (0.08%) ==> mfe
1 (0.08%) ==> netvibes.com
1 (0.08%) ==> Concealed by Juniper Networks DX
1 (0.08%) ==> Sparky
1 (0.08%) ==> Sun Java System Application Server Platform Edition
1 (0.08%) ==> Mittwald HTTPD
1 (0.08%) ==> Yaws/1.65 Yet Another Web Server
1 (0.08%) ==> igfe
1 (0.08%) ==> Phillips Data v1
1 (0.08%) ==> bsfe
1 (0.08%) ==> SimplyServer 1.0
1 (0.08%) ==> DMS/1.0.42
1 (0.08%) ==> Sun-ONE-Web-Server/6.1

You can compare it with Netcraft’s research result of all web server. I won’t claim that mine is a very accurate research (does have some rubbish data), but it showes the picture more or less. You can see that Mongrel gets a greater share here than the general list. At least those are Rails sites (does anyone use Ruby without Rails as a web platform?). I am happy to see Lighttpd (lighty) getting a spot amongst top 4. If any web server goes up in that list significantly in the near future, it will be lighty. The “unknowns” listed in 3rd place did not give any web server info in their http header. So let us assume that they have the same demographics as the visible ones.

And Apache is the leader by far, IIS being the second biggest web server in the market.

I would be happier if more info could be retrieved this way. I will definitely try to know more about the other web servers here I did not hear of before. I would also love to take part in any research project of similar sort in the future. Finally, If you have any suggestion/critique to make this research any better, or just would like to let me know your appreciation, please drop me a line.

Update: I just found that the primary list has some duplications. Too bad. I just updated the post with the new data. Changed the bad list files with new ones too.

technorati tags:, , , , , , , , , , , ,

Blogged with Flock



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: