ASP.net and IIS 2 million pageviews per hour

There are so few performance stats out there for ASP.net showing what it can scale to so I thought i’d share mine.

Plentyoffish is run off of one web server, at peak the site gets over 2 million pageviews per hour and has close to 50,000 active sessions. All traffic is compressed using GZIP, without GZIP i’d be well into the hundreds of megs/second for just ASPX files.

pages a secondsessions

66 Responses to “ASP.net and IIS 2 million pageviews per hour”

  1. Clarke Scott Says:

    Just wish the IIS team had made GZip a little easier to turn on :)

  2. Mayo Says:

    UUUUHHAAAA! (^_^)

    You mentioned that this stat is on only ONE machine??
    What is the current machine config if that is not the problem to ask?

    Sorry that i don’t like ASP.net (^_^), i will currently use RoR until i get into something more powerful, actually some of my friends are testing Lisp web server with capability of cca. 120,000 concurrent users per second on single 2.6Ghz AMD, 16GB RAM server! Grrrhhhh can’t wait till i see what production engine they have cooked! (^_^)

  3. Anatoly Lubarsky Says:

    What about CPU utilization ?
    CAn you provide some info regarding database server configuration, like hardware. I guess it is MSSQL 2005 ?

  4. What.se - Martin Says:

    It’s about you!
    http://what.se/article.asp?id=5060 (in Swedish)

  5. Markus Says:

    CPU is the red line on the chart there. I’m using 3 database servers now and its set up to be very redundent, don’t like stuff dieing on the db side of things.

    As for 120,000 concurrent connections I doubt that, TCP/ip only has 64,000 ports.

  6. Craig Says:

    Good to see that that there are proven results showing that MS products are just as powerful if not more.

    Any plans on adding another web server to further enhance performance?

  7. Mayo Says:

    Sorry, 120K parallel sessions, that is a burst, they have sustained cca. 90K parallel sessions on 24/7 testing.
    Don’t know if they use all 64K ports? (4_^)

  8. Mayo Says:

    This 2MM pageviews p.h. are all dynamically created or partially cashed??

  9. John P. Says:

    So that’s 500-600 dynamic requests/second which are gzip’d on the way out? And each of those requests is fully dynamic, there isn’t static/cached content mixed in with that number?

    Regardless it’s sort of meaningless without knowing what kind of hardware that one web server is.

    Nice of you to talk about your stats, lots of us are interested.

  10. Markus Says:

    Everything is dynamic
    Everything is Gzip’d on the way out
    There is nothing static.
    No caching functionality in asp.net is used.

  11. Alessandro Says:

    Plentyoffish is not run off of one web server as it has image hosting with AKAMAI. pics.plentyofsite.com currently distributes pictures, and this domain resolves to different IP addresses from different countries which means it is globally distributed and has several (tens?) servers around the globe. In large set ups like this, a web server is a simple program which simply manages GET and POST the requests and forwards *.ASPX to the application servers where it gets executed (and/or) compressed and returned back. Does not Plentyoffish have any? With this many *.ASPX requests it should have at least 10 powerful application servers or 50 PCs.

  12. BG Says:

    I understand that Markus may not be willing to share his application “engineering” here because this is a public blog and is also being read by his competitors. I don’t understand why Markus would not be using Caching although it’s provided by IIS and it doesn’t make sense to compile the application every time it is requested. Markus has given so many clues if you closely read all the comments and his interviews that these types of high traffic sites cannot run unless caching is turned on. I have designed open source applications that are high traffic but not as high traffic as his site, and I gotta tell you that it I run my apps fully cached and boy I do love that feature. When you GZIP your content, you also use a lot of processing power and with caching turned on, all the web server’s during is essentially “zipping” the output. So I would assume his processing machine is a power horse. He definitely has “many” processing cores on his web server. He’s also using Partitioning which is an MSSQL enterprise feature. That’s it. I won’t say more. I have said enough. Anybody who knows what I’m saying can immediately figure out what I’m talking about ;-)

  13. Markus Says:

    pics.plentyofsite.com is ONLY image requests. As stated above my server only does aspx files and a couple of images like smilies. If I served images off that machine there would be over 3,000 requests per second. Is there any major dating/social networking website that serves their own images?

    All ASPX request are served off a single machine, since every pageview is dynamic and tailored to ip/registration data its kind of pointless to use akamai.

    By not using caching I mean i’m not using the built in ASP.net cache. The main reason being as soon as I were to put data in the cache its already expired. As for the web server I posted stats before, Its a quad (my retired first db server) from several years ago.

  14. BG Says:

    @Mayo: I don’t think that 2.6GHz AMD machine is powerful enough to serve that many requests unless you’re serving pure static content on a portable web server such as thttpd or lighthttpd.

  15. Alessandro Says:

    This is obvious that pics.plentyofsite.com is for pics only. Plentyoffish consists at least of one web server for *.ASPX, 3 DB servers, and AKAMAI farm for static content, so it is not true to say that “Plentyoffish is run off of one web server”. But still, do not you have any application servers?

  16. Alessandro Says:

    A quad is a quad. It has 4 cores and may be as powerful as 2-3 regular servers. It is still not enough to process this amount of dynamic requests and have them gzip’ed on-the-fly. What is the trick?

  17. Alessandro Says:

    To BG: What you mean by “caching” is probably pre-parsed scripts. Thats is fine, they do not need to be parsed, but they still need to be executed at every request. When Marcus says “No caching functionality in asp.net is used” he probably means that content caching is not involved. However, it would be reasonable to at least cache profile data as it does not change as often as online search results. What Marcus is trying to say here is that supposedly he is using some techniques that allow him to run plentyoffish off of a single machine or at least times fewer than match.com or any other major dating/social networking website.

  18. BG Says:

    @Alessandro: Ask Markus in private. Not in public. Markus will not (heck I will not) give you an answer on the blog. The trick is a minimum of quad-quad core processors (that gives you 16, 32 or even more processors depending upon MoBo), 128 or 256 GB RAM, SAS HDD and it goes on. SAN for DB.

  19. BG Says:

    Markus, whats your choice and what do you use on POF: C# or VB.NET? I ditched Windows platform when .NET came out. So, I have used only JScript.

  20. BG Says:

    @Alessandro: Cached script means: when a call is made to the script, it has to be compiled and converted into object code. The cached copy is essentially using the object code when serving a request to avoid compiling the code every time when requested. However, the script DOES get executed based upon the parameters (zip code etc.) but it doesn’t get compiled every time its called. This takes away tremendous processing power from the CPU by using this technique. It just doesn’t make sense to compile the code thousands of times per second when serving the request. I really doubt Markus is compiling the code on every request.

  21. BG Says:

    @Alessandro: Would you be willing to share details of the infrastructure of your site in private? I see you run PHP…

  22. G Says:

    Markus how do you put up with so many arm-chair quarterbacks telling you why your architecture is bad, why it cant possibly work, and why they are so much smarter than you? I guess the million$ help to muffle their whining…

    I enjoy your technical blogs, please continue to write them. Your efficiency and productivity are amazing.

  23. Gomer Says:

    This comment is unrelated to this post.

    Markus, I know you don’t really need traffic from search engines as you are doing fine but…

    on your city pages such as:
    http://www.plentyoffish.com/personals/143onlinedating.htm

    The Title tag for the page reads:

    Free Online dating inchicago

    where you intended it to be:

    Free Online dating in chicago

    The space probably got lost somewhere.

    Guess who ranks first for:
    Free Online dating inchicago

    but does not have a top 10 for:
    Free Online dating in chicago

    I guess with that much traffic, a few thousand referrrers can get lost ;-)

    All the best.
    Gomer

  24. John P. Says:

    What are the hardware specs of this one web server so we can put this number in some sort of meaningful perspective?

  25. Johnny10g Says:

    Markus,

    Thanks so much for sharing this information – it’s very encouraging for us ASP.NET developers to see these kind of stats. Although some other high-profile sites (such as MySpace) use ASP.NET, their solution seems to be to have a warehouse crammed full of servers. It’s interesting to see the true potential of ASP.NET running on just a single server.

    It’s also interesting that your successful site goes against some of the conventional ‘wisdom’ offered up by many of the technical architects I’ve spoken to – they always look to use caching as much as possible, and generally scale out the servers. The funny thing is, these experts would be horrified by your architecture. Yet you’ve proved that it works, and it works very well.

    Anyway, thanks for being so open. It’s much appreciated.

    Cheers,

    Johnny10g

  26. Mayo Says:

    Markus,
    You are really a unique kind of guy that has shown us a light in many ways! I’m not ASP.net fan a lot, currently i’m transferring development server from Win XP to FreeBSD, but i think i will eventually end on Linux. As for Windows development you really have shown what M$ is capable of doing, and really showed also what Linux capabilities are, and that is: ENORMOUS!

    It shows that it’s not the environment that holds you back, it’s the mind itself!

  27. Zoltan Says:

    Hmm…. extremely interesting.
    Can anyone share a few things about AKAMAI image hosting? How does this work?

  28. Mayo Says:

    BTW Markus what is your current Akamai bill?

  29. Mayo Says:

    I don’t know what is the difference between Amazon S3 and Akamai, any experiences, i presume Amazon also has world-wide distributed servers also?

  30. joe Says:

    Markus what level of Gzip are you using and are you using the built in compression in IIS. If you are is it IIS 6 or IIS 5.

  31. Mike Says:

    lol, its hilarious how much censoring Markus does to these comments. Every time I come back here there are usually a few comments deleted.

    Especially comments that call out Markus himself for how he doesn’t have a clue when it comes to certain things, like TCP/IP and 64K connections being the max. (see his first post)

    I’m sure this one post won’t last long. ;)

  32. Markus Says:

    Mike I deleted your pointless post as you clearly don’t know what you are talking about and it will just mislead people.

    TCP/IP with IIS has 64,000 connections max per ip address. So in theory 65535 computers can connect to PORT 80 (the http port) on the webserver. If you are going to be a troll at least get some facts that are even remotely correct

    http://en.wikipedia.org/wiki/Transmission_Control_Protocol

    TCP uses the notion of port numbers to identify sending and receiving application end-points on a host, or Internet sockets. Each side of a TCP connection has an associated 16-bit unsigned port number (1-65535) reserved by the sending or receiving application. Arriving TCP data packets are identified as belonging to a specific TCP connection by its sockets, that is, the combination of source host address, source port, destination host address, and destination port. This means that a server computer can provide several clients with several services simultaneously, as long as a client takes care of initiating any simultaneous connections to one destination port from different source ports.

  33. Plentyoffish Performance Stats - Daptivate > by Kyle Beyer Says:

    [...] stats from Markus on his Plentyoffish [...]

  34. Alessandro Says:

    “TCP/IP with IIS has 64,000 connections max per ip address. So in theory 65535 computers can connect to PORT 80 (the http port) on the webserver”

    TCP/IP limit of 65535 means that in theory a client can make 65535 simultaneous connection to your server. Web server listens on port 80 and can handle as many connections as your system allows.

  35. Mayo Says:

    Did i misunderstand this:

    “Internet Information Services (IIS) has a listener to share a port between multiple HTTP applications…… This allows multiple HTTP applications to use the same port number without having to compete to reserve the port for receiving messages.”

    from:
    http://blogs.msdn.com/drnick/archive/2006/08/07/690331.aspx

  36. Mayo Says:

    Anyway Markus, can you share your experiences on Akamai??

    Also, again… any experiences with Amazon S3 for images, and how does it compete with Akamai(except for price of course (^_^))??

  37. Mike Says:

    Markus, read your own post, quote:

    Arriving TCP data packets are identified as belonging to a specific TCP connection by its sockets, that is, the combination of source host address, source port, destination host address, and destination port.

    Read closely: TCP connections are identified by SOCKETS. NOT PORTS. A socket is identified by the combination of source address, source port, destination address, destination address.

    How many possibilities is that? Lets see, 4 billion source IP addresses * 64000 source ports * 4 billion destination IP addresses * 64000 destination ports. Got a calculator handy?

    If IIS arbitrarily limits the total connections to 64,000 then thats what you get for using Microsoft products, but the TCP/IP protocol spec itself has no limit that could practically be reached.

    If your still confused, join #TCPIP on IRC and ask someone there, I’m sure they will be happy to laugh at you.

    So this post must be pointless because it proves you are wrong AGAIN, so I don’t expect it to stick around long either. I wouldn’t want to mislead people with the CORRECT information.

  38. Markus Says:

    I think this pretty much says it all, and is basically the same thing the IIS team told me which is what I posted in the original post.

    MaxUserPort (TCPIP.SYS)
    Controls the max port number that TCP can assign. Every unique client making a request to your web server will use up at least one of these ports on the server. Web applications on the server making outbound SQL or SMB connections also use up these ports on the server… so it highly affects the number of concurrent connections. For SMB tuning, read the de-facto IIS6 and UNC Whitepaper.

    HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
    Type: DWORD
    Value: Range from 5000 to 65536

  39. two million page views an hour « Virus In Training Says:

    [...] million page views an hour 13Feb07 Okay!, so few interesting stuff here!. So apperently plentyoffish.com runs on a microsoft server!, so mentioned in [...]

  40. Mayo Says:

    Yes it seems it is based on constraint of 65536 ports, but with flow identifiers they can surpass 2^16 constraint by many fold, i.e. each port doesn’t have to be unique for each user, they could serve 1MM concurrent users if hardware capabilities could enable it.

  41. Mayo Says:

    ^

    I am not talking about M$ IIS. (^_^)

  42. Alessandro Says:

    “Web applications on the server making outbound SQL or SMB connections also use up these ports on the server”

    Using dedicated network cards for outbound connections to database or application servers should help you in this case.

  43. Mayo Says:

    Here is something from for 256K concurrent connections:
    http://www.lewiz.com/le2020.html
    http://www.lewiz.com/talon3220.html

    This will get your hardware to new level, well it’s what they say (^_^)

  44. Alan Says:

    This is a very interesting discussion. the one this that occured to me (in my ignorance probably) is that no mention was made of a firewall in the description of the plentyoffish architecture. Are you using one, or do you figure that a properly locked down server just doesn’t need one?

  45. Mathieu CHATEAU Says:

    hello,
    We should have at least two web server to handle failures, reboot, patch management, maintenance, upgrade…
    Is it all this not just for one of your N servers?

  46. Fabulous Entertainment Blog » Blog Archive » mod_wsgi Problems Continued Says:

    [...] now seeing from Apache benchmark are now in line with the numbers that Plenty of Fish have published although I’m very aware that there’s a huge difference between performance under test [...]

  47. Vijay Says:

    We use one webserver with 2 GB of Ram and serving 1.4 Million page views

  48. Vijay Says:

    We use one webserver with 2 GB of Ram and serving 1.4 Million page views on http://www.autoindia.com

  49. Milo Says:

    Marcus;
    Are the files stored locally in the Web server or in a network share?

    Thanks.

  50. alalreltrchi Says:

    letopasch

  51. spenser Says:

    Hm, MaxUserPort.

    I had forgotten until reading the post upstairs that this is the maximum port number usable on outbound connections. Not inbound. I admit having used this before, but it makes me think a bit harder that the answer lies elsewhere.

    There *is* a KB article around somewhere stating that each w3wp.exe process is limited to 64k connections, as is each ip address. So the solution in the KB was to increase the number of app pools and the number of ip addresses.

  52. web guitar tabs Says:

    Interesting convo. I think I need to look at Akimai for image hosting. What does this cost?

  53. Dan Webster Says:

    This might be of interest to you:

    The Strangeloop AS1000 Application Scaling Appliance is the first application acceleration appliance that automatically optimizes dynamic web applications in real time without code or infrastructure changes. The Strangeloop AS1000 lets enterprises scale Microsoft ASP.NET applications, including those that leverage AJAX, to thousands of users while ensuring performance meets the expectations of their most demanding users.

    http://www.strangeloopnetworks.com/products/

  54. Chris Says:

    To clarify the TCP discussion, here are some scenarios:

    * 2 users hit Port 80, from separate IP addresses. The port can be 80 on both client and server here, because there’s already something to make them unique: their differing IP address.

    * 2 million users hit Port 80, all with separate IP addresses. Still no issue – you can do all of this over Port 80 as above.

    * 1 user hits the server 10 times concurrently, all over port 80. Now the server needs to respond from 10 different ports in order to serve these 10 requests.

    * 200 users behind a NAT Router with a single IP address hit the server. The server needs to use 200 ports to fulfill these requests.

    * Reality: 2 million users from about 1 million IP addresses hit the server. The server needs as many ports as the maximum number of concurrent requests from the most-trafficked NAT making requests – which might be, say, 100.

    TCP/IP is cool, and it’s held up for a long time – of course it can handle more than 64,000 concurrent requests. How would the Internet have lasted past the year 2000?

  55. Jas Says:

    “Plentyoffish is run off of one web server”

    This is misleading, regardless of whether or not *some parts of the website* are run off webserver. Plentyoffish is not run off of “one server”.

  56. Scott Says:

    This is a great convo for sure. I believe MS may be spreading some of the misleading info, and I am still hunting for the right answer. In all of the MS TCP tuning docs they say to increase MaxUserPort. But IIS only uses the 2 ports 80/443. As Mike says, it should be the sockets we are concerned about. If MS Win2K3 Server/IIS is limited to 64k connects simultaneously, I think it is pretty lame. TCPIP should allow millions, especially on a 64-bit machine. More appropriate would be MaxFreeTcbs (0-4.3 billion, a per connection parameter) Netstat -an only lists a few “ports” even when my server has 10′s of thousands of sockets active. So how do you tune Windows optimally to support millions of connections??? Anyone figure this out yet? Thanks!

  57. Mike Jones Says:

    Markus: What a great post, and down with your critics! We run off many servers and have seen traffic patterns like yours off a single server, especially with this type of distributed architecture. If your critics doubt, then tell them to field test their own solutions and yap about it on their own blog. I’d say this one is spoken for.

  58. MrPalooza Says:

    Great discussion. There is no doubt that the mainstream dating sites are scratching their heads at Markus’s success! Keep it up!

  59. Mr. William Says:

    http://plentymatch.com seems far more better than these sites…not sure though its people to judge for themselves

  60. Jess Says:

    very interesting. Yes PlentyMatch looks attractive too.

  61. What are the best ways to improve asp.net website performance? - Quora Says:

    [...] (see Lesson Learned)http://plentyoffish.wordpress.co…http://plentyoffish.wordpress.cohttp://www.codinghorror.com/blog…StackOverflow.com is also a big implementation of [...]

  62. Easy CSS debug Says:

    Functional…

    [...]ASP.net and IIS 2 million pageviews per hour « Plenty of fish blog[...]…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 255 other followers

%d bloggers like this: