8 Way Dual Core opterons with 128GB/ram.

I’ve been spending hundreds of thousands the last few months attempting to build out my infrastructure and build a system capable of handling 5-10 billion pageviews a month.   The whole exponential growth in hardware is going to hit me fast and hard, i’ve delayed it by a factor of 50 compared to everyone else,  but  at this size i’m going to get trouble soon.

At any rate I tried to buy 2 8 way dual core opterons.  suppliers have gone through about 6 test systems now,  failed MB’s,  Raid issues and blown up hard drives.  I’ve tried a series of different suppliers.  Just a string of super bad luck.   

On top of it all,  all these suppliers think they are selling 8 way dual core opterons  but in fact its just a fancy 4 way because of Errors in windows 2003 x64.  http://support.microsoft.com/kb/899656    You see 128 gigs of ram and 16 cpus in the computer manager but only 8 cpu’s in task manager.     I am probably going to drop the dual core and just go with 8 CPU’s and 128 gigs of ram and hope that fixes it.

I am going to connect these machines to a huge fibre channel SAN and run my core db off that.   Anyone reading this have any experience getting a functioning machine with this much ram to work in windows x64?

42 Responses to “8 Way Dual Core opterons with 128GB/ram.”

  1. booster Says:

    10 bln/month = 4000 requests per second

    As a matter of fact, I have a single computer handling this many hits (on the intranet).

    In the world of windows, your options are a bit limited (networking sucks) but there are ways – wanna chat about it?

    You know about memcached, right?

  2. Zoltan Says:

    booster, on what kind of machine?

  3. Zoltan Says:

    And something else. Anyone who built an application using a large database knows that 1 pageview != 1 request. Usually 1 pv = at least 3-4 requests.

  4. Mayo Says:

    Markus, look what Mono project can offer you in foreseeable future, they had started to implement ASP.NET 2.0 in it but its seems its poor for now, you could contribute to it or help financing the project (fortunately i will be using something else😉 ).

    Windows is the last thing you need, even Vista, i posted the link before, but here it is again:
    http://moishelettvin.blogspot.com/2006/11/windows-shutdown-crapfest.html

    You are on the (b)leeding Edge here, mine will be too (if God has Mercy🙂 ), and for that only Linux is the answer!!! Because it is used by freaks like us – it is tested to the edge and no MS option can help you there….

  5. Mayo Says:

    Oh BTW how much did it cost you ?? Around 50$K right?😉

  6. Mayo Says:

    Oh and imagine those Opteron’s Quad Core ….. droll ;->

  7. Mayo Says:

    Just been watching Meetic (http://www.datingpro.com/blog/index.php?title=great_example_of_auditory_exchange&more=1&c=1&tb=1&pb=1) vs. POF, 137 employees at Meetic vs. 1.5 (are you really still alone at this ?????? 8-] ) at POF ;->

    look at the 5 year span at Alexa (i know alexa is not best metric, but as the site is bigger the numbers are more exact… at least i heard so😉 )

  8. booster Says:

    Zoltan,

    On dual core 8GB machine.
    If is funny you noticed the word ‘request’ and ignored the word ‘intranet’.

    To make it more fun for you – 4.000 is a low number. The same machine is capable of serving order of magnitude more.

    It is third enterprise datacenter I’ve built in last couple years – you would have hard time insulting me.

    Billions of requests only sound scary. In reality it is “no single point of failure” that really gets you.

    Have fun, kids.

  9. bob Says:

    How about an xserve cluster with Xsan.

  10. Mayo Says:

    Yeah booster H.A. is the real problem here🙂

  11. Andy Arnott Says:

    Markus,

    I saw a problem with MS technology early on, that’s why I went back to the drawing board and rebuilt the site from scratch using Ruby on Rails, mysql, and Linux.

    With the new grid systems available for Ruby on Rails it is by far the best solution. Your site would be easy to clone to ROR and I would be more then happy to refer you to some great developers.

    Development time is greatly reduced and you can roll new features in a matter of minutes, rather then days or hours.

    Ruby is heavy on cpu, mem usage, but the savings in Dev time and ability to roll out new features makes it all worth it. GOOD LUCK!

  12. Mike Says:

    Hey Andy, no offence but I just created a profile on Collaboradate.com and it was one of the worst experiences I’ve had at ANY dating site. Hopefully people looking at using Ruby don’t take a look at your site and get scared off because it can be a great language.

    BTW: I’m getting failed redirects all over your site too, that doesn’t help.

    One of these days someone will make a GOOD free dating site, heck a GOOD dating site in general. POF is probably the best so far, but that really isn’t saying much because I think it does a pretty poor job.

    I don’t want to come across as being negative, I could spend all day posting constructive critism on here, but this isn’t the proper forum. Feel free to contact me directly if you’re interested.

  13. Mike Says:

    Markus, are you sure you want go down that road of scaling vertically? You haven’t even taken a few steps and you’re already running into road blocks. It makes sense to a certain point, but if your planning on growing POF to the 10 billion page view range, eventually its going to bite you in the ass, hard.

    What happens when one of your fancy new 8 or 16 way boxes fails? Unless you have a spare $50K box sitting there ready to go, or one that is sharing half the load so if one fails the other can pick up all the slack, you’re screwed.

    Either way it will come back to haunt you because the performance gain of adding CPUs is far less than adding additional servers. In my opinion you would be wise to start thinking of horizontal scaling now, while its easy.

    Websites are prime targets for horizontal scaling, take a page out of Google’s books. There is no doubt one of the major reasons for their success is their ability to scale horizontally, they have the hardware capability to add more features and faster than anyone else.

    While you’re sitting there trying to figure out how to get Windows to run on 16way server with 128gb of ram, someone is out there adding cheap dual CPU servers to their farm every couple months to handle the increased load. If one or two fails no one knows the difference. When they add a new feature that requires more processing power they just add a couple more cheap servers again. Mean while you’re sitting there trying to figure out who you can buy a 32way server from and why you only get a 50% performance gain over a 16way.

  14. G Says:

    I worked on a $800M/yr website (all Microsoft technologies) which successfully used Akamai’s geo-localized caches for serving up static content and images rather than a single scaled-up webserver. If the 128GB is intended to cache web content, Akamai probably would do a better job (for a price, of course).

    I’ve also worked on 8-way SqlServer boxes, but with lots of issues and $$$. If possible, wouldn’t a solution federated across multiple databases (customers or keywords named A*-C* in db1, D*-F* in db2, etc.) be cheaper to build and maintain than a single giant database? 128G seems like a lot of memory for a database – my first-hand experience is that generally big databases are bound by I/O rather than memory.

  15. Mayo Says:

    Well that is the problem, Markus wants to have least boxen possible as i understand — i.e. less boxen less racks — less racks — less tech support

    Maybe this will help:

    http://www.itweek.co.uk/itweek/news/2152967/stratus-outlines-fault-tolerant

    … or some tech similar to this

  16. Andy Arnott Says:

    Mike,

    You just proved my point. The current site is not based on Ruby on Rails it’s based on ASP!?!?! Didn’t you notice that??? The new site is still in development. We are looking to launch in a few weeks, but thanks for the interest.

  17. Mike Says:

    Mayo, I guess it depends if your goal is to minimize server administration or maximize your company growth, profits and uptime. For the price of ONE $50K server you could pay a decent server administrator for a year that could manage many boxes. If Markus’s goal is to stay a one person operation, that is great and best of luck to him, but he won’t stay on top forever like that. There is no doubt one or two more smart people could help launch POF even further into orbit.

    The problem of course is Windows more than anything with regards to server administration. Managing hundreds, if not thousands of Linux boxes is trivial, I used to do it all the time when I worked for a web hosting company, one that Markus is infact familiar with. Based on ratio we had several times more employees looking after windows servers than linux servers. Each Linux server on average served 10-15x more customers than each Windows server as well.

    POF wouldn’t need a ton of servers though, if he used servers with 4cores each, only 10 would get you 40 cores in total and the ability to push out many more pages/sec than two 16way boxes could dream of. For less hardware cost too.

    As for using Akamai as “G” suggested, I never understood that unless you were pushing large files (gigabytes) and wanted to get the data closer to your customers because latency is an issue. Static content is a piece of cake, for most web sites, especially dating ones, a single server
    running Linux with Tux (in kernel web server) can push tens of thousands requests per second, take a look here:

    http://www.spec.org/osg/web99/results/res2003q2/web99-20030421-00224.html

    For static content you’re not going to beat that kind of setup.

  18. Markus Says:

    These servers are to manage the core database… Basically the only thing that matters here is RAM.

    My database will soon pass 100GB and no doubt 500GB by the end of next year… I need to keep as much of it in ram as possible and also spread it out over other machines.

  19. Mike Says:

    Markus, if you’re willing, can you mention what your DB stats are, things like average queries/sec, what portion of those are writes, and what portion are selects?

  20. Robert Yeager Says:

    I am curious to understand your decision-making behind scaling vertically at this point, based on where you are today.

    I agree that going vertical in the beginning helps to optimize the software to squeeze out the performance gains. But once that is accomplished, the typical way to scale is horizontally, gaining the benefit of high availability.

    You achieved the first step with regards to software optimizations. Why continue to go vertical with even bigger hardware? You are at the point where I would think going horizontal is the right move, but that’s from the outside looking in.

    Robert Yeager
    Founder, Cooqy

  21. Mayo Says:

    Markus, if you cant RAM cluster with SQL Server then transfer to MySQL cluster… i know it’s a pain in the a## but that would be only viable option to consider ….

    I think you will need for HA in MySQL cluster three machines of same amount of RAM, that means for DB of size of 500GB you will need 1,5TB or RAM😉

    That means you will need roughly 15!! 16 core boxes🙂 == roughly 750$K to 1MM$ for boxes only… but it will pay you off… and pretty handsomely i think!!!

    All the best in your business Markus!!! Live long and prosper!!!!

    Oh hero of all us lonely wannabe coders!!!😉

  22. Mayo Says:

    Oh and for static content Zeus web server can do miracles, i think someone said over 100K concurrent connections….😉 (also over 100K$ software)

  23. Markus Says:

    “I am curious to understand your decision-making behind scaling vertically at this point, based on where you are today.”
    These 2 servers I wanted, which can’t seem to be built were to have connected to the core DB on a san. They would be in a Active/Passive failover cluster. If I need to grow further then slave db’s would recieve data from the core DB. This is the standard way large websites are built.
    For a Dating Site all the complexity is all in the database and there isn’t a single expert in the Database field that won’t tell you to scale up and then scale out. If you try it any other way you are screwed for a whole host of reasons.
    The other dating sites close to the size of Plentyoffish have over $12 million a year in tech costs. Myself and others have the same problem as Ebay which has 18,000 servers. We can’t break up the database because we have to do ranged searches across ALL profiles/data. At any rate i’ve got only a handful of servers compared to the 600+ at match.com and yahoo personals and the same level of traffic, I must be doing something right.

    ******************************

    Match.com’s initial production environment utilized a standard external HP disk storage configuration. Merznoted, “The storage provided excellent price/performance. However, to provide ad hoc querying capability for business analysis purposes, we deployedthe HP StorageWorks Modular SAN Array 1000(MSA1000) technology in our back-office financial reporting environment. It was connected to an HP ProLiantDL760 8-way server and we witnessed its incrediblecapabilities in terms of performance and sub-systemavailability, which was an excellent lead-in, toredesigning the core database server infrastructure.Match.com’s efficient application data storage methodologymeans that its current production database contains a surprisingly low 200 gigabytes of raw data. “As westrategized about our production environment storage, we looked at the price, performance, configurability,scalability and availability of different storage solutions.Other major storage vendors’ solutions were considered,but we weren’t looking to spend $1 million to enhancethe core database. The MSA1000 technology fit our current requirements, budget and allowed for scalabilityin the coming months and year,” reflected Merz.Merz implemented the HP StorageWorks MSA1000 withtwo additional disk shelves and 42 15,000 rpm drives,each with 72 gigabytes of storage. He noted, “TheMSA1000 provided a quantum leap in performance.It delivers high availability and satisfies our needs interms of performance, reliability and scalability

  24. Zoltan Says:

    Booster, it was not my interntion to insult you. I apologize if I did.

  25. Mike Says:

    Markus, you definitely want to scale up to a certain point, but 16way boxes in my opinion is probably passed that point, especially on Windows. Lets be honest, Windows isn’t exactly known for its vertical scaling ability.

    How much of your data ACTUALLY needs to be searched in ranges? The profiles themselves must be a small part the data especially since the average profile has about 100 words in it and isn’t updated that often. You’re not doing ranged searches over “About Me” data though, you’re doing range searches over sex, age and location primarily, correct? Dating sites being as simple as they are right now are definitely on the easier side to scale.

    As “G” mentioned, a server partitioned setup if your database supports it would probably work not too bad, assuming you split up the profiles/data evenly using a hash type algorithm, or by location for a greater cache hit rate. I’m not sure about MSSQL or MySQL, but I know with PostgreSQL you can run queries on different servers like this:

    select ( dblink_query(‘select * from profiles where distance

  26. Mayo Says:

    Markus, how many page views have you today and how many servers??🙂

    ******************
    From Match-HP doc:

    “The architecture allowed us to dynamically allocate a
    significant number of transactions away from the core
    database server allowing for horizontal scaling of the
    architecture. As we expanded, we added servers to the
    web tier and this resulted in over 120 HP ProLiant DL380
    servers supporting 30 million page views per day,”

  27. Mayo Says:

    I think this is important:

    ProLiants delivers tremendous
    throughput improvements, allowing the company to
    “save money by not having Data Return(their colo service) manage and support as many servers.”

  28. Mike Says:

    Hrmm, apparently wordpress didn’t like my earlier comment and cut most of it off.😦

  29. kwa Says:

    You can find interesting ideas here:

    http://www.mysql.com/customers/

    Some companies have detailed the way they scalled their online services. It’s worth reading.

    Markus, if your database grows quicker than CPUs and RAM, you _will_ have to grow horizontally. It’s just a matter of time before you won’t have any other option.

  30. Mayo Says:

    Hey, what happened to: Criminals find way to disable internet, did they disable the post too???🙂

  31. Norman Says:

    Markus: next big step might be to help reduce a lot of disk I/O between DB’s and SAN : http://www.superssd.com/products_sub.htm

    I think it’s going to be painful to watch you start using SANs and attempt to scale Windows — especially on 64 bit — but my Linux-coated prayers are with you.

    I guess this is the end of “Do you know how cheaply Markus @ POF scaled?”

    I still have Brad @ LiveJournal, who has did an amazing job scaling LJ on the cheap.

  32. G Says:

    No direct experience, but IBM claims DB2 has 99% linear scalability from 4 to 64 CPUs:

    http://www-306.ibm.com/software/data/db2/benchmarks/111704.html

  33. Louis Says:

    Hi Markus,

    Two things for you to look into, if you like:

    First, have you thought of running your app (i.e. log “playback”, etc.) on a platform where you could use Dtrace to profile it in depth? Something like Solaris 10? I’m thinking that way you could have VERY fine grained real load/performance/hardware subsystem utilization stats to find and design/deploy around whatever bottlenecks you are encountering.

    I’m not sure what all platforms dtrace has been ported to. But am thinking with the fine grained profiling capability — even on a live production server (with no impact on production performance) — that dtrace might be something that would give you the specifics you need to diagnose current issues — and monitor things on an ongoing basis, too.

    Here’s a link you might want to check out. You can scroll down to the Oct 27th entry where he talks about profiling a database with Dtrace:

    http://www.lethargy.org/~jesus/categories/3-Damaged-Bits

    With the above discussion on other operating systems, here’s another link you (and others reading your blog) might find of interest:

    http://www.lethargy.org/~jesus/archives/44-PANIC!-Linux,-FreeBSD-and-Solaris.html

    Second, have you ever looked into a non PC platform? maybe something midrange; something made to a greater MTBF spec than what PC hardware is made to? There might be an option here for you if you want to keep the number of boxes low. Don’t know how tied to a Wintel platform your app is, of course, but wanted to mention “bigger iron” platform options.

    I hope something above is helpful food for thought.

    Best wishes to you and yours in all regards,

    Louis

  34. fez Says:

    Markus,

    I was going to make the same comment as echoed by others, but your subsequent comment on the DB being the “scale up” reason makes sense.

    That’s the same limitation with MySQL, no? (for those in the know) That is, when you cannot partition out the data.

    I’m also curious about people’s experiences with Amazon S3 for serving static content. Any thoughts?

    It just makes so much sense from the perspective of headacheless scalability of static content.

    I am going back and forth with my admin about whether or not we should serve / scale up serving static content using Amazon S3 or a simple nginx / lighty box.

    ps. feel free hit me up at fez.ummyeah @ gmail if you have any thoughts or want to pitch your services. (just did 12k uniques, 17k pageviews in a day after launching a few months ago and doing zero promotion, so we’ll need some help.)

  35. Dankind Says:

    Have you checked out the slides about ebay’s and livejournal’s expansions?

    “Myself and others have the same problem as Ebay which has 18,000 servers. We can’t break up the database because we have to do ranged searches across ALL profiles/data.”
    According to these, they have broken up their databases and scale them horizontally
    http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf

    I’m gonna have to agree with mike in saying that dating site data should be easy to spread out across databases. Read about how they partitioned data by different scaling / usage characterists. They also do their searches by multicast from primary database to search nodes.

    The livejournal slides are interesting too:
    http://www.danga.com/words/2004_mysqlcon/mysql-slides.pdf

  36. Markus Says:

    There are many ways to skin a cat, it depends on what are you trying to optimize.

    Of course there comes a time when you can only scale horizontally, but for Ebay and dating sites that is a brutal problem and creates a lot of complexity. Ebay according to that slide ditched their DB system and pushed all the stored procedures and joins etc into the application level. Designing a database from nothing with a couple of hundred employees over a few years is not a option that most companies have. For large sites scaling on low cost servers means you need to hire a ton of people to take care of it and keep it running and the added complexity in the application level causes all kinds of problems.

  37. elhoim Says:

    I just found something that might interest you, sandisk will start (has started?) offering a 32GB SSD disks doing ~7000IOps for $600.
    Some references:
    http://www.linuxdevices.com/news/NS7676844023.html
    http://www.msnbc.msn.com/id/16545386/
    http://www.digitimes.com/news/a20070109VL203.html

  38. elhoim Says:

    Forget my last comment, it seems it was just (once again?) vaporware….

  39. joe Says:

    We dare to be diffrent, No more hassle rebillings,
    Bemygal.com delivers service. The way it suppose to be!
    users that never log in are weeded faster than your backyard weedwacker! Meet people,post,chat,upload as you like.
    What are you waiting for?
    check it out.

  40. Jordan Wood Says:

    Booster gets Mayo (ed) why won’t the old farts just give in and submit to the new studs?

    http://www.facebook.com/group.php?gid=2384202648

  41. Christy Says:

    lxbuid Kudos to you! I hadn’t thought of that!

  42. Major food manufacturers Says:

    It’s actually a great and helpful piece of information. I am happy that you just shared this helpful info with us. Please stay us up to date like this. Thanks for sharing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: