The coming revolution that will change everything.

We are on the verge of what is and will probably be the greatest  technical revolution since before the creation of the internet.  Today nearly all major sites performances are limited by the speed of the hard drives used in servers.    Most dating sites and companies like Myspace.com  are using hundreds and in some cases thousands of servers in order to keep serving up pages.    Hard drives are brutally slow  several thousand times slower then attempting to access RAM.

This is all about to change,  in the last few years Solid State Drives  which are essentially RAM modules that retain information when power is turned off  have been falling dramatically.   Most of the projections out there say that the industry will go from 50 million in sales last year to  $5 -$10 billion in sales by 2010 pretty much completely replacing the current hard drive industry.

So what does all this mean?   A solid state hard drive is currently about 250 times faster at accessing data than standard hard drives.   Right now my database server peaks out when trying to serve 20 million pageviews a day and I need a second server to take over some of the load and all the issues are disk/IO bound.     If my database server was using solid state drives  I wouldn’t have any disk issues even if I served a billion pageviews a day. 

Now of course  CPU and bandwidth issues would prevent me from serving 1 billion pageviews a day off a single DB server.   But if I were to scale my infrastructure to handle the load of myspace using Solid State Drives I could probably do it by only adding  only 5-10 new database servers and  20-25 web servers.  This is a far cry from the thousands of servers myspace is currently using and I certainly wouldn’t need the 50+ tech staff that myspace has.

In summary,  Solid State Drives are going to dramatically change the way the internet works.  With these drives extremely low cost/ free services are possible on a truly massive scale.  More importantly  sites like facebook and myspace.com  are going to go from having tech budgets that run into the tens of millions a year  to low digit millions making them wildly profitable business.

24 Responses to “The coming revolution that will change everything.”

  1. soj Says:

    Well this seems to be great news, with these new solid state drives, would that also mean that pc’s will now boot up into windows a lot quicker, im so sick of waiting for it to load😛 yes i am very lazy

  2. Nick Says:

    I think a solution is allready here, but it will work mostly on *nix based servers.

    1) There are SCSI controllers that have RAM cache of more than 1GB, plus the OS can cache to the RAM

    2) Database servers like mysql can be “locked” in memory giving you the same result that you would have with a solid state drive.

    Of course this technology will help, and propably will make the smaller companies more capable to compete with the giants…

  3. greg Says:

    Pstt… who told you about google’s secret

  4. threz Says:

    Solid state is not going to catch up with magnetic any time soon, price/MByte at least. And if price is no object, then RAM based RAID is an order of magnitude faster than any flash drive.

    That is why big companies just throw more machines at the problem, its just cheaper and easier when things are designed to scale in clusters of PCs anyway.

  5. Rush Says:

    revolution? more like evolution.

  6. Markus Says:

    Its an evolution in hardware but a revolution in what is possible.

    Google stores stuff in ram, but the problem is you can’t store a database in ram. A single glitch and everything is gone. As for booting your computer it is instant as you just leave where you left off.

  7. ZF Says:

    Once again your analysis is perfectly correct. You would be surprised if you knew what was currently being developed by some small teams in stealth mode to take advantage of this.

  8. Dave Starr Says:

    I’d agree with Rush, this is hardly evolutionary. “In the Beginning”. long before the Internet was even a dream _all_ computer memory was solid state. Magnetic media evolved not because it was better (faster) but because it was cheaper … and engineers knew a lot about recording magnetically (analog on wire, later tape) than they did any other media.

    Pundits like Jerry Pournelle in tech journals like Byte (RIP) have predicted the death of “mechanical monster” disk drives for many years based on the principle that sand (silicon) is much cheaper than meticulously machined metal. That their predictions have not yet come to fruition is an anomaly of economics. not technology.

    I did have to laugh a lot at Nick’s “*Nix uber alles” view … operating systems do not make up for faulty mechanical structure even if they are “open source”.

    If you’ll excuse me I have to get back to my project of converting my 500 cubic inch Cadillac to a full Linux OS … it’ll get 400 mpg at 120 mph and jam all speed radars within 5 miles too, just as soon as I can get the darn kernel to compile 😉

  9. Ali Says:

    Faster hardware means you can scale easily, but it doesn’t solve the bigger and more important, getting to the stage where you need to scale.

    What this means is just that the costs of google and other big players will reduce. It doesn’t sound that revolutionary to me, unless you also magically get 100 million users with the faster RAM to serve it.

  10. threz Says:

    Store a copy of the DB in RAM, not the DB itself. Then most visitors use that copy instead of the disk copy.

    ie. DB writes should go straight to disk, but most reads come off the RAM based copy.

  11. threz Says:

    Regarding reliability of RAM based drives, the ones we use an onboard battery backup. So you have at least 8 hours to get the latest updates off if things really go tits up.

  12. Henrik Says:

    Vast savings possibly, but you still have to pay for bandwidth.

    What is does do is that it allows you to simplify storage logic. 2gb solid state change log and 4gb read cache on the application server, and it doesn’t matter much how fast the backend to that is.

  13. ZF Says:

    Ali and Henrik,

    Good points both, but you have to think about what kinds of text-based services are made possible by this technology, which could both attract millions of users and not require a lot of bandwidth at the distribution end. Where this technology leads is towards ‘good enough for the consumer’ solutions to problems currently addressed by some of the largest (and most legacy-constrained) IT applications on the planet.

  14. Nick Braak Says:

    Hmm… I see many of the comments reference RAM as the storage medium.

    Now this is NOT my area of expertise, but I believe the that “revolution” is in FLASH, which is, of course, writeable and non-volatile. Think the cards in digital cameras, cell phones etc., not SDRAM (which does need battery backup).

    I read and recommend http://storagemojo.com/ where stuff like this gets written about.

  15. Will Hardware Improvments Make Hardware Intensive Companies More Profitable? » Conversion Rater Says:

    […] I admittedly am no expert in computing hardware, enterprise server management, or solid state disk drives, but if Markus Frind is correct about coming advances in hardware it may make a lot of the newer web businesses profitable in the next few years, or at least move them from losing money into profitability. […]

  16. Nudecybot Says:

    Marcus I agree that solid state drives for primary storage and cache, will provide a huge leap in performance as soon as this is affordable. MRAM replacing RAM could even provide the ability to instantly boot/reboot a server.

    I/O performance as well as ability of systems to recover extremely quickly both stand to gain dramatically, so I would agree with the revolutionary nature of these technologies – operating systems will be adapted or new ones will arise to take advantage of the new physical contrainsts on performance.

    Now imagine combining these technologies with storage and OS virtualization and perhaps centralizing your solid state storage as a SAN (with ultra high speed connections to your MRAM and 64+ core CPUs). Data centers in 10 years will start to change shape noticeably.

    Vista is taking a more evolutionary step including a feature to take advantage of a flash RAM cache to aid in performance providing a tier between RAM and Hard Disk. See here:
    http://www.extremetech.com/article2/0,1697,2017818,00.asp

  17. Robin Harris Says:

    Flash-based SSD have great performance – all the benchmarks show that – but they don’t have the durability to handle frequent updates, such as database indices, for very long. That makes them very costly.

    There are technologies on the horizon that promise flash speed with disk durability, but they aren’t here yet. The multi-copy strategy – RAM and disk, or on multiple machines as in the Google File System – is likely to be the most efficient solution for some time. For more discussion check out http://storagemojo.com/?p=209 “Architecting the Internet Data Center” and other articles at StorageMojo.com.

  18. Gomer Says:

    Is anyone aware of who the leading players are for Solid State Drive? I am interested in finding out more about investment opportunities here.

    If Markus is right, going from $0 million to $5-10 billion in sales in four years… well if you bought some stock in those companies, you may do as well as plenty of fish. Okay, not that well but you get the idea.

  19. Gomer Says:

    Typo in my last post, meant going from $50 million to 5-10 billion in my last post.

  20. Robin Harris Says:

    SSD’s have been around for decades and done little. Flash drives, like the thumb drives we all have now, are the hot ticket.

    Samsung and Sandisk are big players. There is also a growing software market that provides unique software designed to be used on flash drives. Ceedo and U3 are two examples and there are probably others. I also expect Apple to ride this wave pretty hard, since they have a great flash supply chain.

  21. Gomer Says:

    Thanks Robin.

  22. Markus Says:

    I talked to some suppliers, Myspace.com and other big sites are already using SSD’s. Prices are $200 to $300/gig down from $600 at the start of the year.

  23. samuel Says:

    very nice article, i also think that flash drives will replace HD, more speed and low power consumption, a big evolution is coming🙂

  24. Reppe Says:

    Thanks for share. This is the best site i’ve read today. I love reading the informative articles about summer sale discount deals and couponss..And your is really helpful and easy to understand.. I will also try to do.

    most up-to-date Coupons

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: