Hiring my first dba.

I was reading this article today about match  talk about how they sent 1.2 billion emails in the last 6 years.    It reminded me how far Plentyoffish has come,  and how I’ve been doing all the database work except billing since the beginning.    In a few weeks i’m going to give up much of that responsible and hand it off.

I can’t believe the scale of what’s been done.  Its weird because day by day the growth isn’t that much but when you look back over the years its huge.   Plentyoffish users  send  5.5 Billion Messages a year..   That is 27 times what match.com users send.    The hard part about scaling is making it cost effective.   Match doesn’t have a problem as they earn 400M a year which works out to $2.00 an email sent on their system per year.   For match buying new hardware is nothing.    For plentyoffish we have 27 times the messages even paying a penny per email is not viable.     But what truly takes computing power is our Behavioral Matching Engine, talked about it a bit in 2007 on the today show  years before any other dating site even knew that kind of stuff was possible.   I feed it with nearly 20 billion pieces of data that are stored in the database,  and all of this is done in real time.   That is what requires hardware and lots of it.

At the end of the day the site is so large,  I need to give up the day to day running of the SQL databases.   Not only do I not have as much time to pay attention to it but I also don’t know every aspect of SQL Server anymore.  Its funny one day you knew everything there was to know about SQL server because you had to work with it daily but than a year goes by,  or 2 a few upgrades and suddenly you only kind of know whats going on.

29 Responses to “Hiring my first dba.”

  1. Lurker Says:

    It would be a mistake. Do not hire any DBAs, it is not DB admin that is really required, but SQL tuning for queries for improving response time and cpu spikes and possibly eliminating storage level IO bottlenecks. Which I think you are already good at. I am sure you are not adding new feature/code that introduces new SQL all the time. The topmost resource consuming queries would already be tuned by now.

    I have followed you Markus for sometime and in fact even been inspired and released my own dating sites such as NOTforPlayers.com and LesbianCougarDating.com. I am a 10yr + Oracle DBA / Database Architect myself & I know you run on SQL Server, but I know the general responsibilities of a DBA.

    Do not get overwhelmed by the new features that the vendors introduce, it’s usually to want you to upgrade (although yes at one point the vendors stop supporting you and you are forced to upgrade to be able to get patches for bugs etc.). Do u really need a dba to upgrade once in 3 yrs ?

    Unless of course you are planning to go totally hands off. That’s different, but in your write-up above I did not see a strong case for the need you mentioned. By the way if volume of data is an issue look at partitioning those tables by a date column so you can just drop by partitions, since continuous deletes by older dates can sometimes degrade performance on the table unless you rebuild indexes periodically – so probably an archiving strategy, that’s all. Still does not justify a full time DBA. just figure out one time how you plan to do it, and once the solution is designed it can be repeated on a regular basis or on schedule. Volume of data is not necessarily an indicator of more work. For some reason I think you would already know what I am saying especially when you run a tight shop already, and maybe this piece is written partly for marketing reasons.

    In any case it is pretentious of anyone to advise you who has been so successful and has been hands-on with every aspect and thinking of running your shop, and I usually never think of commenting for that very reason, but this one I could not resist.

  2. Lurker Says:

    BTW I did not click on the article for match.com before I commented above. I just read it now. What they are doing with Synapse / matching algorithm / neural pathways / self learning behavior – or whatever crap they call it – is all about coding and intelligent captures of data points during user’s browsing and via interactive daily match feedback. Which as you said you already do to a certain extent.

    A DBA will not be able to do any of that for you, It needs a developer. In short, it is not that rocket science as people make it sound. It needs a grand scheme of concept of what data points you want to match by, and how you are going to calibrate/mesaure – once that concept is formed – it can be done many different ways really – there is no real one right way. After that, it boils down to implementing via code and an additional table or two to store intelligent data for each user in a separate table.

    As people browse and search through data maintain, an array of session variables, that learn (or update it’s value) based on a tipping point score – such as if the person browsing someone with age over 40 + , then each click makes an increment in the session variable value, and once its a reaches a tipping point, it could be considered a strong enough indication to store the preference trait as a data point in the table – which will then be used later to present date suggestions or matchmaking. Also each time the person browses or searches again in a later session – you validate against the same preference data points, if they keep holding or they change and update accordingly – so that occasional deviation from their search behavior is not stored permanently in a static way.

    Anyway capturing browsing data to use intelligently for marketing purposes is nothing new and nothing novel about it. But using it is definitely wise, and having a news article about it, is even smarter. If I had that volume of traffic and number of members, I would slice and dice the matchmaking to be more granular also, – but for me – I do not even maintain a separate table for intelligent data based on browsing behavior- only the primary profile – stated preferences.

    Actually my biggest challenge is scammers, and I use intelligent browsing behavior, ip address, and then a process of manual checks to find so many people that are scammers, I only allow about 2 or 3 out of 10 people in my site.

    Again behavioral pattern based matching is not a DBAs forte , unless he/she is a Developer/DBA – its mainly a programmer’s area of strength and an algorithm / scheme to go along with and using a good data structure to make the performance as efficient as possible since each page view will entail additional work.

  3. Markus Says:

    I can’t babysit the servers 24/7, so if something crashes I need someone with the skills to actually bring stuff back online. Thats the main purpose.

    As for scammers half my code is to stop them and automated bots.

  4. Luca Pescatore Says:

    Try with NoSQL

  5. Eric Says:

    Was it just me or did the figures in the article just seem way off?

  6. Sideah Shannon Says:

    HI Markus. I wasn’t sure how to get a hold of anyone at pof.. So I decided to comment here.

    Last night I was suddenly wondering why plentyoffish does not have a Live Support Chat? I have seen them coming up on websites and they are immensely helpful.

    Reasons why they would be helpful to POF..

    -You are able to see exactly what clients are having trouble with
    -You are able to receive direct feedback
    -For a very small amount of money you are able to receive valuable information on why users do not stay
    -You are able to better understand how to make the website function
    -You are able to help users who may not be good with computers find help
    – It is a great idea even to be done as a trial to gather information honestly from users.. because most people actually dislike doing ‘surveys’.. but what better way than to casually drop some questions while they are seeking help?

    As one of the largest dating sites.. you would be the first to implement it…

    So let me know if you are looking to hiring a young & eager service rep.. I live in the Vancouver are and I’m looking for some meaningful employment!

    -Sideah
    sideah.a@gmail.com

    (604) 440 -8755..

    Oh and by the way.. very intriguing blog.

  7. w3bmoon Says:

    sounds good

  8. Dating Industry Links 8-5-11 Says:

    [...] at Plentyoffish says Plentyoffish users send 5.5 Billion Messages a year, 27 times what Match users send. My only question is how many emails are responded to? Sent emails [...]

  9. Global dating, Love and Sex Articles by you! » Dating Industry Links 8-5-11 Says:

    [...] at Plentyoffish says Plentyoffish users send 5.5 Billion Messages a year, 27 times what Match users send. My only question is how many emails are responded to? Sent emails [...]

  10. Harry Says:

    Did you count the spam messages also?

  11. crazycanuck Says:

    Here is an idea instead of storing the messages on the server why not automatically forward the emails to the user’s actual real email. This way they can get to keep the emails without copying and pasting.

  12. crazy Says:

    Hmmm what about NOSQL? http://blog.zawodny.com/2011/07/23/nosql-is-what/

  13. Craig Whiteman Says:

    Please Markus, your upload photos doesn’t work in firefox (c) this for (u) 2. IplaythisuneffectednowLordJesus there’s nothing like your smile made of sun, you’re the 1one i need coolEwa coolEwa coolEwa indiGENous baby your hounour, like (n) I will I can (n) I do teach you how to victory one day m8 brewin blue and purple DNA style testmatching PlentyOfFish.com MyHertitage.com Greyhound-data.com HotOrNot.com TAB Marry Night Style NCV 4671

  14. Andrew Says:

    NoSQL? It’s a joke? Really exist?

  15. Makais Says:

    seems like a smart move

  16. Mike Won Says:

    Andrew it does exist. http://nosql-database.org/

  17. Lindsay Says:

    Hi Markus, I just wanted to let you know that I found my best friend and soul-mate on Plenty of Fish in September of 2009. In March of this year he proposed! I never actually thought that I’d find the one I was going to marry on your site. I’m 28 years old and couldn’t be happier with the way things are going in my life now. How many people have actually gotten married after meeting on your site? You should do commercials about dating successes and have us in one (you can come to the wedding if you want!) haha Thanks again. (I know my comment has nothing to do with database work but I didn’t know where else to post this)

  18. Catering Says:

    I think, he counted the SPAM messages also, any comment?

  19. enniewidhia Says:

    Dear my friend, how are you? I like your site but i don’t know about this topic, but i want to leave good comment to this site. Good work my friend!
    Best Regard : power of mind

  20. Jenny Garber Says:

    Hello, POF rocks

  21. Jeremiah Actos Lawyer Says:

    Suggestion, maybe allow POF users to search by professions like law, medicine, etc— Not your average personal injury lawyer looking for love ;)

  22. brad Says:

    I have an idea! Why don’t you go to a computer, then try to sign up. Dump your cookies and cache first to make sure it’s clean then you’ll be able to experience how fucked up your site has become. You can fill out all the tests upload pictures and then it will log you out and never recognize you again. over and over. That is what will destroy plenty of fish, not the competition. I’m actually going to move on to the competition tonight because it is impossible for me to use POF. It doesn’t work. It doesn’t even recognize email addresses when you do a send me my password thing even though you entered the correct password – it’s fucked up.

    If you want to save server power you could (obviously) do all of the Behavioral Matching processing using encrypted java. You can streamline your front end with a simple series of high latency apache allocation modules and if data collisions are what causing it to be so intensely fucked up on the back end then that could be solved with memory and cue and for the love of God why on Earth are you using SQL Server when your system clearly needs a more robust solution.

    If I were you I would sell it right now because unless you find someone like me who knows how the hell to fix your site it will go down in flames faster then shit comes out of a gay gooses ass after 30 White Castles.

    Sincerely,
    Brad

    account = a35e3af5e1as5e5e33a5ef1e3

    pass = 1478963

    email = gigalocool@gmail.com

  23. Richard Says:

    Hi Markus, I’d like to apply for the job.

  24. nfljerseyzj (@nfljerseyzj) Says:

    Once in the cowboy
    ) image of popular American movie star, is also a country music singer and television broadcasting group boss legend Kim Autrey ( Gene Autry ) in 1960 also participated in the St. Louis held major league expanded conference. Autrey is planning to mark Losangeles expansion team broadcast rights. The original League Hall of Famer Hank Greene Berg ( Hank Greenberg ) and Bill Vick (Dez Bryant Jersey ) partner seems most likely to be the new team boss, but Marley don’t want and VIC in the Losangeles market competition, is also afraid of seeing yourself in the southern California area oligopoly situation was broken, so he refused Greene Berg and Vic combination. Two other groups of competitors are from Chicago’s insurance industry, and later the Oakland A’s owner Charlie Finlay ( Charlie Finley ), but they also have failed. The last American Union lobbying Autrey joined this bid, in fact the original Austrian Cui Shou got the part belongs to the Pacific Coast League stars ( Hollywood Stars ) Hollywood shares, the team used to be the old Losangeles angelic adversaries. Autrey finally agreed to the American League ‘s invitation, bought the new team.

  25. Joe Says:

    Dude, your html coding blows on pof. Instead of a dba, why not hire someone who can actually write good quality html and css?

  26. Lsati Says:

    Dear admin, I’ve seen much of this kind of website before,
    but this one is really very nice and helpful, you’ve done a great job.
    Keep them and successful for you. sign of peace : human anatomy-physiology

  27. Sadaf Farooqi Says:

    Nice blog very much informative i enjoy my visit here hey watch that video i hope you really enjoy it http://www.youtube.com/watch?v=CboHs-y0nkc

  28. Mykal Anthony Mitchell Says:

    I have a suggestion for the site that would make it fairly easier for people to find what they’re looking for and weed out what they do not want. You should make a deal breaker section. Where the male/female user selects or puts thing in the field that would be a deal breaker for them such as race, religion, etc., that way someone won’t have to go through the process of messaging that person only to find out that the person doesn’t say…date outside of their own race for example. Some users make that known but some do not. If there were an area specifically designed for that I think that people would use it and it would make the site a lot better.

  29. Ray Rice Jersey Says:

    I savour, result in I found just what I was looking for.
    You have ended my 4 day lengthy hunt! God Bless
    you man. Have a great day. Bye

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 260 other followers

%d bloggers like this: