Facebook trapped in MySQL ‘fate worse than death’

So… let me get this right? Facebook is in “MySQL Hell”.

-They have one of, it not the most visited sites on the net.
-Their site have next to no lag.
-And almost every page on… the social network… requires how many db calls?

I’m trying to figure out what people actually expected to happen here because it’s kind of confusing. Google runs search queries… not impressive. Amazon runs prices and publishers… okay. Facebook runs status updates, games, likes, groups, fan pages, applications, and hosts their on API to manage anyone’s account and pages remotely.

4,000 Shards…. even if it was a baseless accusation the OP can’t back up… suddenly, doesn’t feel that impressive or bothersome.

“If /dev/null > is fast and web scale, I will use it. Is it webscale? MongoDB is webscale. ” -from http://bit.ly/qMZnFl
This sums up 80% of what I read here between the people who have little or no idea of the technology barriers demolished by Stonebraker and the disagreements about which company is using which technology. Most if not all of these top 0.001% companies are using multiple technologies depending on the problem space they occupy. NoSQL is fine when if fits. So is Hadoop, Oracle, Mongo etc. WHen you have ONE hammer everything looks like a nail. When you have a tackhammer,sledgehammer,screwdriver,clawhammer,rubber hammer you pick the one that fits tacks,spikes,screws,nails,wtc. I expect in this interview as in most, quotes were taken out of context, by interviewers who don’t understand the technology very well. Take it all with a grain of salt. -sign me an Oracle dba with time on Ingres, Oracle, dBase 3, MS Access, Informix/Illustra, Postgres, PostgreSQL, DB2, Sybase, UDB, StreamBase, NoSQL, MS SQLServer, and some homegrown stuff.

You have noticed no problems w/FB? This morning I logged on. In my 10 minutes there I had 7 “can’t write to database ” errors.

They might very well be having problems internally but from a user’s point of view, Facebook is running smoothly. I personally have not noticed any significant delays. I’m not aware of anyone leaving Facebook over performance issues, so I don’t think they need a complete redesign.
I think they are nearing their peak data usage. Proliferation among people under 30 is very high around the world. I’ve noticed that now people in their 40s and 50s are joining in, but they are not as active and thus do not generate a lot of demand. If they keep adding servers they’ll be OK for the next few years.

Right now I’m using NoSQL (Redis in this case) to help MySQL. When there are data spikes we save the data in memory with Redis so we don’t lose it because of slow MySQL.

bullshit. he’s got too much at stake to even listen to him on this front.

Too many trolls in comments. Every php-coder thinks he is mysql guru. Writing sites for 10 hits/day a different process instead of writing sites for 10k hits/second. Come back to mommy, lamers.

There is no where out but to continue with mysql, too painful to bite the bullet at this stage to replace the mysql. Introduce more server may be the choice out as for now.

I have a lot of respect for Michael Stonebraker as a computer scientist. To give him credit, when credit is due, he has made indisputable contributions to field. Having met him in person, I have to say that he is not just a great computer scientist, but a great guy overall.
Whether Facebook, or any other technology company for that matter, should use MySQL, NoSQL, z/OS mainframe or paper and pencil for keeping their data is an engineering decision and a business decision. And a team of people that would be qualified to make that call should consist of businessmen and engineers by trade, not computer scientists. There are many considerations that Stonebraker is not even aware of. Being highly intelligent and knowledgeable he, nevertheless is not an engineer, nor is he privy to the internal business information about Facebook. He simply does not have enough information that would allow him to make that call.
Facebook is the new Google. Out of 10 engineers in the Silicon Valley, at least 9 would drop whatever it is that they are doing and go work for Facebook, if given the chance. Consequently, some of the best engineering talent is already employed by Facebook, and they likely have a good reason to be doing the things the way they do it. Engineering is ultimately a practical discipline.
Most relational databases, save for a few column stores, store the data much the same way IBM System R did back in early 1970s. This approach to storing data works, and its practical. The computer science part here is ancient history at best. It’s great from the engineering standpoint, because it’s tried and true and it works. Done is better than perfect.

These seem like nice problems to have, regardless of what system(s) they are using.

Caveat: I represent NimbusDB, a NewSQL vendor.

That OldSQL has failed for web-facing applications is self-evident. Every substantial web-facing application has had to supplement or replace their MySQL, SQL Server, ORACLE etc systems, in addition to caching, sharding, denormalizing and in some cases re-engineering parts of the database system. At NimbusDB we talk to people every day that describe this OldSQL pain.

The case Mike Stonebraker makes is that the problem is not inherent to SQL but to the 30 year old internal architecture that all of these database systems use. There is no theoretical reason for SQL/ACID not to scale, and there are NewSQL products that provide existence proof of the point.

Would Facebook and others be in a better place had they started with a SQL database that goes faster when you add nodes to a live database, and is resilient to node or datacenter failure? Obviously yes.

SQL and ACID do scale out on commodity machines; historical implementations do not.

Yeah, I just had to answer Stonebraker claims on Facebook. So I join the fiction club too (and disclosure – I work for a competing newSQL company) – and say what FaceBook DBAs could have answered this interview – see here

The bottom line is that relational databases have a tremendous amount of overhead consumed by ‘keys’. They are fine for data that can be arranged in specific fields, such as a phone book or accounting data. In other words, the relations between objects (fields) are predetermined and the code takes advantage of this.

Further, the maintenance costs for ‘tuning’ is huge.

However, they fall apart when there are non-predetermined associations and the database is ‘navigated’, then the size of the database ‘explodes’ with increasing amount of data and ‘links’ are randomly created by users. This is the paradigm that Facebook uses. Examples of links in Facebook are ‘friend of’, ‘likes’, etc. A robust object-oriented data base is far more suitable for sociability and transaction processing speed. An additional advantage of the OODB-based systems is that code doesn’t have to be rewritten to add object classes and attributes.

One example of overcoming this conundrum for IBM DB2 is to use an OODB as the front-end transaction processor and to update the DB2 in the background. See IBM paper http://www.redbooks.ibm.com/redbooks/pdfs/sg246561.pdf describing how DB2 uses the Versant OODB to improve performance. Also see http://www.versant.com/index.aspx. There are competitive OODBs around, but I personally know that this one has been around for at least 20 years and have used and customized user applications for a sophisticated system engineering tool.

As soon as someone trots out the term “Webscale”, they lose all credibility in my eyes. It’s such a pointless term. It’s even worse than “NoSQL”, which as troll-marketing terms go is pretty bad.

This is a very shoddy piece of writing. Far too many paragraphs are spend presenting MySQL / “OldSQL” as the problem with little to no substance as to why. Then, finally, after 90% of readers have probably already grown bored and gone away the reader is introduced to “NoSQL” and “NewSQL”. Then the article ends leaving the reader with no idea what “NoSQL” and “NewSQL” are except a vague notion that “NoSQL” was an attempt to get away from “OldSQL” and “NewSQL” solves everything.

So let say they have 10K low end machine, each costing 200$ a month. That mean each machine manage 75000 users and that it cost 0.0026$ per user per year. 24 million total cost for a company that is quoted at more than 80 billion.

How the solution is unsuited?

For business savey but non-technical people you may want to uncover your unintentional, somewhat blanket statement, regarding open source: “…the company’s penchant for open source being among the reasons…”. This type of statement makes anyone look biased toward closed source software and makes it look like open source software as part of the problem which I’m sure you would agree is not true.

It’s nice to bring NewSQL to the forefront as the database technology for handling the woes of the social networks limitations but FaceBook has been functioning just fine. There’s #1 rule for all computer systems that works even today and that is K.I.S.S (keep it simple stupid). If the system works then leave it be, meaning don’t mess with it.

Apparently the so called NewSql/mysql expert who provided the information for this article, is all wrong about database issues at Facebook. They widely use cassandra database which is not relational for thier massive content. The system that Facebook innovated is far better than google’s and amazon’s technology.
The author didn’t convince me on his assessment that NoSql will offer only little performance over RDBMS. He should go to Mount Everest and take a break before starting out his professional life.
For webscale apps, sacrificing ACID properties provides huge performance benefits that cannot be achieved by RDBMS.

1. Engineers don’t understand data
2. If Engineers could scale, scalability as a problem wouldn’t exist.
3. VPs in-charge of data operations have glaring holes in their understanding of data I/O patterns Facebook needs.
4. The rest is then misalignment of skills/toolsets/ combined with chaos + egos

“We know the problem. We can’t do anything until the … ” screams “we don’t know the problem and don’t know how to fix it.”

They should call it “SQLSequel”..

From the start of this article it looked like Stonebreaker had no idea what he was talking about. His claim that SQL “is good for nothing” is ludicrous right on the face of it. He then further destroys his credibility by claiming that SQL skills “aren’t hard to come by”. Sure, you can find plenty of .NET or Java developers who can write syntactically correct, but horrible SQL code or who can design a poorly performing database (there are also some .NET and Java coders that are talented at SQL, so this isn’t a dig at them). The fact is though, even in this job market I get constant calls from recruiters because there is a shortage of *good* SQL developers.

At the end of the article it all becomes clear – he’s just trying to sell his own product. A product which, despite him referring to it as “NewSQL” includes nothing that isn’t already being done out there with existing SQL engines.

What a waste of an article (or should I say sales pitch).

So basically….

A database pioneer, computer scientist… who developed Postgres and Ingres relational databases.. and an active involvement in the development of other types of databases….

Against

Others who build and work on database systems…. correcting him….

Seems fair to me….

Are transactions that important for facebook?

One difference is that Walmart, fed gov, stock exchanges, etc. that have large db with high transaction volumes have adopted a three tiered architecture, middleware, I.e. TUXEDO. But that’s so ‘old school’, plus not free nor open source, that it’s seldom considered. Too bad, it works well, there is a reason all the highest TPC benchmark results are still achieved using TUXEDO. I wonder what the future holds now that TUXEDO and MySQL are controlled by Oracle????

Of course, Stonebraker is a businessman who will claim whatever he makes money with. But I hate this kind of dumb articles that have title and content of a single-person opinion.

There are lots of snippy little genii (at least in your own minds) here and too few bottles available with which to shut them up.

How sad. :(

This article explains why so many times you cannot get on FB or you get thrown off…thx for the article, it helps to enlighten us non techies.

From the article: “Facebook is operating a huge, complex MySQL implementation equivalent to ‘a fate worse than death,’ and the only way out is ‘bite the bullet and rewrite everything.’â€

This is *so* disconnected from reality.

I work for a $20 billion dollar company, which is growing 20% year over year, and run entirely on MySQL.

Is this ideal? No. Do we have problems? Yes. But would consider rewrite everything? Never! *That* would be a fate worse than death.

” There’s no guarantee they’re the solution for Facebook’s MySQL woes — the complexity of Facebook’s architecture and the company’s penchant for open source being among the reasons”

The article talks about open source as if it was derogatory.

I can tell you that my company wouldn’t be possible without several open source projects (OS, database, programming language, etc).

Heck: great companies like Facebook, Twitter, and Google, wouldn’t exist without open source software.

Now they are huge companies, and they could afford proprietary licenses; but they still choose free — not because of the cost, but because of the freedom.