Jay Taylor's notes

back to listing index

It now 404's so I've posted it here:Stevey's Google Platforms RantI was at Amazo... | Hacker News

[web search]
Original source (news.ycombinator.com)
Tags: news.ycombinator.com
Clipped on: 2012-12-07

Image (Asset 1/3) alt= It now 404's so I've posted it here:

Stevey's Google Platforms Rant

I was at Amazon for about six and a half years, and now I've been at Google for that long. One thing that struck me immediately about the two companies -- an impression that has been reinforced almost daily -- is that Amazon does everything wrong, and Google does everything right. Sure, it's a sweeping generalization, but a surprisingly accurate one. It's pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn't let me show it to anyone, even though recruiting loved it.

I mean, just to give you a very brief taste: Amazon's recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they've made to level it out. And their operations are a mess; they don't really have SREs and they make engineers pretty much do everything, which leaves almost no time for coding - though again this varies by group, so it's luck of the draw. They don't give a single shit about charity or helping the needy or community contributions or anything like that. Never comes up there, except maybe to laugh about it. Their facilities are dirt-smeared cube farms without a dime spent on decor or common meeting areas. Their pay and benefits suck, although much less so lately due to local competition from Google and Facebook. But they don't have any of our perks or extras -- they just try to match the offer-letter numbers, and that's the end of it. Their code base is a disaster, with no engineering standards whatsoever except what individual teams choose to put in place.

To be fair, they do have a nice versioned-library system that we really ought to emulate, and a nice publish-subscribe system that we also have no equivalent for. But for the most part they just have a bunch of crappy tools that read and write state machine information into relational databases. We wouldn't take most of it even if it were free.

I think the pubsub system and their library-shelf system were two out of the grand total of three things Amazon does better than google.

I guess you could make an argument that their bias for launching early and iterating like mad is also something they do well, but you can argue it either way. They prioritize launching early over everything else, including retention and engineering discipline and a bunch of other stuff that turns out to matter in the long run. So even though it's given them some competitive advantages in the marketplace, it's created enough other problems to make it something less than a slam-dunk.

But there's one thing they do really really well that pretty much makes up for ALL of their political, philosophical and technical screw-ups.

Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not.

Micro-managing isn't that third thing that Amazon does better than us, by the way. I mean, yeah, they micro-manage really well, but I wouldn't list it as a strength or anything. I'm just trying to set the context here, to help you understand what happened. We're talking about a guy who in all seriousness has said on many public occasions that people should be paying him to work at Amazon. He hands out little yellow stickies with his name on them, reminding people "who runs the company" when they disagree with him. The guy is a regular... well, Steve Jobs, I guess. Except without the fashion or design sense. Bezos is super smart; don't get me wrong. He just makes ordinary control freaks look like stoned hippies.

So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

His Big Mandate went something along these lines:

1) All teams will henceforth expose their data and functionality through service interfaces.

2) Teams must communicate with each other through these interfaces.

3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.

5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6) Anyone who doesn't do this will be fired.

7) Thank you; have a nice day!

Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.

#6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at Wal*Mart, and is a big genial scary man who used the word "hardened interface" a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made LOTS of forward progress and made sure Rick knew about it.

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon's vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon's dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

- pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.

- every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.

- monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says "oh yes, I'm fine", it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum.

- if you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.

- debugging problems with someone else's code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.

That's just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers.

This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.

At this point they don't even do it out of fear of being fired. I mean, they're still afraid of that; it's pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they've come to understand that it's the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it's the right thing because SOA-driven design enables Platforms.

That's what Bezos was up to with his edict, of course. He didn't (and doesn't) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.

You wouldn't really think that an online bookstore needs to be an extensible, programmable platform. Would you?

Well, the first big thing Bezos realized is that the infrastructure they'd built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel' o' other services browsable at aws.amazon.com. These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.

The other big realization he had was that he can't always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn't use the goddamn website. It's not even super clear whose mom he was talking about, and doesn't really matter, because nobody's mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I've just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.

I'm not really sure how Bezos came to this realization -- the insight that he can't build one product and have it be right for everyone. But it doesn't matter, because he gets it. There's actually a formal name for this phenomenon. It's called Accessibility, and it's the most important thing in the computing world.

The. Most. Important. Thing.

If you're sorta thinking, "huh? You mean like, blind and deaf people Accessibility?" then you're not alone, because I've come to understand that there are lots and LOTS of people just like you: people for whom this idea does not have the right Accessibility, so it hasn't been able to get through to you yet. It's not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software -- or idea-ware for that matter -- fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an Accessibility failure.

Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there's more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.

But I'll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.

So yeah. In case you hadn't noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I've worked at. But I will never get this little rant published, and you'll never get it read, unless I start to wrap up.

That one last thing that Google doesn't do well is Platforms. We don't understand platforms. We don't "get" platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.

But no. No, it's like our tenth or eleventh priority. Or fifteenth, I don't know. It's pretty low. There are a few teams who treat the idea very seriously, but most teams either don't think about it all, ever, or only a small percentage of them think about it in a very small way.

It's a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they're building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I'm concerned, it's none of them. Stubby's great, but it's like parts when you need a car.

A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: "So is it the Stalker API?" She got all glum and said "Yeah." I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.

Microsoft has known about the Dogfood rule for at least twenty years. It's been part of their culture for a whole generation now. You don't eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.

Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that's not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there's something there for everyone.

Our Google+ team took a look at the aftermarket and said: "Gosh, it looks like we need some games. Let's go contract someone to, um, write some games for us." Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.

You can't do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don't have a Steve Jobs here. I'm sorry, but we don't.

Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn't need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.

I apologize to those (many) of you for whom all this stuff I'm saying is incredibly obvious, because yeah. It's incredibly frigging obvious. Except we're not doing it. We don't get Platforms, and we don't get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.

So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don't "get" much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to msdn.com, and spend some time browsing, and you've never seen it before, prepare to be amazed. Because it's staggeringly huge. They have thousands, and thousands, and THOUSANDS of API calls. They have a HUGE platform. Too big in fact, because they can't design for squat, but at least they're doing it.

Amazon gets it. Amazon's AWS (aws.amazon.com) is incredible. Just go look at it. Click around. It's embarrassing. We don't have any of that stuff.

Apple gets it, obviously. They've made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dogfood. And you know what? They make pretty good dogfood. Their APIs are a hell of a lot cleaner than Microsoft's, and have been since time immemorial.

Facebook gets it. That's what really worries me. That's what got me off my lazy butt to write this thing. I hate blogging. I hate... plussing, or whatever it's called when you do a massive rant in Google+ even though it's a terrible venue for it but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it'd be pretty easy to just go. But Google is home, so I'm insisting that we have this little family intervention, uncomfortable as it might be.

After you've marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn't look because I didn't want to get too depressed), head over to developers.google.com and browse a little. Pretty big difference, eh? It's like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.

Please don't get me wrong here -- I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They're kicking ass as far as I'm concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.

I'm just frankly describing what developers.google.com looks like to an outsider. It looks childish. Where's the Maps APIs in there for Christ's sake? Some of the things in there are labs projects. And the APIs for everything I clicked were... they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it's all snouts and horse hooves.

And also don't get me wrong about Google+. They're far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producters.

Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs -- Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it's hard for them to get funding for it because it's not part of our culture. Maestro's funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it's a fluffy rabbit versus a T-Rex. The Docs team knows they'll never be competitive with Office until they can match its scripting facilities, but they're not getting any resource love. I mean, I assume they're not, given that Apps Script only works in Spreadsheet right now, and it doesn't even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.

Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook -- that is, the stock service they offer with walls and friends and such -- is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.

You know how people are always saying Google is arrogant? I'm a Googler, so I get as irritated as you do when people say that. We're not arrogant, by and large. We're, like, 99% Arrogance-Free. I did start this post -- if you'll reach back into distant memory -- by describing Google as "doing everything right". We do mean well, and for the most part when people say we're arrogant it's because we didn't hire them, or they're unhappy with our policies, or something along those lines. They're inferring arrogance because it makes them feel better.

But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we're being fools. You can attribute it to arrogance, or naivete, or whatever -- it doesn't matter in the end, because it's foolishness. There IS no perfect product for everyone.

And so we wind up with a browser that doesn't let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I'm actually going blind. For real. I've been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they're quite brazen about it, and Fuck You if you're blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.

It's not just them. It's everyone. The problem is that we're a Product Company through and through. We built a successful product with broad appeal -- our search, that is -- and that wild success has biased us.

Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers... if only they could be monetized somehow... you can see how he arrived at AWS, in hindsight.

Microsoft started out as a platform, so they've just had lots of practice at it.

Facebook, though: they worry me. I'm no expert, but I'm pretty sure they started off as a Product and they rode that success pretty far. So I'm not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.

Maybe they just looked at us and asked: "How can we beat Google? What are they missing?"

The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don't do internal service-oriented platforms, and we just as equally don't do external ones. This means that the "not getting it" is endemic across the company: the PMs don't get it, the engineers don't get it, the product teams don't get it, nobody gets it. Even if individuals do, even if YOU do, it doesn't matter one bit unless we're treating it as an all-hands-on-deck emergency. We can't keep launching products and pretending we'll turn them into magical beautiful extensible platforms later. We've tried that and it's not working.

The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate -- ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

I'm not saying it's too late for us, but the longer we wait, the closer we get to being Too Late.

I honestly don't know how to wrap this up. I've said pretty much everything I came here to say today. This post has been six years in the making. I'm sorry if I wasn't gentle enough, or if I misrepresented some product or team or person, or if we're actually doing LOTS of platform stuff and it just so happens that I and everyone I ever talk to has just never heard about it. I'm sorry.

But we've gotta start doing this right.

Image (Asset 2/3) alt= Although the etiquette of posting it here is debatable this post is gold, it is not (totally) an ordinary disgruntled coder rant. It's worth reading even for the small list of problems that he gives that Amazon faced when trying to go to an all-services architecture.

But if I had to summarize the main point, I would choose: "The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later." Start with the platform. This is so obvious, yet a lot of middle and higher managers ignore this either because (i) they are totally ignorant (see Peter's Principle), sadly the case that I usually face, (ii) they pragmatically ignore it, because they are doing greedy optimization for the next 6mos-year and developing the platform is just a "waste of time", or (iii) they ignore developing and making available the platforms they already have because they think it's their "competitive advantage".

The way I read his post, Google's case seems to be (iii). They have great internal tools and platforms for many years now, why they wouldn't develop good APIs around these and make them available is a mystery to me. Wouldn't they have eaten AWS for lunch if they platform-atized their Map-Reduce tools, say around 2004?


Image (Asset 3/3) alt= It's not that. The procedure for open-sourcing Google-owned code is very easy: you e-mail Chris DiBona and he almost always says yes. There've been a bunch of major pieces of the Google infrastructure open-sourced (eg. cTemplate, Closure, Protocol Buffers, Guice).

The problem is really technical. Because of the open codebase and "edit anything" culture, basically every piece of infrastructure at Google, unless it was designed at the start for open-sourcing, has dependencies on every other piece of infrastructure. Untangling them takes far more time than you could 20% on. That's why you don't see more open-source platforms from Google.


lamnk 422 days ago | link

Ugh, I think you are mistaking between open source and being a platform here. A platform need not to be open source.


nostrademons 422 days ago | link

But all the same issues apply, i.e. if you can't disentangle your code from proprietary interfaces enough to open-source it, you probably will not be able to put a service API on it that third-party programmers will be able to use without a lot of handholding. (And Google doesn't do handholding, as just about everyone knows by now.)


lamnk 422 days ago | link

No, it's not the same issues. I'm not a programmer, but I know that offering a well designed API to a service has nothing to do with its code being closed or not.

When I was learning to program, I was very interested in 3 Google's core products API: Search, Adsense and Adwords but got turned off because their APIs are very crippled. I understand that Google do not want to expose their core products (cash cows) to abuse, however that's exactly what Steve argued in parent's quote. If the closeness (nothing to do with code) is already in Google's mindset, how do they become a "platform oriented company"? Steve says that it will take a dramatic cultural change to catch up. Steve also chooses Amazon's example to prove: when the years of explosive growth were over, Jeff Bezos had to think of some new expansion strategies. He was betting on the cloud, whatever that means, or on being a platform, and he implemented his vision with an iron fist, because paradigm shift is hard, you have to consistently push people into doing it.


SoftwareMaven 422 days ago | link

I would call that very dangerous and very likely a premature optimization. Nobody cares you are a platform until they use your product, and if you spend all your time building a platform, there will be no product to use.

The path requires far more finese than "always create a platform". You need to balance over-engineering with the ability to re-evaluate previous decisions.

I agree Google has wandered far off this path, but imnot sure they are on any more well defined path than Yahoo. They just happen to be earlier.


bluedanieru 420 days ago | link

This applies to startups, less so to more mature companies who can and will throw more resources behind a project. After the prototyping phase, once you're reasonably sure of the direction you're going to go in, it isn't a bad idea to keep extensibility in mind from the start. It's more like premature architecting, if anything, than premature optimization, which is much less offensive (and not even necessarily bad).


skaphan 422 days ago | link

But ... in the one case we are aware of, of a whole company, Amazon, using a service oriented architecture, they did it after the fact. The did bolt it on later, or rather, they (according to Steve) ripped everything up and re-did it in this way. I'm sure it was absolute hell to do this, but that is what they apparently did. So it isn't a requirement to build a platform first, then build everything else. It might be nicer to do it that way, if it's the best prioritization of resources at the time, but that's not what happened in the case being discussed.


debaserab2 422 days ago | link

I agree with you in principal but I think practically this isn't always possible.

Sometimes you don't know if a product is worth being an entire platform until it's been thoroughly field tested. It's too expensive to assume that everything is.

Perhaps in Google's case there could have been more foresight, though.


flgr 421 days ago | link

Isn't it the same with scalability? You won't know if you need a product to be scalable until you have actually launched it and seen its success.

Yet, every product developed at Google seems to be developed with absolute scalability in mind from the get-go, wasting lots of effort invested into scalability in the case of the product failing to get significant customer adoption.


asr 421 days ago | link

But Google will only consider a product a success if it achieves sufficient scale, while it would (apparently) be happy to have a successful product that is not a platform.

Since 100% of successful Google products will be large, it makes sense to design for scale from the get-go. But if some of Google's products won't end up being platforms, it might make more sense to bolt that on later.


anigbrowl 422 days ago | link

Thanks for reposting. It's a testament to Google's flat-seeming corporate culture that team members are encouraged to speak their mind so frankly.

The key paragraph for me is this one: Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs -- Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it's hard for them to get funding for it because it's not part of our culture. Maestro's funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it's a fluffy rabbit versus a T-Rex. The Docs team knows they'll never be competitive with Office until they can match its scripting facilities, but they're not getting any resource love. I mean, I assume they're not, given that Apps Script only works in Spreadsheet right now, and it doesn't even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.

This is my perennial Google grumble in a more elegantly stated nutshell: Google is a magical playground of wonderful things, few of which are finished. It's magical because the parts that work do so at great speed and with almost spookily good results. It's maddening because some things are automagically interoperable and others are head-deskingly separate or incomplete. The main thing I use Google Docs for is word processing, and though it's great to have stuff in the cloud it's just absurd - absurd - that when I move to an Android device the UI is unavailable, or that the full UI lacks support for things like user-defined text styles.

Thing is, Docs owuld be fine if I was just using it for a personal journal or some glorified notepad. But now that I'm a student again, I'm writing letters and model legal briefs and things like that, and it's just a pain in the ass for that, so every day I flirt with reinstalling MS Office...2003. The stuff I need to do was all in Office >10 years ago, and I would even put up with Clippy again just to have things like outlining, styles and templates.

Steve Yegge should get a seat on the board, pronto.


VonLipwig 422 days ago | link

He took it down for a reason. I can't help but feel reposting it is bad online etiquette


Cushman 422 days ago | link

Who downvoted this? We should probably have a real conversation about it.

Personally, I'd like to think that if I posted an essay or especially a rant that I later thought was a mistake, and decided to retract it, other hackers would respect my decision and not reproduce it. Sure, it's still cached somewhere, and assholes on 4chan will repost it just to fuck with me. If you really want to, you can probably find a copy. But I'd like to think that our community is classier than to spread it around deliberately, knowing it's something I'd rather not put my name to.

What's the benefit to the community the other way, aside from voyeurism?


redstripe 422 days ago | link

"If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place." - Eric Schmidt


Cushman 422 days ago | link

Uh... Wrong-headed as that notion is, it's also talking way past me. How do you see it as relevant to what I said?


cobrausn 422 days ago | link

We approve of this comment.


DHS, CIA, NSA, FBI, ATF, and the DEA.


wazoox 422 days ago | link

> What's the benefit to the community the other way, aside from voyeurism?

It's one of those fantastic, really wonderful Stevey's rants, on a par with "the emacs problem". I'm really incredibly happy to have had a chance to read it. So yes, it's probably bad etiquette, but OTOH will it hurt Google in any way? Certainly not. So, hey, thanks for the mistake M. Yegge, and continue the good work.


ChuckMcM 422 days ago | link

Steve and I overlapped at Google and I've always felt his missives were pretty clear and actionable. (unlike some which were simply 'this sucks') That this one made it outside the 'plex is probably a mistake, I hope he doesn't pay to high a price for that.

Of course there is the risk that he might get something changed inside of Google (I'm thinking unlikely) it would be a pretty amazing thing if the Google infrastructure was available as a platform (and no AppEngine does not count)


brlewis 422 days ago | link

He deliberately took it down: https://plus.google.com/110981030061712822816/posts/bwJ7kAEL...

See my comment on that post for why I think his taking it down was a mistake.


outside1234 422 days ago | link

I suspect he meant this only for internal Google consumption. These sorts of rants happened all the time but this one is very good and very right on.

He probably proved the accessibility point by the simple fact that he accidentally posted it with his personal and not corporate account.


monochromatic 422 days ago | link

I haven't read this whole thing yet, but based on length, I am willing to assume it's a faithful reproduction of something Steve Yegge wrote.


asnyder 422 days ago | link

While the above might be true, I know that Amazon excels at customer service whereas google's is essentially non-existent. So from the customer perspective, I would choose Amazon over google any day.


notatoad 421 days ago | link

do you give google money? if not, you're not a customer and have probably never experienced their customer service or a "customer perspective".


JezChatfield 421 days ago | link

I'm a multiple paying customer. I use Google Apps, AdWords. I can't use Google Plus under the same ID as Apps, because Google appear to think that paying customers are less valuable than free users. And we (Apps Users) went for ages without any explanation other than "we're working on it".

To do any work using AdWords and Google Apps and Google Analytics, and Webmaster Tools I end up using three accounts and two browsers.

If Google did Platforms, then the first one to fix is "who the F* am I?" (Authentication and Access Control) and make it get used everywhere.

Hell, Google can't even work out what to name things. So you get scammers exploiting Google.com naming to convince marks that the scammer really works for Google. This is also a Platform issue - Google.com should be for Google stuff - not abusable to allow, for example, spoofing a search results page so you can claim that Google has given you special access to modify search results (one SEO scam that I keep encountering). Non-Google stuff should be on an identifiable domain, which will eventually permeate user consciousness as being "not Google content".

I speak as a pure outsider and consumer of Google stuff. But I'm sufficiently vexed by Google Apps/Accounts/Profiles to want to lose the paid for services as much and as soon as possible. I get better (or no worse) treated when I don't buy. :)

And Stephen's point about Google arrogance - yes, I think Google is arrogant. I was an AdWords Help Forum Top Contributor until Google decided that small businesses should only receive AdWords support from other users, not Google. That's seriously annoying. I stopped contributing when it became clear that Google thought small businesses should be denied any customer service - you really can't have third parties answering questions about why Google isn't accepting a specific credit card, or even why a specific disapproval has been given in an account you can't see. That was pure arrogance - dumping on volunteers and crapping on small businesses. I'm told it's different now but I still have such a bad taste in my mouth from the experience I don't want to get involved again. Google can be arrogant, 100%.


johnyzee 421 days ago | link

I give Amazon money for AWS and no customer support in sight.


frwololo 421 days ago | link

Unless you're constantly blocking ads, you're a google customer and you give them money. Don't be fooled by an apparently "free" service :)


noblethrasher 421 days ago | link

Viewing ads makes you the product not the customer.


nl 421 days ago | link

There is a thing called a "multi-sided market".

Saying "you are the product not the customer" is not only popularist, but wrong.

Think about Skype - most of their customers don't pay, but they derive value FROM Skype - AND they add value to the paying customers.

Simplifying complicated markets with statements like "you are the product not the customer" is a disservice to the discussion.


calibraxis 421 days ago | link

I've worked with a number of tech/advertising corporations. The parent poster is right. The real product is eyeballs, and the customer is another corporation called the advertiser.

For example, The New York Times offers some value to the owners of those eyeballs, but note how they work: they lay out the advertising first, then what's left is what they call the "news hole" and they put some stuff there. The news hole lets them farm the eyeballs. (The ripe juicy professional-managerial variety.)

This is not something programmers like to think about, so they tend to wear ideological blinders. They don't simply observe the money flow: who is buying and who is selling? The eyeball owners are not participating in a market transaction as either buyer or seller. Their attention is the product.


nl 421 days ago | link

To say "eyeballs are the product" simplifies this too much.

Think of sports teams - advertisers pay their salaries, but when they are performing on the sports field advertisers are irrellevant.


UpTil4Music 421 days ago | link

You either pay for the product, or you ARE the product.


ksolanki 421 days ago | link

This is what I am taking home:

1) There IS no perfect product for everyone.

2) Making something a platform has some chance of addressing the above.

Me trying to make sense of it, but meanwhile let me say it aloud again. There IS no perfect product for everyone. Thanks, Stevey.


bdr 422 days ago | link

Thank you for rescuing this. It has great engineering wisdom and historical value.


option1138 421 days ago | link

We do mean well, and for the most part when people say we're arrogant it's because we didn't hire them, or they're unhappy with our policies, or something along those lines. They're inferring arrogance because it makes them feel better.

I LOL'd when I read this. How ironically arrogant.


bitmonk 411 days ago | link

A lot of this is great, but it's clear that the author has not actually followed the process of launching AWS. Bezos did not wake up one day and decide to start selling his datacenter space. AWS was developed, clearly after this person left Amazon, for Amazon developers. The engineer (CTO?) responsible for this woke up one day and realized, well, fuck, we could sell this. Pairing some of the stories here with that basic knowledge actually explains a lot of the SOA startups popping up in EC2. Companies are beginning to form in the shape of departments at Amazon, which is interesting, I'm not sure it's awesome.

That said, let's not fool ourselves. There is nothing more wrong with The Internet right now than EC2. It is poorly executed and Amazon's position as a market leader has made them incredibly lazy about improving it or contributing to the core technologies (Xen!) which are responsible for the most complex bits of it.


pitdesi 422 days ago | link

For all those who are criticizing this public posting, first note that it is publicly available in several dozen other locations, including some tech rags. More importantly, STEVE HIMSELF SAID IT'S OK TO REPOST - https://plus.google.com/112678702228711889851/posts/eVeouesv...

Part of me feels that this is actually written to the general public, and the whole public/private thing is a cover up.


Lewisham 422 days ago | link

Part of me feels that this is actually written to the general public, and the whole public/private thing is a cover up.

This is highly unlikely. This sort of internal conversation happens a lot internally. There's no need to take it public. What would it serve? Some sort of escalation of the problem? Unlikely. It's more likely to get you a stern email than it is to get it heard.

Steve has no problems getting the ears of developers. I can't see why he would want to make it public.


frossie 422 days ago | link

There's no need to take it public. What would it serve?

Well I can speculate: Because some issues seem like real debates when you have them internally, and then you tell a bunch of outsiders and they go "uh, it's a no-brainer" and that tells you something you didn't know before: that you can't see the wood for the trees.

The thing that surprised me most about the OP is that Google actually sees itself as a product company. With the exception of search and maps (and perhaps not even search so much now) they actually aren't strong with their products, and (aside from the gargantuan scaling issues that they have to solve) they don't give the impression of working to improve their products. How much has gmail evolved over the years? Calendar? Reader? Sure, for all I know they are all tinkering furiously under the hood, but that is not what the outside world expects from somebody positioned as a product company.


fprefect 421 days ago | link

"You know how people are always saying Google is arrogant?" Yup, bidding funny numbers like pi for the Nortel patents was incredibly arrogant. It certainly wasn't humble. There was no humility there. The representatives from Google acted like a group of teenagers giggling and making fart noises in the back of the classroom. I'm sure they thought it was funny at the time, but this is a prime example of what people mean by arrogant. The public perception is that Google people are geeks, and geeks and arrogance go hand in hand. Watch Fallon's Nick Burns sketch on SNL sometime if you need an illustration.


teja1990 422 days ago | link

Its out of context, but please bear with me. Is it ok if amazon has an internal library(hypothetical) that is a collection of all its online books that they sell, can they have access to it , i mean normal guys have to pay for just getting a chapter out of a big book. For Amazon guys is there any concession , can they check on some cool book just like that?


GryMor 422 days ago | link

That is not the sort of library he is talking about, I, unfortunately, may not give details, but rest assured that is not what is being referred to, at all.


danssig 422 days ago | link

I only got down the first few paragraphs but it sounds like most of the complaints are "retail vs tech company" stuff. Retail has razor-thin margins so of course they will all be cheap as hell. If you think Amazon is bad, try working for Walmart HQ.


rahoulb 422 days ago | link

you should read the rest of it - it's not about amazon's working conditions, it's about a mistake that google is making.

(btw i didn't downvote you)


danssig 421 days ago | link

I did read the rest of it and I still say the complaints he has against Amazon sound more like complaints against working in retail. If you like having a nice working environment, stay far away from retail because they're all like that. There's no money to be anything else.

P.S. When I can trade HN karma for time off from work I'll start to care about it.


0x12 422 days ago | link

Posting a link to a web based copy is one thing, copying the text integral to HN is probably not ok.


Lists | RSS | Bookmarklet | Guidelines | FAQ | DMCA | News News | Feature Requests | Y Combinator | Apply | Library