"On June 6, 2020, your Google Kubernetes Engine (GKE) clusters will start accruing a management fee, with an exemption for all Anthos GKE clusters and one free zonal cluster.

Starting June 6, 2020, your GKE clusters will accrue a management fee of $0.10 per cluster per hour, irrespective of cluster size and topology.

We’re making some changes to the way we offer Google Kubernetes Engine (GKE). Starting June 6, 2020, your GKE clusters will accrue a management fee of $0.10 per cluster per hour, irrespective of cluster size and topology. We’re also introducing a Service Level Agreement (SLA) that’s financially backed with a guaranteed availability of 99.95% for regional clusters and 99.5% for zonal clusters running a version of GKE available through the Stable release channel. Below, you’ll find additional details about the new SLA and information to help you reduce your costs."

Oh, and per their docs, three-control-planes decision is not reversible - I cannot in fact shut two of those down without shutting down my production cluster and starting a new one. https://cloud.google.com/kubernetes-engine/docs/concepts/typ...

Awful. Just so awful.

Edit: To answer some questions below - we have a single-tenant model where we run an instance of our async discussion tool (https://aether.app) per customer for better isolation, that’s why we had bought into Kubernetes / GCP. Since we have our own hypervisor inside the clusters, it makes me wonder whether we can just deploy multiple hypervisors into the same cluster, or remove the Kubernetes dependency and run this on a Docker runtime in a more classical environment.

There's also one completely free zonal cluster for hobbyist projects.

The main issue is that not charging for the control plane and charging for the control plane leads to two very different Kubernetes architectures, and as per your docs, those decisions made at the start are very much set in stone. You cannot change your cluster from a regional cluster to a single zone cluster for example. So you have customers who built their stacks taking into account your free control plane, and you’re turning the screws in by adding a cost for it — but they cannot change the type of their cluster to optimise their spend, since, per your docs, those decisions are set in stone. That’s entrapment.

You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today.

That said, many of us made a bet on GCP. For us in particular, we made a bet to the point that our SQL servers are on AWS, but we still switched to GCP for ‘better’ Kubernetes and for not nickel and diming, since AWS had a charge that looked like it was designed to convey that they’d much rather have you use their own stuff than Kubernetes. It is a relatively trivial amount, but it makes a world of difference in how it feels and you guys know more than anyone how much of these GCP vs AWS decisions are made based not on data sheets but for the ‘general feel’ for the lack of a better word.

AWS’ message is that they’re the staid, sort of old fashioned, but reliable business partner. GCP’s message, as of this morning, is stop using GCP.

These changes won't take effect until June - customers won't start getting billed immediately. I'm sorry that you feel trapped, that's not our intention.

> You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today.

This is great feedback, but clusters should be treated like cattle, not pets. I'd love to learn more about why your clusters must be so static.

What’s inside our clusters are indeed cattle, but the clusters themselves do carry a lot of config that is set via GCP UI for trivial things like firewall rules. Of course we could script it and automate, but your CLI tool also changes fast enough that it becomes an ongoing maintenance burden shifted from DevOps to engineers to track. In other words, it will likely incur downtime due to unforeseen small issues.

It’s also in you guys’ interest that we don’t do this and clusters are as static as possible right now, since if we are risking downtime and moving clusters, we’re definitely moving that cluster back to AWS.

Heh... how many teams actually treat their clusters like cattle, though? Every time I advocate automation around cluster management, people start complaining that "you don't have to do that anymore, we have Kubernetes!"

Some people get it, yes, but even of that group, few have the political will/strength to make sure that automation is set up on the cluster level—especially to a point where you could migrate running production workloads between clusters without a potentially large outage / maintenance window.

Please don't do this. You can apologize for your actions work to improve in the future , but you cannot apologize for how someone feels as a result of your actions.

Also, intent doesn't matter unless you plan to change your behavior to undo or mitigate the unintended result.

https://aws.amazon.com/eks/sla/

My problem is that this fee doesn't look very "cloud" friendly. Sure the folks with big clusters won't even notice it, but others will sweat it.

The appeal of cloud is that costs increase as you go, and flat rates are typically there to add predictability (see BigQuery flat rate). This fee does the opposite.

That about sums up most things Google does for developers.

What about clusters that are used for lumpy work loads? Like data science pipelines? For example, our org has a few dozen clusters being used like that.

Each pipeline gets its own cluster instance as a way to enforce rough and ready isolation. Most of the times the clusters sit unused. To keep them alive we keep a small, cheap, preemptive node alive on the idle cluster. When a new batch of data comes in, we fire up kube jobs which then triggers GKE autoscaling that processes the workload.

This pricing change means we're looking at thousands of dollar more in billing per month. Without any tangible improvement in service. (The keepalive node hack only costs $5 a month per cluster.) We could consolidate the segmented cluster instances into a single cluster with separate namespaces, but that would also cost thousands in valuable developer time.

I don't know how common our use pattern is, but I think we would be a lot better served by a discounted management fee when the cluster is just being kept alive and not actually using any resources. At $0.01, maybe even $0.02, per hour we could justify it. But paying $0.10 to keep empty clusters alive is just egregious.

(As a security person: ugh.)

More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.

I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use.

For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that.

If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage.

Then you just use normal helm install routines to install and setup logging, dashboards, and all that.

Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't.

As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging.

Azure doesn't charge for cluster management (yet), IIRC.

For example you could use NFS for 90% of the storage needs for logging and sharing files between pods. Then use local storage, FCOE, or iSCSI-backed PVs for databases.

If you are doing bare hardware and your requirements for latency are not too stringent then not hosting databases in the cluster is also a good approach. Just used dedicated systems.

If you can get state out of the cluster then that makes things easier.

All of this depends on a huge number of other factors, of course.

I think NFS is heavily underrated. It's a good match for things like hosting VM images on a cluster and for Kubernetes.

In the past I really wanted to use things iSCSI for hosting VM images and such things, but I've found that NFS is actually a lot faster for a lot of things. There are complications to NFS, of course, but they haven't caused me problems.

I would be happy to use it in production, and have recommended it, but it's not unconditional. It depends on a number of different factors.

The only problem with NFS is how do you manage the actual NFS infrastructure? How much experience does your org have with NFS? Do you already have a existing file storage solution in production you can expand and use that with Kubernetes?

Like if your organization already has a lot of servers running ZFS, then that is a nice thing to leverage for NFS persistent storage. Since you already have expertise in-house it would be a mistake not to take advantage of it. I wouldn't recommend this approach for people not already doing it, though.

If you can afford some sort of enterprise-grade storage appliance that takes care of dedupe, checksums, failovers, and all that happy stuff, then that's great. Use that and it'll solve your problems. Especially if there is some sort of NFS provisoner that Kubernetes supports.

The only place were I would say it's a 'Hard No' is if you have some sort of high scalability requirements. Like if you wanted to start some web hosting company or needed to have hundreds of nodes in a cluster. In that case then distributed file systems is what you need... Self-hosted storage aka "Hyper Converged Infrastructure". The cost and overhead of managing these things is then relative small to the size of the cluster and what you are trying to do.

It's scary to me to have a cluster self-host storage because storage can use a huge amount of ram and cpu at the worst times. You can go from a happy low-resource cluster, then a node fails or other component takes a shit, and then while everything is recovering and checksum'ng (and lord knows what) the resource usage goes through the roof right during a critical time. The 'perfect storm' scenarios.

And if that is your central infrastructure, shouldn't it be worth the money?

I do get the issue with having cheap and beefy hardware somewhere else, i do that as well, but only for private. My hourly salary spending or wasting time on stuff like that costs the company more than just paying for an additional cluster with the same settings but perhaps with much less Nodes.

If more than one person is using it, the multiplication effects for suddenly unproductive people, is much higher. Also that decreases the per head cost.

Do you want to mention anything more about what you're hoping to get out of hierarchy? Is it just a management tool, is it for access control, metering/observability, etc...?

Thanks, A

Nice.

That said they do make it damn hard. Our k8s cluster is as basic as it comes, no databases, simple deployments, but we do still have a dependency on Google Cloud Loadbalancer (which we hate).

If pricing goes up too much from this we'll move, but the GCL dependency will be a PITA :/

Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.

If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allows you to scale nodes quickly and easily. Works on bare metal or in the cloud (or both, but make sure you have super low latency between data centers if you are going to do that). It's also pretty valuable to be able to call Red Hat support if something goes wrong. (I still shake my head over the number of days I spent debugging arcane networking issues on EKS before moving to OpenShift, which would have paid for a year or more of support just by itself).

https://pages.awscloud.com/Gartner-Magic-Quadrant-for-Infras...

Also lack of SLA / shady SLA does not help.

Ps. Talking as someone with hands on experience.

Ps2. Azure support is terrible and their response times are constantly breaking SLA..

Honestly, I've been pleasantly surprised.

https://www.canalys.com/static/press_release/2020/Canalys---...

AliCloud and Rackspace are very close to GCP as well.

That being said, if you're planning on running Kubernetes, I'd choose GCP over any other offering - the tooling and support just seems better, in my entirely subjective opinion.

My experience is that once AWS offers a new service that gets attention, a few month later also Azure offers it - and vice versa.

It has more features, yes. How well those features work is another matter entirely.

With this fixed fee model, the change will barely make a difference (== Google revenue) for the large customers who can spare the money, but will create a significant entry barrier to that side project / super-early stage that considers getting hooked on GCP, specifically GKE.

Then again, not my decision to make.

There is no billing protection (which could make you very poor very fast) and every service has a certain cost and quality which is just not feasable in the beginning.

Even GKE with its free kubernetes master does block a lot of resources on the nodes: https://cloud.google.com/kubernetes-engine/docs/concepts/clu...

Also a ton of great features on gke you will probably never use if you are too small. It is so much cheaper to just get cheap hardware somewhere and put your own k8s onto it if you have more time then money.

Even on Digital Ocean you have the load balancer problem: you need to use the provided and also 'costly' LoadBalancer service. There is only one hacky way to prevent it by exposing your ingress on the host and mapping that one ip but then you loose all the self healing stuff and loadbalancing capability.

After seeing what they did to Google Maps and Api.AI / Dialogueflow jumped from free to 5k$ overnight - just can't trust them.

There is a billion dollar company on the horizon for whoever can best commoditize bare metal with an apple-esque usability model.

How is Google different from other cloud poviders in terms of vendor lock-in?

For AWS or Azure I developed way more trust over time - could be subjective - but also could be that there is a reason Google is distant 3rd in the game.

I also think of them as the second, inferior cloud, but they're almost certainly the better k8s hoster. If you're serious about running k8s on AWS, there's a good chance you're doing something like CloudPosse's Terraform-based setup, not EKS.

I'm not sure what your usecase is that you would choose gke and you are worried about 300$ per month infra costs.

For Corp we use gke. For private I use selfhostet k3samd for our startup asuper cheap digital ocean cluster.

That feels like the wrong attitude.

If your kubernetes cluster is part of your core infrastructure, then 300$ more or less should not be an issue at all (not to say that i think 300$ is nothing).

That should not mean that you should waste money but often enough, if you buy cheap and your hardware breaks and your time&material costs much more then what a better hardware would have cost, then you wasted money by buying cheap.

Unfortunate with IT products, there are certain things which are not directly visible: Like how secure is your product. GCP offers 2FA, Digital Ocean does not. How much money is it worth to you to have your whole infrastructure protected by 2FA? For me in a business context, non 2FA would be a no go.

Everyone's having EKS cost problems while I'm just sitting over here paying nothing for ECS control planes.

https://aws.amazon.com/eks/pricing/

> As of November 28, 2017, Google Kubernetes Engine no longer charges a flat fee per hour per cluster for cluster management, regardless of cluster size, as provided at https://cloud.google.com/kubernetes-engine/pricing. Accordingly, Google no longer offers a financially-backed service level agreement for the Google Kubernetes Engine service. The service availability of nodes in a Google Kubernetes Engine-managed cluster is covered by the Google Compute Engine SLA at https://cloud.google.com/compute/sla.

> Uptime for Google Kubernetes Engine is nevertheless highly important to Google, and Google has an internal goal to keep the monthly uptime percentage at 99.5% for the Kubernetes API server for zonal clusters and 99.95% for regional clusters regardless of the applicability of a financially-backed service level agreement.

https://cloud.google.com/kubernetes-engine/sla

Interesting that they're now walking back from this...

I mostly try not to be too Google-focused here, but I have to say...

I'm pretty proud of GKE, and I think it offers a lot of value other than just being cheap. Managing clusters is not always easy. GKE handles all of that for you - including integrations, qualifications, upgrades, and patching clusters transparently BEFORE public security disclosures happen.

We have a large team of people who deal with making GKE the industry-leading Kubernetes experience that it is. They are on-call and active in every stage of the GKE product lifecycle, adding value that you maybe can't see every day, but I promise you is there. When things go sideways, there isn't a better team on the planet to field the tickets.

I don't understand the anger here - you're literally saying you'd rather pay more for a service of lower quality because... why? Because they will continue to charge you more? Does not compute.

For those people who use a large numbers of small clusters, I understand this may make you reconsider how you operate. As a Kubernetes maintainer, I WANT to say that a smaller number of larger clusters is generally a better answer. I know it's not always true, but I want to help make it true. GKE goes beyond pure k8s here, too. Things like NodePools and sandboxes give you even more robust control

GKE is the best managed Kubernetes you can get. And we're always making it better. Those clusters actually DO have overhead for Google, and as we make GKE better, that overhead tends to go up. As someone NOT involved in this decision, it seems reasonable to me that things which are genuinely valuable have a price.

Also, keep in mind that a single (zonal) cluster is free, which covers a notable fraction of people using GKE.

If Google Cloud would have charged 73$ from the start (or after beta), i think there wouldn't be so much anger.

The anger comes from, a product was free and now it is not. A lot of people made architectural choices that depended on the price of 0. (You mentioned these cases in your post).

However, i believe the bigger issue is, that Google Cloud broke essentially a promise.

As a customer I need to be able to trust my cloud provider, because I am literally helpless without it.

Can I trust an entity that breaks promises ? No, I can't. I need to worry. Especially, if I cannot follow the reasoning behind it.

If it is true, that Google's overhead went up, because of improvements, then it would have better to have two kinds of clusters (better and paid, old-school and free). You would have not broken the promise. People can choose on their own pace to upgrade if they need to.

Also keep in mind, that you also carry the Google brand. Hence, if other teams of Google break promises (like f.e. Stadia) this will also reflect on the Google Cloud team. Unless you keep a crystal clear track record, i need to assume it can get worse than what you have done right now.

My conclusion is that, I will design the cloud architecture I am responsible for, such that it has minimal dependencies on Google Cloud specifics.

I'm the CTO at a very small company. All our stuff is running on GKE. Our monthly bill tends is less than $10,000/mo. We're currently in the process of splitting our stack into separate projects and clusters, because co-locating projects in a single cluster has gotten messy. We'll probably end up with 4-5. That will increase our bill by $292/mo, worst case, assuming the first cluster is free. For a company our size, it's not a huge expense. But these things add up.

Since moving from DigitalOcean, our Google Cloud setup has more than doubled our monthly bill. We're paying for more compute, but certainly not twice the amount, as we've only gone from 14-15 nodes to around 20; it's just more expensive across the board, both node cost and ingress/egress. We're even cost-cutting by using self-hosted services instead of Google's; for example, we use Postgres instead of Cloud SQL. I ran the numbers earlier today; the equivalent on Cloud SQL would be 3.4 times more expensive.

In short, Google Cloud is expensive, and it's not like the bill is getting smaller over time.

Developments like these factor in my choice of cloud provider for future projects.

This response, right here, is everything you need to understand about why Google Cloud is failing to sell to the enterprise market.

The enterprise market only really cares about one thing: rock solid stability. It doesn't care about features, and it doesn't (really) care about price. It wants a product that it can forget is there.

What's really sad is, technically, GKE is that product. It just works. It is solid. You do get to forget that it's there. Until you get a random email telling you that you get to explain to your boss that your bill is going up next month and your project might end up running over budget as a result.

If you can understand why a large segment of the market prefers to pay a higher but stable charge over a lower but undependable charge, then you can understand why Google Cloud is failing at selling to enterprise.

It's not the fee itself, it's the worry that GKE will do what Google Maps did and massively increase fees with very little notice, causing people to scramble to migrate.

Google has a really bad reputation right now when it comes to cancelling projects that people have built their businesses upon, or jacking up fees quickly. The $73 is irrelevant on its own - the issue is (a lack of) customer trust.

I encourage everyone to always stay nimble and keep your eyes on portability. I also encourage you to try to assess the REAL costs of doing things yourself. It's rarely as cheap as you think it is.

As a Kubernetes maintainer, I am fanatical about portability.

As part of the GKE team, I think we provide tremendous value that people tend to under-estimate.

NOTE: I was NOT involved in this decision, but I understand it, and I want to help other people understand it.

Thanks for the props. It means a lot to me personally.

I'm not trying to nitpick here, but that justification is awful. It goes against reliability engineering on a deeply fundamental level, pretty much guarantees to make already not that reliable things even less reliable. Generally the more isolated entities you have and the smaller they are the less they affect each other and the environment when something bad happens, the faster they can be recovered, the fewer end users they affect, etc. If I remember correctly, this is even how some Kubernetes people justified ideas behind Kubernetes itself that you want to drop now.

Everything is a tradeoff. If you want total isolation, you pay for it. If you don't want to pay for it, you make more value-based tradeoffs.

Concretely, Google runs "a handful" of "pretty reliable" services on a relatively small number of clusters.

250 * 0.1 * 1000 = $25,000/month.

This is quite the price hike for something that was a) free until now. b) Not a service that warrants such a fee given that it uses existing (GCE) resources and can frankly be done manually by one of the DevOps engineers for a few hours/month and some scripting. It's just a charge for convenience it seems.

If you guys find it hard to maintain/upgrade clusters, that's your business. All I am saying is that as a company giving you business, with this change, you are now no longer the cheapest, most reliable or most convenient. As a result, we will be moving to provision instances with other providers from now on.

> Google's inability to actually commit to long term support...

This is _exactly_ what Google is doing in this case. We are providing an SLA - a legal agreement of availability and support. These changes introduce a guaranteed availability of the management control plane.

As someone else noted, it breaks a lot of recommended architectures where you would have auto provisioning and a lot of clusters to separate concerns and keep costs down.

Finally, the pricing changes are starting to look like a pattern, every time Google deems the usage of a product is good enough, they will increase the price.

They are the Ryanair of the cloud.

Edit 1: moreover, it will increase the cost of composer, and on top of that, the recommended pattern where composer is paired with a kubernetes cluster for executing the workloads

Isn't Ryanair literally the Ryanair of the cloud(s)?

And having it opt-in will save face with those users of GKE where an additional $73/m is significant.

On a hopefully more constructive note, if this is the way it's going to be from now on, I would at least expect to see an exemption on such a management fee/SLA on preemptible nodes - having an SLA and management fee on the cluster whereby nodes can be killed in a 30 second window without prior warning seems to be a little more than pointless.

Do I still have to pay the bill first, fill out forms, get account managers involved, at some point receive a partial credit, and repeat this until the delta between what I was expected as the SLA credit and what I got as the SLA credit is less than the cost of the time to fight for another cycle?

We were thinking of using Stackdriver for logging, but we were scared of vendor locked in due to price increases or other changes that we've been warned about with Google. In this case, I think it's safe to say we'll be using Prometheus + Grafana + Loki instead since there may be a random Stackdriver flat fee introduced or some other weird fee and we may need to migrate out of Google.

Here is a very good tutorial on how to set up your own cluster: https://community.hetzner.com/tutorials/install-kubernetes-c...

[1] https://emvi.com/

- EKS: $0.10/hour/cluster

- Digital Ocean: Free (only charges for the nodes)

- Azure: Free (only charges for the nodes)

In the long run we'll probably try and build our stack on vendor-agnostic tools..

- Rancher - https://rancher.com/products/rancher/

- Infra.app - https://infra.app (mentioned a few weeks back on the Kubernetes podcast)

- Prometheus https://prometheus.io/ - metrics

The cloud providers all include their own tooling (logging, monitoring) built-in but I'm worried this will only lock us on to further price increases.. has anyone found a good vendor-neutral logging system? We don't really want to use ELK stack right now since it's really heavy and costly to run...

The sentiment with my clients before was that Google Cloud was a great choice because of the security and expertise with GKE. It's also free!

Meanwhile, in the back of my head I've always had this fear because of your reputation that you do not keep your promises and that you do not care about your users. Because of this fear, we have tried to make every infrastructure decision not use a managed service by Google even though it may be easier to do so short-term.

For the product I'm working on, we decided to use Kubernetes just in case you baited and switched us with the reputation you have. In terms of monitoring, we really wanted to use Stackdriver, but now we're 100% using fluent-bit + prometheus + loki + grafana. It's the only way to protect ourselves from your reputation which is becoming a reality.

So yeah, this is pretty sad and a bad decision. Should have priced GKE at $70 / month to begin with and we would have been fine with it. Now we're (actually) looking at EKS since Amazon doesn't seem to have this reputation and you've spooked us. We never would have thought about using any other provider until today.

I think GKE provides tons of value, but people tend to under-estmate that. In order to keep providing that value, we need to make sure it is sustainable.

I'm really, truly sad that you perceive it as bait-and-switch, but I disagree with that characterization. If you want to move off GKE, I'll go out of my way to help you, but I urge you to take a big-picture look at the TCO.

AKS, well, I don't have any insight into their business, but I have my suspicions.

G Cloud is already unreasonably expensive and nearly impossible to price manage. It's cool to see them double-down on that.

https://news.ycombinator.com/item?id=20064169 https://news.ycombinator.com/item?id=17225665

It’s a race to the bottom on price so this doesn’t surprise me. They chose this life.

If they delete the droplet the second a single CC payment fails, that's one thing, but I don't believe that's how their system works.

In both of these instances I am referring to a balance of less than $20.

So for less than $20 (a few weeks late) DO says, welp fuck this customer we are going to terminate all of their resources immediately.

This is what DO and others need to do: Put it in your terms that you will keep racking up charges and then send it to collections. Charge interest, charge fees, do whatever you want. Turn $20 into $40. Why? Because businesses do not give a shit... if it is between losing everything or a slap on the wrist (monetary fee) they will chose the latter every time.

One of my clients had to painstakingly trudge through archive.net to recreate their missing blog posts. How fucking miserable is that? Over a few hundred megabytes of disk that DO could have kept around...

Also, actally make an effort to reach out before doing anything serious. Call phone numbers, email other members on the team to alert them to the issue, etc...

Too many times I have seen some script kiddie throw together a client's WP site and toss it on DO because it is 'so cheap and cool' and yet they forget about everything else: backups, security, managing the box, etc... and inevitably shit will hit the fan.

I was really rootin' for DO in the beginning. I even applied to work there when they were first starting out but did not want to relo to NY. Now I am moving three clients OFF of DO because they are all very unhappy with the level (or lack) of service they've received.

In my experience, AWS is a very expensive cloud with clunky UI and big brand name. During consulting gigs, I have seen many customers want to go with AWS only because of brand. And later they cry when bills start to hit roof with vendor lock in.

====================== First email from AWS (several days after my request): ======================

Thank you for submitting your Limi Increase request.

I'm contacting your to inform you that we've received your Workspaces Application Manager - Total Products limit increase request, for a max of 5 in the Oregon region. I will be more than happy to submit this request on your behalf.

Please note that for a limit increase of this type, I will need to collaborate with our Service team to get approval. This process can take some time as the Service team must review your request first in order to proceed with the approval. This is to ensure that we can meet your needs while keeping existing infrastructure safe.

You may rest assured I will push towards expediting your request to be addressed as soon as possible. As soon as the Service team contacts me I will definitely let you know by email.

In the meantime, please feel free to let me know if you have any additional questions or concerns and I'll be happy to help!

I appreciate your patience while we evaluate your request.

====================== Second email: ======================

Thank you for your kind patience whiIe we continue to evaluate your Workspaces Application Manager - Total Products limit increase request.

I apologize for the time is taking to provide you with a resolution as we've always aimed to provide our customers with a rewarding experience that meets and goes beyond expectations. Unfortunately, from time to time there are cases where the final outcome is handled by another department and the time they take is completely out of our hands.

We certainly understand the sense of urgency that you have for this particular request and therefore, we have spent time communicating with the service team to let them know about it. Rest assured that your case is active, being looked into and the sense of priority has been transferred. As soon as we have an update from their end we'll be touching base with you immediately.

I am committed ensuring that you will get the help that you need as fast as possible, so we can ensure everything is being handled to your satisfaction, please feel free to let us know if you have any further questions or concerns through this case, so we can address them as soon as possible.

============= My response: =============

You can go ahead and cancel my request -- I've decided to not go forward with my project.

============= Their reply: =============

Greetings from Amazon Web Services.

We're sorry. You've written to an address that cannot accept incoming e-mail.

If you need to contact us, please visit http://www.aws.amazon.com/contact-us .

Thank you for your business.

I run a Kubernetes cluster on Vultr and I haven't had any problems.

There's no way I'm going to pay an extra $73/mo -- I already pay for the computing resources, this should be free.

Looks like I'll be moving away from GKE. It's a shame, I _was_ a big advocate.

You could have grandfathered in current deployments but - nope. In the tech world this is up there with killing Google Reader.

https://www.forbes.com/sites/janakirammsv/2017/11/29/google-...

YMMV, but I think the value prop of using a managed GKE cluster vs the raw costs and engineering time to run your own control plane is still strong.

For hobby projects nothing will change.

Flat fee is gonna suck for people running a lot of clusters. I bet there are some people out there spinning up a cluster per x who are going to be real unhappy about this.

We have a couple of dozen clusters, two per client, and can't change the architecture. We use helm and terraform and can build new clusters quickly but we can't treat them entirely like cattle because we don't own all the DNS. Our clients are not the sort to do things quickly - or even slowly.

Does anybody have any good and up to date resources comparing the current options for K8 providers? I'd like to get a feel for what it would take to switch.

Thanks for being the recipient of everyone's (justifiable) frustrations. They probably don't pay you enough.

I think, what is especially frustrating about this, is that we do already pay for resources that are provisioned by our K8S clusters. We pay for the network traffic, the storage, the compute. I saw you mention StackDriver... we pay for that as well.

I can appreciate that actually setting up and managing GKE backplanes is a non-trivial expense, but I generally assumed that that cost was amortized out, just like I don't pay for the backplane that runs GCE and the rest of GCP's service suite.

I also appreciate that you mention some customers are perhaps taking advantage of this "free" resource. But, isn't that quotas are for?

Frankly, more concerning than the fact that now I have a new $73/mo. fee attached to my account (which, is not the end of the world) is that this really comes out of left field, and in the context of concerns about the the nature of GCP's new leadership, and reports of Google leadership debating GCP as a going concern. I realize a lot of that isn't well founded, but it's surprises like this one that keep that narrative alive. AWS ain't no saint, but they are pretty consistently who they are: not full of bad surprises.

This just leaves a bad taste in the mouth, and makes me wonder if I can expect other surprising cost increases, or perhaps, if these don't "work", worse surprises like deprecation notices. Is this the precursor to you all discontinuing GKE because, as the DevRel class likes to tweet, nobody should be using Kubernetes if they can use (more expensive) services like Cloud Run?

Are we about to get Oracled?

Can confirm :)

> ...we do already pay for resources that are provisioned by our K8S clusters

Customers are charged for worker nodes, but until this point, the control plane ("master") nodes have been free. In addition to the raw compute costs for those nodes, there's the SRE overhead for managing, upgrading, and securing them.

> ...but I generally assumed that that cost was amortized out

<googlehat>I'm not really sure.</googlehat> <civilian>My guess would be that, initially, this was the case. However, over time, people have created many zero-node clusters. Now the amortization isn't. Again, pure speculation.</civilian>

> But, isn't that quotas are for?

See my comment above about zero-node clusters.

> I have a new $73/mo. fee attached to my account (which, is not the end of the world) is that this really comes out of left field...

Acknowledge, but I do want to highlight that changes take place a few months from now (June 2020), not immediately. Furthermore, each billing account gets one zonal cluster with no management fee.

> Is this the precursor to you all discontinuing GKE because, as the DevRel class likes to tweet, nobody should be using Kubernetes if they can use (more expensive) services like Cloud Run?

100% no. Also, Cloud Run is almost always cheaper than running a Kubernetes cluster.

> Are we about to get Oracled?

I'm not sure what you mean by that verb.

> In addition to the raw compute costs for those nodes, there's the SRE overhead for managing, upgrading, and securing them.

By that logic, can we expect to see charges for GCP Projects and the GCP Console? Cloud IAM?

> people have created many zero-node clusters

I'd be really curious what is driving folks to do that. Are they using the backplane for CRDs and custom controllers and no compute?

This feels like it could be addressed similar to alpha clusters, or with a quota, e.g.: clusters with 0 nodes for > 24 hours will be terminated?

Separately, It seems like handing everyone 3 months to figure out what to do about a new $73 * X fee isn't the best plan. Including some kind of estimate in the emails that were sent out would have been helpful. There was a change in pricing for StackDriver a while back that did this. It was very helpful to understand how we would be impacted.

> Furthermore, each billing account gets one zonal cluster with no management fee.

My feedback is that you would probably get getting way less blowback if that free-tier didn't come across as inadequate. I can appreciate that there are use-cases where it makes sense for you all to be charging. But one zonal cluster... It makes the whole thing feel punitive.

> I'm not sure what you mean by that verb.

I have a feeling we're all about to go on a journey of discovery together.

I was one of those people. I got an email from Google this morning, and thought "that's weird. I didn't even know I had a Kubernetes cluster." I think I created it years ago to work through a Kubernetes tutorial and, since it was free, I never bothered to delete it.

So, I can imagine this being a problem. Though it seems like having a minimum hourly charge per cluster would have been a better way to handle this (i.e. if your cluster is using less than $0.10/hr in resources, you get charged the difference). On the other hand, you'd probably still have people complaining about it because, well, people love to complain.

To your point about running your own K8S cluster - two things:

1. That's something you have always been (and still are) entitled to do.

2. Having personally run large-scale K8S clusters, the challenge isn't provisioning, it's maintenance, security patches, upgrades, etc.

You guys roll out a free service. You tell your sales people to hype it up as a benefit over other providers. You somehow don't anticipate that some users will "abuse" the free service, so you hike up rates for everyone?

Sorry, I don't think you're likely to find much empathy on this one.

I believe the community is keen to engage with you on this based on the comments in this thread. If your team would like to talk to a disappointment (very small but hoping to grow) customer I'd be happy to jump on a call. I hope others here would be happy to do the same.

Edit: Additional comments: You could limit the number of nodes in the always-free-tier clusters. Above n nodes the free tier clusters aren't free.

With the new pricing, I can't choose to use GKE instead of app engine/cloud run and get the same availability without having to pay for both the nodes and the new control plane cost. Those managed products run over multiple zones in a region. It's disappointing that even just one multi-zone cluster is charged.

With the new pricing, I can't choose to use GKE instead of app engine/cloudrun and get the same availability (by this I mean multiple zones) without having to pay for both the nodes and the new control plane cost. Those managed products run over multiple zones in a region. It's disappointing that even just one multi-zone cluster is charged. I'd be very happy to see you include at least a single multi-zone cluster control plane in the free tier.

To be clear, it's not my team. I'm relaying feedback, but I can't make any guarantees or promises.

All this feedback is super valid and important, and it's being synthesized to the product team.

Seems weird, given that GKE is basically the main reason people seem to use Google Cloud. These kinds of users aren't big fish, but I suspect a lot of them are going to run.

I am part of a small company which has separated our deployment into a number of sub projects, some of which are: dev, staging, production, ci, etc.

The difference for us will be several hundred dollars per month, and that will make an actual (negative) difference for us. We didn't need a "financially backed SLA" before and we don't need it now.

You asked for a question and here it is: Why isn't a financially backed SLA a part of a billing negotiation? I mean, there are some really cool features in "Anthos" but I am not picking up a phone to find out how much that is going to cost.

If a really useful feature like "Cloud Run for GKE" is awkwardly placed in the "Anthos" box, then why isn't the SLA part of "Anthos" too?

Free clusters was a huge part of why we selected GCP. If this SLA nonsense isn't made optional, our next project is not landing on GCP.

I'd like to share some feedback that echoes that of other commentators, from a different perspective.

I run a local cloud developer community with regional pull for attendees, as well as working directly with local early-stage startups looking to become cloud-native.

GCP has always been my go-to for recommendation for our attendees (mix of developers and technical founders, and some enterprise technology folks) given the affordability factor, pathways to additional credits to flesh out ideas, learn new technologies, or stretch the limited runway of their new organization, and ultimately my belief that GCP is one of, if not the best, clouds for developers given the investment in documentation and engagement DevRel channels.

With the rollback of open-enrollment into a smaller plan of Google Cloud for Startups, and price changes like this, I'm fearing I've chosen the wrong hill to die on when talking with these new customers.

I appreciate the inclusion of a free regional cluster per account, which will still afford myself the opportunity to demonstrate k8s at meetups and to end users without taking more of an out-of-pocket hit, and for folks to learn on their own or maintain hobbyist projects on the same budgets they are accustomed to.

My fear with this announcement is that the negative repercussions of this will not be felt on the bottom line or figures that it seems more and more is the priority of the Google Cloud leaders. Rather, it will be felt hardest by the smaller customers; the hobbyist developer or technical co-founder looking to learn new technologies, to scale up their operations, and who at least in my experience, are driving growth in the mindspace around GCP in their communities.

Put another way, moves like this will further tarnish the reputation of Google for those who the sales engineers have for the last two years, promoted heavily no cluster management fees like "the other guys," and in the eyes of many starting out in these areas (of which I recognize most will never become the big customers that satisfy the requirements of executives).

I hope that when the dust settles, this does not lead to a retraction of what makes Google Cloud great in my mind, which is specifically to developer experience and outreach.

With that in said, I would suggest really driving home this change through dismissable in-console communication at the point of cluster creation, on the dashboard, and in email communication to folks who this will impact, with a clear picture of the impact on them. No one wants another large disruption of thousands of small organizations and users, as was the case with the Google Maps pricing change.

Personally, I'd love to see it increased to one free regional or zonal cluster per account for the remainder of 2020, and then making only one zonal free per account effective 2021. Given the uncertainty around engineer capacity and scheduling given the ongoing human malware crisis affecting companies large and small, I think this could be a good middle ground to satisfy most customers affected by these changes, while still achieving the objective of moving this away from being a loss leader of sorts.

I know you are just the messenger here, and I send my sincere sympathies that you have to work with a product manager there that can't compute strategic impact of this change :)

I have now lost faith in you, GCP.

I hope GCP re-thinks this.

For a business I prefer a company that starts with higher prices and then only lowers them to one that may increase them at any time.

AWS lowering compute costs is fairly largely shared, but I am curious if anyone has compiled a list of the cloud providers (AWS, Azure, and GCC) increasing the costs of services.

Makes me rethink whether I want to do any business with Google anymore.

Replies on another thread by alleged Google employees appear to indicate they are learning on the fly that doing support matters and costs money.

There are a lot of comments from people wanting free things. You may get those things for a while but it won't last. Free only works for so long in anything no matter how hard that is for some people to figure out across all aspects of life.

The real issue comes into play when you try to make developer environments.

To give our developers any semblance of a "real production-like" workload, they need to work with an entire kubernetes cluster - maybe even a couple - to simulate what's happening in production.

This means at any given time, we have hundreds of GKE clusters because each developer needs a place to try things. Yes, these are ephemeral and can be tossed aside, and yes they cost a tiny bit in VM prices, but adding a per-cluster management fee is going to skyrocket this expense and push us towards trying to figure out ways to share these clusters between developers, which defeats the entire purpose of the project.

We'll have to seriously consider abandoning GKE for this use-case now and that sucks, because it's by far the fastest managed k8s solution we've found so far.

There are reasons to have multiple clusters, but I think the current trend takes that too far.

TO BE SURE - there's more work to do in k8s and in GKE.

>A single-zone cluster has a single control plane (master) running in one zone. This control plane manages workloads on nodes running in the same zone.

https://cloud.google.com/kubernetes-engine/docs/concepts/typ...