This would be interesting if I could think of a common scenario on the internet where you get that combination of features.

Don't be distracted by latency/packet loss/bandwidth configurations, the main problem here is that TCP was never intended to serve the needs of soft-realtime communication. It's not relevant to deterministic lockstep, but for most realtime problems (I'm thinking of multiplayer FPS, but you can think of Skype if that's not "serious" enough for you), if you miss a packet of data, it might as well have never existed. Better to hear blips and glitches on a phone call over a shitty connection than to start adding seconds of latency.

You missed the point about deterministic lockstep as well, which is intended for scenarios where trying to transmit i.e. the entire state of a virtual army per tick is prohibitive. Designing the program to behave deterministically so that you can send only the player inputs per tick is the exact opposite of an "extremely high available bandwidth" solution.

The solution described only works when very high bandwidth is available, because it aggressively sends duplicate copies of data until acked, using on average 2-15x more bandwidth and 120x more bandwidth in the worst case (numbers taken directly from the article).

If you happened to have that kind of spare bandwidth available then yes, this is an excellent solution. When do you have high latency, packet loss, and bandwidth overprovisioned by a factor of 15?

> Audio/video streams are a completely different problem - lost data is just lost. This article is discussing a very specific problem where no data loss can be tolerated.

I know that, which is why I separated deterministic lockstep from other solutions in my comment. You're missing my point, which is that soft-realtime problems aren't suited for TCP, and cannot be addressed by any single protocol because constraints differ.

Consider what happens when many clients are using this protocol over a shared transit link and it approaches 80% utilisation, so starts to drop packets at a low rate: all the clients will start batching up larger amounts of history and sending them, which increases the utilisation, which increases the packet loss, which increases the amount of history each client sends... it's a destructive feedback loop that launches a DoS attack against the congested link. The absolute size of each individual client stream is unimportant, as we are merely assuming that the number of clients sharing a link is high enough to congest it. The last-mile user connection is not the worst problem here.

He's also written about flow control to properly handle congestion when it does happen: http://gafferongames.com/networking-for-game-programmers/rel...

When optimizing, you always profile and optimize the bottleneck first. For a lot of people in a lot of situations, latency, not bandwidth, is the bottleneck, and that's exactly what this solution solves. Compared to e.g. Standard web usage, most games are actually very small amounts of data sent, but very high time criticality, and hence very sensitive to latency.

Please do not implement protocols on the internet which respond to congestion by increasing the amount of traffic they send, if there is any chance that they might be used by a large number of people. This is poor internet citizenship and causes real problems. It means your code is a DoS attack against the internet transit links, which can be triggered by anybody capable of inducing low levels of packet loss, and the only effective mechanism to stop it is for ISPs to block the traffic entirely.

Packet loss is how the internet signals that you're sending more bandwidth than it has capacity to forward. Internet protocols need to respond politely to this signal.

If it is really desired you could implement throttling according to packet loss, but not in the way that TCP does it - by buffering and waiting - instead you'd just send packets every N frames. You can't do that if you're just using TCP since you don't know when packets are dropped.

It is really desired (and there are a bunch of ways to do it, and plenty of libraries that already implement them). Please, please implement protocols in this way, and not in the way described in the article.

For example, one of Glenn's earlier protocols kept a maximum of 32 previous message IDs, and operated at a rate of less than 30 UDP packets per second. Unacknowledged mandatory messages were given up on after 1 second of loss.

And lastly: the amount of extra bandwidth that Glenn proposes consuming in order to provide quasi-reliability on top of UDP is about as big as the TCP header. So you could consider his protocol a degenerate case of TCP that just usually doesn't send headers.

That seems far more reasonable than what is described in this article. The protocol described in this article has terrible worst-case behaviour and should not be used.

> I have good news for you. We game designers design these protocols to have a (low) maximum rate and a (low) maximum fixed length

That doesn't help at all. You can pick any arbitrarily large or small per-stream data rate - it makes no different to the outcome. Let's suppose the arbitrary internet link under consideration has a capacity of X bits/sec, and your per-stream data rate is D bits/sec under zero packet loss. The number of users that this link can support is X/D. As the number of users approaches X/D, packet loss will begin to occur due to congestion. Suddenly this protocol causes the per-stream data rate to increase towards 120D. This means that X/D users are each sending 120D bits/sec towards the link, for a total of 120*X bits/sec.

Now, recall that X is the capacity of the link, so we have just flooded it with 120 times its capacity. This link is now dropping approximately (1 - 1/119) == 99.2% of packets sent towards it.

Consider: at no point did it matter whether X or D was large or small. The fatal behaviour is that when congestion is detected, the bandwidth usage increases by a significant factor relative to the norm. The factor of this increase will control the steady-state level of packet loss that the protocol converges on, when the network becomes congested.

Now, in a fictional universe where a game protocol operating at 20 updates per second and putting down 50 bytes is causing network congestion, I happen to agree with you. That would be terrible! Never, ever, ever happens, ever, not even .0001% of the time, ever; not ever. And further, never will. But I've only been doing this for 30 years.

I would still agree that you should have a bit in the TCP stack that says "use preemptive retransmission instead of waiting so damn long". Can you actually tune the wait time for the ACKs?

I'd say that 1% packet loss is high, and the most likely reason why you'd have packet loss rates that high is because your network link is congested - in which case, sending more packets is the exact opposite of what you want to do.

High latency, high packet loss, spare available bandwidth. I can think of lots of scenarios where you have two out of three of those things. I can't think of any where you get all three, and this idea only seems useful when you have all of them.

Also, this scheme is not really asking for massively over provisioned lines; we're looking at fairly small data sets with infrequent bursts to larger sets, and an ack from the receiver knocks the size back down again.

That said, if I were implementing this, I would probably implement a sliding window for messages I send (waterfall encoding would work great here, though it is patent encumbered), and add the ability for the client to request a set of messages it missed. This would make the traffic a bit more predictable and offer the server & client the ability to make up for traffic which was lost during its send window.

I implemented opportunistic resending last year for netplay in a branch of the Dolphin emulator. The target audience was a group of people playing a fighting game (Super Smash Bros.) at a semi-high level, so it was essential to keep latency to an absolute minimum. Even a few frames of lag is enough to substantially impede high level play (might be mitigated by prediction/rewinding if I ever implement that, but since fighting games are based on very quickly responding to others' actions, there's only so good it can get); the 100ms from the article, for example, would be totally unacceptable for anything serious. Even if it were feasible to implement client-side smoothing in an emulator, it would still be much superior to actually get the real data in time.

The question is then whether packet loss is common/significant enough in practice that opportunistic resending's effects are visible in any significant way. For Dolphin, the main advantage of using UDP over TCP is actually NAT traversal, so there is no reason not to do it, but it would be nice to know whether it helped. Unfortunately, I don't have data here: I didn't include code to explicitly record packet loss statistics or anything, and I haven't had the time to actually clean up the code enough to promote it to master, so there aren't many people using it anymore. Anecdotally I implemented opportunistic resending, there was feedback that it was a noticeable improvement; there would certainly be some confirmation bias in there, but the feedback was positive enough to convince me it was likely a real effect. They didn't necessarily have to be losing a lot of packets; as the article mentions, with TCP a single packet lost makes you wait something like 2*RTT, which is certainly noticeable if your input buffer is set to the lowest possible value that produces smooth play under normal conditions.

Lets make it even harder for the TCP truthers. Hold on to your neckbeards.

Lets say that somehow sending 90 byte packets 60-times a second for 2 seconds is bad. Lets also say it would be nicer to have a longer timeout like 10 seconds. I agree.

Here's how to do it. Clamp the maximum number of inputs per-packet to 60 (1 seconds worth). If you don't receive any acks for one second something is clearly wrong. If this happens go into "congested mode". In this mode, send packets only 4 times per-second, and keep resending those 60 inputs until an ack comes back. If no ack comes back within 10 seconds, time out. If an ack comes back, go back to non-congested mode and resume sending at 60 packets per-second.

There you go. Congestion avoidance over UDP. Problem solved.

> There you go. Congestion avoidance over UDP. Problem solved.

Yes, that sounds entirely reasonable. Please do this, instead of what is described in the article.

Somewhat? Possibly one of the most overblown statements I have ever seen. Just try to picture the headline - "Halo 57 multiplayer brings whole ISPs down due to UDP design DDoS". Hilariously overblown.