TCP Sucks
snell-pym.org.ukTCP pretty obviously doesn’t suck as it’s pretty much powered the internet for the last 30ish (maybe more) years. I’m sure it isn’t perfect for every purpose and maybe I can be improved but enough with the click bait.
Coal power obviously doesn't suck as its pretty much powered the planet for the last 100 years.
It's perfectly possible for something to both be incredibly popular and to suck. An appeal to popularity is certainly not better than a click-bait headline.
I'd argue that most of the stuff we use sucks in some ways as well; suck isn't an absolute term. TCP sucks in very specific ways the article talks about.
It's not just popularity; it's an adaptable enough design to work in the range from 150 baud acoustic modem links to 40 Gbit fiber optics. Not many designs can withstand that, no matter how hard people try.
I'd suggest an update to your lower bound - avian carrier
To my knowledge that has only ever been deployed to send ICMP messages, not TCP. With a TCP stack tuned to give more lenient timeouts it might work but you'll need a lot more trips to do a whole connection setup and teardown.
Yeah, TCP is the worst protocol except for all the others.
I always liked SCTP more than anything. I would like QUIC if it wasn't for mandatory TLS.
Complaining about the article’s title is no substitute for replying to the article’s content. Writing a provocative title is how you get people to read your article. If you have an opinion to share, you shouldn’t try to disguise it as a drab technical report. The article makes sound points and the title makes sense.
Compare “TCP Sucks” (easy to read—communicates, concisely, that this is an opinion piece, and what the opinion is)
or “Limitations of TCP” (misrepresents the article as informational)
The author says "TCP sucks", and is promoting his own protocol. Iridium. Which he hasn't even designed yet. That's a bit much. We don't get to criticize his approach.
(Also, if you're designing a communications system, don't give it the same name as another communications system.)
TCP has its limitations, of course. It was designed to work over a wide range of connections, including dial-up, and it does. It's suboptimal for broadband server to client connections where big server farms from a small number of vendors dominate. Hence QUIC and HTTP/2/3. Still, those don't provide a huge improvement in performance.[1] Even Google merely claims "on average, QUIC reduces Google search latency by 8% and 3.5% for desktop and mobile users respectively, and reduces video rebuffer time by 18% for desktop and 15.3% for mobile users." That's marginal. An ad blocker probably has more effect.
The author is worried about the overhead of the three-way handshake, but the overhead of setting up TLS is far worse.
[1] https://conferences.sigcomm.org/imc/2017/papers/imc17-final3...
> The author says "TCP sucks", and is promoting his own protocol. Iridium.
Well that won't be confusing at all:
* https://en.wikipedia.org/wiki/Iridium_satellite_constellatio...
> The author is worried about the overhead of the three-way handshake, but the overhead of setting up TLS is far worse.
Oh, "TLS sucks" is a whole separate blog post for another day ;-)
If the title had been “Limitations of TCP for X” I would be more likely to read it than with its current title. For the simple reason that 9 out of 10 times, when people say “TCP sucks” they ignore all the things we ask TCP to do. Not least of which is “try not to break the Internet”.
I’ve tried to design stream transports on top of UDP. It is doable if the scope is narrow and you actually understand a bit of what went into other protocols (like TCP). But it isn’t easy.
There is an unwritten rule that it is allowed to comment on clickbait articles without reading them.
There is a hitherto-unwritten rule that it is allowed to comment...
TCP is fine. There is some questionable things here and I'll just touch on a couple of them.
1) TCP checksums are a bit weak.
The author claims that we don't know how many errors are not caught by TCP's CRC and while that's true on some level the probability of bit errors is driven by properties of the physical medium over which the packet is traveling, not the transport layer protocol in use. That's why most, if not all, physical layers that do have a significant chance of bit errors have FEC built-in at the physical layer.
As a point of reference, I pulled stats on one of our servers and see about 10k bad packets out of 1.3 billion. If we assume that bit errors caused all 10k, we get an error rate of about %0.00076. Since we need two independent bit errors to trigger a state where the CRC might not catch the error, we can calculate that on this particular server we would expect .076 packets that might be corrupt in a way in which it's even possible for the existing checksum would miss it (and it probably would still be caught). If that is an unacceptable error rate, you should absolutely be using some way to verify message integrity at a higher layer. Still, for most applications, a fast hash that lets a bad packet through every couple of years is a good trade-off.
2) TCP's over-eager in-order delivery
The author talks about getting packets 1,3,2 in that order in the context of a file transfer, and in that specific scenario, it might be marginally preferable. Far more likely than out of order packets are lost packets and lost packets may never come if the connection gets closed/reset before its retransmitted. For the vast majority of applications having a partially transferred entity isn't helpful, but not having to deal with the complexity of out of order and partial transfers is.
File-transfer is a bad example. A good example would me multiplexed REQ-RES protocol. There is no reason for Message #2 to wait on message #1.
This issue, however, is so overblown by the author. It's not that often earlier packages delayed and new packages arrive faster and when that happens often you problem is usually much greater than server waiting for 101-200 before sending 201-300 to application...
To me the most annoying part of TCP are:
- Slow Start - Single channel - Single Mode
Everyone life would be so much better if within a single established connection you could open multiple channels in multiple modes like framed vs stream. Each channel could have its own order with congestion control overseeing entire bunch of channels.
You can build something similar on top TCP today, but it's not the same. Best part of TCP is that it's a standard available everywhere without conflicting implementations.
It's weird that the article doesn't mention SACK which is pretty much supported everywhere these days and reduces the head-of-line blocking problem significantly.
I was surprised to see no mention of QUIC!
How can there be so much TCP bashing, without a mention of SCTP. I mean at least that's implemented in some BSDs with ports available for other OSes.
Sadly the state of the internet means that only TCP and UDP will be widely used for the foreseeable future. There are a quantity of old routers which just throw away protocols they don't know, and they don't know much. See https://en.wikipedia.org/wiki/Protocol_ossification
SCTP is deployed more widely than you think, all mobile network(3G/LTE/5G) use SCTP for varies signaling protocols like S1-MME
Yes but no. It is used in operator networks but that isn't on the Internet.
I think you might be mistaken on the flexibility of the infrastructure.
I personally don't think IP (tcp and udp included) will be used on Internet 2.0. Theres just too much baggage with those protocols.
Perhaps we'll see Internet 1.5 leverage encapsulation and build a huge underlying canvas for new protocols to blossom, but we seem to be approaching the point where anything with an IP address is too much of a hassle to maintain and a longterm security liability/commitment.
Just moving IP to the borders of a network would open up space for more secure protocols.
Its probably time we put the middle one behind port 79 anyways.
SCTP also lives on in the WebRTC spec for data channels, so all (modern) browsers also implement it
It sounds like the author wants a protocol that allows for arbitrarily-sized datagrams, with the ability to interleave the transmission of those datagrams across a some shared master connection.
From my relatively-inexperienced perspective, notwithstanding the widespread protocol ossification of Internet network infrastructure, the underlying MTU for each connection seems to be the primary constraint that any viable alternative must deal with effectively. This article did not seem to discuss MTU considerations at all. It's hard to take a protocol argument seriously that neglects to deal with such a fundamental constraint of network infrastructure.
I still think that NDN [0], Named Data Networking, is the proper successor to TCP/IP. TCP is oriented around streams, but NDN is oriented around blobs of data and names which represent them; as such, NDN is much closer to the application-level concerns that we care about.
The main problem is that namespacing is an extremely political process in any system, including NDN, and so we cannot have nice things.
Are there ways to try NDN-like networks today? I spend most of my time building P2P stuff and the "how do I connect my computer to yours" is consistently a major pain point. STUN/TURN can help, but there are so many edge-cases that hurt my brain to think about.
I think NDN is largely orthogonal to the problems presented in the article.
Agreed; any NDN implementation would still need a handshake, flow control, in-order delivery, etc.
TCP is only one of the transport layer protocols. There are 17 transport layer protocols listed in [1], including TCP.
It was never designed to be the all applicable universal transport protocol either. Or it was, but sane people realize that time window when TCP could be deemed universal is very short.
The 4,5,7 -- depending on who you ask --layer model is nothing but pseudoscience.
Let's start with the fact that TCP obviously has problems. People who wrote the protocol did not expect for it to live that long. At the time the life expectancy of any standard was pretty short.
But... before you get too caught up with bashing... if I were you I would spend just a tiny bit trying to really understand why we are in this situation.
It might be because it is "just enough" to build upon.
You are free to use bare UDP or even bare IP for your application if you are masochist or have spare budget to allocate for fun NIH projects.
For some reason none of these projects get traction.
Depends on what you mean by “traction”. You and I may sit at home with our gigabit internet connections and think that the limitations of TCP are a minor annoyance at best. However, at the internet factories, where internet is made, in large data centers spanning the globe, TCP’s flow control translates directly to poor utilization of expensive, high-bandwidth links between different data centers. The problem gets worse as the links get bigger. The utilization left on the table is easily enough to pay for the salaries of a few world-class network engineers and kernel programmers.
As an analogy, consider high power factor power supplies. Nobody is gonna care, at home, what the power factor of their PSUs are. However, a poor power factor at a large scale (electrical grid) translates to millions of dollars of unused current capacity. The money left on the table was so large that PFC is everywhere these days. The same thing will happen with TCP's replacement. Just give it time.
That, again, is static view.
Game-theory view is that any organization that would like to push this would have to spend enormous amount of resources to reinvent almost anything that has anything to do with TCP. Remember, it is implemented in hardware in many different types of devices, stacks, applications, it permeates almost anything. The application I am working on right now which has lived for over a decade and will live for another has TCP artifacts all over it. Who's going to want to fix it when it gains maybe a tiny bit of additional performance?
There aren't very many applications that have TCP as their single biggest performance problem with best ROI. Almost every is integrated with bunch more other applications over TCP causing chicken-egg problem.
Maybe Google could do that? IDK. They would maybe do it after they have reinvented almost everything else in their DC infrastructure.
It sounds like you are evaluating the idea that protocol X will replace TCP, but I don’t think anyone is making that argument. Protocol X would be used alongside TCP, and you wouldn’t be able to use protocol X in all networks. This is already the world we live in.
Then I don't understand what your argument is. There is already a bunch of protocols you can use if you don't like TCP.
I’m saying that these protocols are getting traction.
> in large data centers spanning the globe, TCP’s flow control translates directly to poor utilization of expensive, high-bandwidth links between different data centers.
For internal traffic they have the option to choose custom congestion controllers that don't care about being fair to Reno and can also tolerate a few packet losses. Linux has offered options here for a long time.
Some of the articles point have nothing to do with the age of TCP, some do, if I copy just two points: 1) TCP makes applications build their own messaging protocols on top 2) The three-way handshake (and connection shutdown) is slow and wasteful for short connection (1) is a big issue but has nothing to do with the age of TCP whereas (2) is related to TCP's age: using big SYN cookies with the underpowered hardware of the time..
Unfortunately we're stuck with TCP, UDP, and nothing else if you want to build on IP, thanks to protocol ossification. That's one of the main reasons why QUIC built on UDP rather than directly on IP (and why they didn't just use SCTP).
Standards always generate debate. They almost always can be improved. But hey, they are usually good enough for most use-cases. The price we end up paying, though, many times without noticing, is that layer over layer of "good enough" ends up making life much harder than it needs to be. But we only change when a single layer is clearly identified as painful enough. The article is technically very interesting, but for the general discussion my main question is: what mechanisms do we have to effectively move forward with standards? Are standards committees (good) enough (when they exist)? Are we moving at the right pace?
This is a really weird article. It's dated today but seems to contain information from 2010 or so.
This is a really weird comment. It's dated today, but seems to contain words from 1300 or so.
I am a networking novice but the fact that my networking book (TCP/IP illustrated) has one chapter on UDP and several on TCP speaks (literal) volumes.
My entirely uninformed opinion is that if its possible to do everything TCP does over UDP we should do so just based on the fact that it is a simpler protocol. This means that broadly speaking I think the QUIC protocol is an improvement. (Whether or not its worth implementing the change I have no opinion on.)
Obviously this is a very naive understanding and I welcome others telling me where my simple model breaks down. Understandably the complexity is not removed and is only pushed up the protocol stack but I would argue that the higher complexity belongs at the application layer rather than the transport layer.
> My entirely uninformed opinion is that if its possible to do everything TCP does over UDP we should do so just based on the fact that it is a simpler protocol.
UDP is actually a nightmare precisely because it is so simple. There's nothing more than incoming data to your network on a particular socket. Where should your router send it after NAT? There are so many RFCs and articles on UDP NAT traversal and you'll find it's never as reliable as plain old TCP. Good luck with UDP hole punching when 2 clients pick the same port to receive data (so many games do this and if you find yourself and someone on your local network can't play this same game online at the same time this is why).
Oh and if you actually do need packet ordering you'll now be wasting time re-implementing what is essentially TCP within UDP. The major advantage of UDP is it gives the option of not caring about packet ordering which is good but only for very very specific use cases (lossy voice/video communication and games).
The article also talks about how TCP doesn't know the speed of your network and has a slow ramp up. You know what UDP does? It sends out data as fast as you tell it to and then if that's too much for the network it's simply dropped. TCP stops you sending to much.
That's just a start too.
So unsurprisingly TCP has complexity. If you do go down the path of UDP which can only ever be justified due to a very specific need you'll need all that TCP knowledge and more as you'll inevitably have to re-implement much of what TCP does.
Packets aren’t created equal. Don’t think of a network as just a medium which lets you send packets from point A to point B. It’s a reasonable simplification but it’s not accurate!
Routers read the contents of TCP packets to figure out the state of the TCP connection, and then make decisions based on the state of that connection. With UDP that doesn’t work. This makes a HUGE difference for NAT. With UDP, there’s no connection and no state, so at best a router might guess correctly using some loose heuristics, but at worst the router will just drop your UDP packets on the floor.
TCP/IP illustrated has one chapter on UDP and several on TCP
Because TCP does far more than UDP.
if it's possible to do everything TCP does over UDP we should do so just based on the fact that it is a simpler protocol
The complexity of TCP is mostly needed, so reimplementing the same complexity yourself on top of UDP doesn't really gain you anything. The overall result isn't simpler.
> My entirely uninformed opinion is that if its possible to do everything TCP does over UDP we should do so just based on the fact that it is a simpler protocol
Everything which can be done with a truck can be done with a horse wagon.
Yeah, that analogy doesn't fully work, but QUIC is no perove of UDP being the right tool, QUIC is an iteration of TCP. Looking at some shortcomings of TCP (in the view of it's creators) and trying to fix those. If it is successful the chapters in your book will be as long as those of TCP and probably even lots of the content will be similat. It will do some form of connection negotiation (even more complex than TCP as it enforces TLS, which in TCP is fully in the application layer) and flow control. Basing it on top of UDP instead of using IP directly is not due to UDP being best, but routers, firewalls and operating systems being able to handle it. A new IP packet type would need lots of work by everybody.
If you do everything over UDP then you have to reinvent a lot of what TCP brings to the table, in every single application. The most critical part is congestion control. And people are bound to reinvent or reimplement those parts poorly because most high level programming language standard libraries don't expose all the ancillary socket data you would need to get the header data (e.g. ECN flags or MTU). With all that complexity you will be back to worth several books, but this time you don't just need to be able to use it but also maintain it.
For many usecases TCP works well and you should avoid adding another dependency.
Is this article meant to be sarcastic or something?
I feel like the article cherry picks examples of TCP doing either too much or too little given specific applications. Of course we could always go by the end-to-end principle and tailor protocols from scratch each time we build an application but in a lot of cases the state handling provided by TCP is a pretty solid base to work with. Also, unrelated to the article, there seems to be some Dunning-Kruger peak of hating TCP that a lot of people pass at some point, myself included.
These kind of articles are made by ignoramuses that don't understand that TCP is an edge protocol and those profit from ossification because it allows the backbone to evolve without everyone re-implementing everything over and over again at the edges!
Over-engineering and technical debt is a bi-product of: “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
On the backbone nothing uses TCP, not even IP. It's all BGP or other proprietary formats. Edit: I'm pretty sure the backbones with billions invested in fiber across oceans are not running vanilla IP from the 70s on their hardware/cables, don't you? Or if they do then it's case closed for anything else really, so any way you turn this.
Re-implementing delivery guarantee wont solve anything better ever, TCP and HTTP/1.1 are the final protocols for the human race in this universe, get over it and start building something useful on top of them instead.
> On the backbone nothing uses TCP, not even IP. It's all BGP or other proprietary formats.
Do what? "The backbone" usually refers to the core networks internet service providers and so on use. They carry whatever higher-layer protocols (including TCP and UDP) that their users want to use. It's like saying "on the highway nobody uses cars". The two are different parts of the stack.
Also, BGP is a routing protocol (to share IP reachability information) which runs over TCP. It's not a replacement for TCP. I can't download cat pictures over BGP. Are you referring to things like MPLS? Even that isn't a TCP-level thing. It's a halfway between layer 2 (Ethernet and friends) and layer 3 (mostly IP). TCP is still on top of all of this.
> On the backbone nothing uses TCP, not even IP. It's all BGP or other proprietary formats.
BGP is not a protocol for moving data - it's a way to find the path to deliver IP packets. Nor is it proprietary :/
Sure, but I'm pretty sure the fiber cables under the oceans are not running vanilla IP/TCP/UDP.
The submarine cables are more or less like other physical mediums for networks, like WiFi or ethernet.
You could easily say that your home network “doesn’t run vanilla IP/TCP/UDP” since it’s actually running 1000BASE-T, or Ethernet.
What you do is take your IP packets and shove them into ethernet frames when you send them across your local network. Your modem will pull the IP packets out from the ethernet frame, and send them out to the internet inside something else like PPP or a different ethernet network. We call this vanilla IP/TCP/UDP, running on top of ethernet.
Submarine fiber optic cables are the same. It’s still vanilla IP/TCP/UDP.
I have worked for an NSP for 7 years. These fiber cables (and all other ones) have IPv4/IPv6 traffic going over them with Ethernet. Sometimes there's MPLS too, sometimes some VLAN headers, but still IP/Ethernet. There is not a single scenario I know of where this is not the case.
Standardized protocols are important to a network which must rely on multiple router vendors.
Okay, so let's imagine I want to send you an email using SMTP, which runs over TCP, and we have to cross this cable. TCP has a lot of state: first we need to agree on terms like window size and sequence numbers, and then you and I need to keep track of which bytes were received by each other, resend lost bytes, slow down if the other end is overwhelmed, and so on. How do you imagine the special undersea cable protocol handles this so that each of us is still presented with a consistent view of our TCP state? Are you saying that each end has a magic box which can understand all of the TCP state management and seamlessly map it onto whatever undersea cable protocol it uses, then reverse that at the other end so the recipient knows this TCP state? That would be crazy. TCP is just carried over whatever goes on at the lower layers.
"Replacing" IP is a bit more likely: large backbones typically run MPLS, which encapsulates IP and provides for a simpler label-based switching process that can be more efficient than IP. But IP isn't replaced: it's merely wrapped in something more efficient in this case. It's like a company sending mail from one office to another over its internal post instead of routing every individual letter through the public mail system.
The point of having different layers of the network stack is so that each layer has its own responsibilities that are largely separated from the other layers. As long as IP A wants to send IP B something over TCP those cables will be carrying IP and TCP over whatever protocol they run underneath.
Undersea fiber cables are only Layer 1. They might not be using Ethernet protocol, but everything above that- IP, TCP, etc is the same.
> On the backbone nothing uses TCP, not even IP. It's all BGP or other proprietary formats.
.... BGP runs over TCP.
> TCP and HTTP/1.1 are the final protocols for the human race in this universe
Right, QUIC is only used by all things Google.
BGP uses TCP transport...