email encryption

Email, one of the most pervasive and enduring technologies of computer networking, was invented in about a dozen places by dozens of people in the 1960s. It's hard to lay out a clear history of the technology because it's just so obvious—pretty much as soon as more than one person could use a computer, there was some kind of mail facility. These ranged from mainframe-centric systems where all of the users of a single computer could write messages to each other, to PC-centric systems where workstations would mount a network share to store and retrieve messages. Pretty much any scheme you can come up with for moving messages was probably in use somewhere from roughly the 1960s to the 1990s, by which time the ARPANET-derived family of email implementations had taken hold.

This form of email has a clearer heritage, to Ray Tomlinson, who came up with the core idea that addresses could identify both a user and a host, and that some kind of open protocol could be used to send messages to another host when necessary. Over time, and with many revisions, Tomlinson's design became SMTP and was joined by protocols like IMAP that built out the form of email we use today. This is a form of email that is in some ways decentralized (any user is free to choose a host) and in other ways centralized (each host assumed to be continuously online to store-and-forward messages for its users). Tomlinson's design was flexible enough that we have not had to totally get rid of it, but enough has changed about the modern Internet that we have had to take a new approach.

Email has many vexing limitations, artifacts of its age. For example, email handling should not be assumed to be "8-bit clean"—email protocols were originally defined over 7-bit ASCII and ran on many machines that used the eighth bit as a checksum. These machines were prone to changing the last bit of each byte, or otherwise mishandling email with 8-bit content. That wasn't a problem when text was completely limited to that 7-bit plane, but both Unicode and the desire to send binary files made 7-bit email unworkable. MIME was developed as a workaround, an encoding technique that solves a few problems in one go by encoding all non-ASCII content of email in the form of ASCII characters.

This is rather inelegant. MIME's ASCII encoding, whether quoted-printable style or base64 style, is annoying for humans to read and inefficient with regard to message size. As with all base64-like encodings, it counts as a suspicious smell, an indication that we are papering over past mistakes. Email is full of them. Let's consider another: security.

"Internet email," a term we can use to try to explicitly specify the SMTP-type email system inherited from ARPANET, uses a simple design that dates to an era when the operators of networked computers all knew each other. While there were other possibilities, most email was delivered directly to the computer of the recipient, and in a lot of cases the host-level security of the era wasn't very good either. The possibility that someone else could read your private email was known and there wasn't a whole lot you could do about it. The internet existed within a context of trust.

We don't extend that same trust today. Email is usually handled by third-party mail services on either end, which are not necessarily trustworthy, and to get between the two third-party mail services it has to pass through a series of internet links that are also of questionable security. Email became one of the first, probably the first, compelling application of large-scale computer networking. It may have also been the first of the field's enduring security failures. The system was designed with no concern for message confidentiality or integrity, and much of it still operates that way today.

Still, there's a long history of efforts to improve email security, and we should probably start the story with the greatest of all: not just an email protocol, but an entire stack of protocols that promised a better internet in general: the OSI suite.

X.400

The modern internet was born of an era often called the "protocol wars." When we tell the story of the internet today, we tend to simplify it to a nice linear tale in which ARPANET was created, everyone thought it was a good idea, and it proliferated. That's not totally wrong, but it leaves out a lot of the other actors. The telecom industry famously had its objections to the packet-switched model, which had originated from the computer industry rather than the communications industry and showed it. During the 1970s and 1980s, the telecom industry defined its own set of network protocols, and fundamental approach to computer networking, which came to be known as the OSI protocol suite.

Today, the relevance of the OSI protocols is limited to the frustrating insistence of modern networking textbooks on trying to explain the internet in terms of the "OSI model," a now more speculative than actual layer model that describes a different system incompatible with the internet. The reason that academia has this weird fixation on using the design documentation of a failed internet to explain the fundamentally different architecture of the other internet that succeeded... well, that itself is fallout of the protocol wars. History may be written by the victors, but the losers still get to dictate theory.

Despite my ongoing project to discourage the use of the OSI model for learning, we can indeed learn from the OSI protocols. The reasons that the TCP/IP stack succeeded over the OSI stack are many, one of them being that OSI's "design by committee" approach led to a set of protocols that are very thoroughly thought out but quite complex. This is a big difference from the internet protocols, whose design philosophy was closer to "good enough for now."

The difference in philosophy is especially stark when we look at the application-layer protocols. The "internet protocol suite" as it is conventionally defined does not include application protocols at all, itself a telling fact. The OSI suite does. Internet email started out as a special-case use of FTP, before moving over to a loose set of commands over Telnet that eventually formalized (or ossified, depending on who you ask) into SMTP.

By the time the OSI standards were published in 1984, it was known that email was one of the killer applications—the first revision of SMTP was published in 1982, a post-facto response to the popularity of email on the internet, and besides, email had been a key feature of most proprietary networking systems since the 1970s. So, OSI had a solution: X.400.

X.400 is the "Message Handling System" part of the OSI suite, and provides features which are recognizable as email but also a bit more generalized. X.400 was, in most regards, a failure. It was unsuccessful in displacing internet email in almost all environments. Still, X.400 was decidedly influential. There was a period, albeit brief, in which it was thought that major governments would require X.400 compatibility in future purchasing. This was similar, and had a similar effect, to the period where POSIX-compatibility was a government requirement: a lot of vendors designed their products to meet the requirement, the government lost interest, and it became a curious legacy. In email, this is most obvious in the form of Microsoft Exchange: Exchange started out as an X.400 implementation, mostly for government reasons, and still provides a lot of X.400 features.

X.400 is vastly more complex than internet email. For example, the encoding of X.400 messages is ASN.1, a binary serialization format that is more efficient and capable than MIME but also a lot more complicated. That X.400 has complexity stacked on top of complexity is useful in understanding why one of X.400's most compelling features didn't go very far: encryption.

If you are sending an email to someone, and you don't want anyone else to read it, the most obvious approach is end-to-end encryption. For many practical reasons, you'll want to use asymmetric encryption. You find a public key for the recipient, encrypt the message to that key, and then send it over the wire as usual. The recipient then decrypts it with their private key. Simple enough, except that every part of this process is laden with complexity.

To underscore that point, let's take a look at X.400 itself—the 1988 "Blue Book" version, which seems most appropriate to when email was ossifying.

Aspects of an asymmetric key management scheme to support the above features are provided by the directory system authentication framework, described in Recommendation X.509. The directory stores certified copies of public keys for MHS users which can be used to provide authentication and to facilitate key exchange for use in data confidentiality and data integrity mechanisms. The certificates can be read from the directory using the directory access protocol described in Recommendation X.519.

Recommendations for other types of key management schemes, including symmetric encryption, to support the security features are for further study.

End-to-end encrypted email requires two basic parts:

First, there must be a technical format for moving encrypted messages through the message transfer system. X.400 handily resolves this by the use of ASN.1 definitions that allow for encrypted messages bodies, thus making most of that someone else's problem. X.400 actually contemplated a lot more when it comes to security features, things like non-repudiation of delivery, that required more complexity in message transfer but didn't make it to the modern day. We'll just ignore those for now.

Second, there has to be a key infrastructure. You need some way of obtaining a public key for the person you want to send a message, and of being sure that it really belongs to them. This is important both for encrypting outbound email and for authenticating inbound email, since end-to-end encryption is usually deployed alongside end-to-end authentication using the same cryptographic infrastructure. In the case of X.400, that whole problem is deferred to the X.500 family of protocols.

What is X.500? It's our old friend, directory access protocol. The X.500 directory standards are the comparison by which LDAP is "lightweight," and there is a parallel story: LDAP is the scrappy internet protocol that won against the X.500 Goliath.

So the gist of X.400 email security is this: use the directory to look up the recipient's public key, and then encrypt the body of the ASN.1 message object to that key. In actuality, the details are different and more complicated, but that goes for everything OSI and the simplified sketch here is enough to get the gist. It's straightforward.

There are just two problems: ASN.1 as a format for messages did not succeed, and the concept of a global directory never materialized.

To the extent that X.400 succeeded, its security features were actually a big motivation. The ISODE Consortium, which had created something like a reference implementation of much of the OSI application suite, is still around today as Isode Ltd and still sells a message handling system. There are a few other maintained X.400 descendants as well. One of the largest customer bases is military: while less common in the US (keep in mind that internet email was originally developed under the aegis of the US military), a number of European countries adopted X.400 as their secure messaging solution and still have extensive reliance on it in military and intelligence organizations. Other enduring strongholds of X.400 include global aviation (ICAO's messaging system is X.400-based) and electronic data interchange, or EDI, a standardized system of financial and supply chain messaging between ERPs.

It is no coincidence that these are the same kinds of applications where protocols like X.25 survived unusually late (indicating a high affinity to long-running, legacy systems) and that they are contexts where the participants and messaging are governed by a central entity. That means that a directory exists, which means that the key distribution problem is vastly simplified. It also means that the software in use, both transfer agents (e.g. servers) and clients, are standardized and usually purchased from vendors that have specifically designed them to work well in an X.400 environment.

In other words, they are contexts that are very different from email, in which people exchange messages between a huge number of organizations and many users are simply on "mail providers" that are not an organization in a traditional sense. This means that there is no meaningful directory capability in the email system. Email is also used with a hugely diverse landscape of clients, imposing a steep penalty on the "complicated message encoding that requires interaction with a complicated directory system" part of the design.

Since X.400 proved a nonstarter in most of the email world, what about encryption for the rest of us? Well, this situation gets complicated and sad.

MIME

First, it's helpful to expand a bit on the involvement of MIME, the Multipurpose Internet Mail Extensions. MIME originated as a solution to the 8-bit problem with internet email, as a way to encode non-ASCII text and binary files in a consistent, reliable way. Along the way to achieving this aim, it introduces the concept of a "multipart" message which has multiple distinct objects as a body instead of just one string of text. We use this multipart feature heavily today, both for obvious applications like file attachments and for more subtle ones like HTML emails that provide a plaintext variant. I don't think it's totally wrong to summarize MIME as "email 2.0," because while it is limited in scope to the message format (not changing the transport protocols), MIME adds a lot of new functionality to email that we now consider basic requirements for a messaging system.

The history of MIME has a lot more twists and turns than you might expect, which reflects the difficulty of making major changes to a system as distributed and heterogeneous as internet email. Still, the MIME authors seem to have had a similar view that MIME was a significant evolution of email by standardizing message bodies and paving the way for future extensions to that standard, adding new functionality. This quote from RFC 1521 (1993), the first MIME standard, is informative:

STD 11, RFC 822 defines a message representation protocol which specifies considerable detail about message headers, but which leaves the message content, or message body, as flat ASCII text. This document redefines the format of message bodies to allow multi-part textual and non-textual message bodies to be represented and exchanged without loss of information. This is based on earlier work documented in RFC 934 and STD 11, RFC 1049, but extends and revises that work. Because RFC 822 said so little about message bodies, this document is largely orthogonal to (rather than a revision of) RFC 822.

In other words, existing email standards addressed how messages are conveyed between hosts, but did little to address what would actually go into those messages. MIME addresses that problem by creating a proper standard for message bodies, one that adds features while contending with the unfortunate limitations of the many pre-MIME email implementations. The many RFCs around the general topic of MIME (and other proposals for email body formats) spend a lot of time contemplating how these new message types would be handled by email clients that did not understand them. Email technology is generally easy, email standards are hard.

PGP

As these standards efforts marched on, the fast-growing field of computer cryptography had its own ideas. The first major event was cryptographer Phil Zimmerman's unceremonious 1991 release of a tool called PGP, or Pretty Good Privacy. The story of PGP is a long one, jumping from a casual release to an activist Usenet group to criminal charges to the foundation of PGP Inc to commercialize the system.

The funny thing is that PGP itself is not that important to the story of PGP. Despite the corporate ambitions of the company bearing that name, PGP started as a loosely academic or activist project, and it ended up that way in the long term as the open-source clone GnuPG (GPG) displaced PGP in the marketplace. PGP Inc became part of Symantec and faded into the obscurity that most Symantec products do. Along the way, PGP and GPG implementations rode the roller coaster that was 1990s cryptography, swerving from algorithm to algorithm and growing a complex cipher selection system along the way.

PGP was a general encryption tool and could be used for files or messages of any kind, but email was a key application. So how is a PGP-encrypted email actually conveyed? Well, it's not exactly pretty, but it aligns with the other messes we've already seen. PGP typically outputs ciphertext as either binary data or in a format called "ASCII-armored," which is base64 except with an optional CRC checksum trailer (which was deprecated decades ago). This is very similar to the PEM format widely used with modern cryptographic applications, and not by coincidence: PEM stands for Privacy-Enhanced Mail and is itself an artifact of a failed email encryption standard that produced quite a few RFCs. PGP's email application and PEM were under development just about simultaneously, and there was definitely cross-pollination of ideas between the two. PGP won, but not without taking parts of PEM along for the ride.

In practice, ASCII-armored PGP payloads are sent one of two ways: first is "inline" format in which each part of a MIME-encoded email is separately encrypted and have otherwise usual MIME types (e.g. application/octet-stream), so that non-PGP-capable email clients will treat them as opaque files that the user can save off and process with a separate tool. The second approach is the "MIME" format in which the entire MIME-encoded email is encrypted whole and dropped into a new MIME document with the type "application/pgp-encrypted." The latter has better security properties (less metadata leakage) and tends to be more user-friendly and elegant in clients that are designed with PGP in mind, but fails very badly in non-PGP-capable clients which are not even able to separate out attachments. You can still process the email with an external PGP tool by saving off the whole body, MIME-decoding, decrypting, and MIME-decoding again, but you get a little frustrated just reading that, don't you?

Today, this mess has become one of the two main criticisms of PGP. PGP is not the result of a huge, multi-party standards effort like X.400, but rather the product of a small group of people taking a pragmatic approach. Nonetheless, it grew up in the same climate and so survived with most of the same scars. From bottom to top, PGP is very complicated. Almost every part of PGP-encrypted email has multiple options, some of which are known to be insecure. Modern tools mostly follow sane defaults, but plenty of older tools remain in use. Key tools like gpg itself show their age with some of the least user-friendly interfaces ever designed. The immense difficulty faced by PGP users was the subject of seminal 1999 "usable security" paper "Why Johnny Can't Encrypt" (the paper that more or less invented the field of "usable security"), and most of the criticisms raised in that paper are still true of PGP implementations 27 years later.

And that brings us to the second aspect of PGP, key distribution. We've seen, at least in sketch form, how PGP email is conveyed, the first challenge of encrypted email. But what of the second challenge—is PGP for use with a directory system? Well, the earliest examples of PGP de facto were, because they did not address key distribution at all. Very early on, though, PGP gained a novel concept called a "web of trust." The web of trust, or WOT, is simultaneously one of computer security's most intriguing and appealing concepts and one of its most inane and stupid. I'll try to explain it succinctly.

You are Alice, and you want to send an email to Bob. The "why" is not important, these are timeless characters of cryptographic explanations and they have a timeless need to securely exchange messages. If you know Bob in person, say because you are colleagues at an institution, the key distribution problem is simple. You can perform calculations on a key, like a checksum, to produce a "fingerprint" which is unique to the key and short enough to not be completely impractical to read by eye. You, Alice, walk over to Bob's office and ask to see the fingerprint of his key. You write that down on a sticky note, and then in the future, you make sure to encrypt your emails to Bob using a public key that has that correct fingerprint. Easy as pie so far, but that doesn't scale well to situations where Bob is not within walking distance.

And so the "web" enters: Bob may not be conveniently accessible to Alice because, say, he works somewhere else, but Alice's coworker Charlie might have been at a conference with Bob a few months ago. If Charlie took down Bob's key fingerprint, he can give Alice a copy when he gets back. Since Alice knows Charlie and believes him to be trustworthy, this is a pretty good substitute for getting the fingerprint straight from the horse's mouth.

PGP's WOT model encodes this sneakernet approach to key distribution into a software implementation. A PGP user can "sign" another user's key, a way of cryptographically attesting that they believe that to be the correct key. These signatures are uploaded to public "keyservers" that allow anyone to query for a key and for the set of signatures it has received. If there are enough keys with enough signatures, you can treat the whole thing as a graph and try to find "trust paths" between Alice and Bob, even if doing so requires six degrees of separation¹. The tangled interconnection of keys and signatures verifying them forms a web: the web of trust.

In practice, the tide seems to have turned against the WOT model by the 2010s, and today it has been completely abandoned. It didn't scale as well as its proponents had hoped, usability problems with the tooling discouraged participation, and both the implementation and concept were complex enough that poor user understanding undermined the benefits. Today the WOT has fundamental problems (mostly related to the open keyservers) that have made it practically unusable, and I think it is fair to call it a failed experiment. Further extending that fairness, we should note that the WOT was surely an inspiration for the vastly simplified key verification schemes that appear in most modern end-to-end encrypted messengers.

From its messy academic origins to its transformation into an open project, PGP enjoyed a lot of prominence in academic and "cypherpunk" communities during the 1990s and 2000s, but was less common in corporate environments. The WOT model only made sense in the absence of a centrally maintained directory, and while PGP could be used with a directory it was not the path espoused by its advocates. PGP was mostly used among researchers and hobbyists, the cryptographic establishment. In the Enterprise, there was another player: S/MIME.

S/MIME

S/MIME originated with RSA Security, the company founded to commercialize the RSA encryption algorithm. RSA Security survives today as a subsidiary of teacher's pension funds, although its prominence in the industry has generally declined. Back in the 1990s, RSA was on the cutting edge, so 1996's introduction of an RSA solution for email encryption got quick attention from industry.

Over the following two years, RSA's product evolved into an IETF standard called S/MIME, for Secure MIME. S/MIME saw a series of new versions up to the present day, and is exceptionally rare in the academic and hobbyist world but fairly common in corporate and government environments, especially with Microsoft Exchange (having adopted S/MIME as its secure email option of choice after the failure of X.400 to take over inter-organization mail). Let's see how S/MIME handles the first challenge of encrypted email, encoding the message for transport.

Well, as the name suggests, MIME is involved. The simple sketch is like this: an email is prepared as a MIME document, which is then encrypted into the form of a PKCS#7 document. PKCS, the Public Key Cryptography Standards, refer to a series of open standards memos published by RSA Security. PKCS#7 describes "Cryptographic Message Syntax" or CMS, a standard container for an encrypted payload along with various metadata. In the terminology of the time, S/MIME is said to "enhance" messages with the "security services" of PKCS#7. We still use the term PKCS#7 today, mostly to refer to cryptographic certificates serialized in that format. For email, on the other hand, PKCS#7 has been superseded by an IETF standard called simply "CMS."

The CMS version of the email, which remember is an encrypted MIME document plus cryptosystem-related metadata, is then encoded as PEM (functionally Base64) and made the body of a MIME email with a type of "application/pkcs7-mime" (yes, I just said that PKCS#7 is gone from this application, but naturally the name has stayed). This is pretty much equivalent to the MIME version of PGP emails, and has basically the same upsides and downsides: it's relatively elegant and avoids a lot of weird edge cases in clients that support it, but it's quite hostile to clients without S/MIME support, which will be unable to make anything of the message. It also keeps many of the general downsides of MIME, and for example the RFC has to spend some paragraphs justifying why binary data is base64d and then converted to binary and then base64d again. Not having the extra layers of base64 involved would break interoperability scenarios with non-S/MIME tools!

Overall, S/MIME isn't that dissimilar from PGP in this regard, although I would say that it is overall simpler and better standardized. That is not to say that it is "simple" or "well-standardized," these are only relative observations. S/MIME implementations are still quite complex, and interoperability problems can appear even between organizations running the same software, due to the many configuration knobs involved. RFC 2311 includes this gem, to give some of the flavor of it:

S/MIME provides one format for enveloped-only data, several formats for signed-only data, and several formats for signed and enveloped data. Several formats are required to accommodate several environments, in particular for signed messages. The criteria for choosing among these formats are also described.

"Several formats are required to accommodate several environments" is the story of email in a nutshell... if not all of computing.

Let's consider the second aspect of the encrypted email problem, then: key distribution. Here, S/MIME takes a very X.400 style dodge. The original S/MIME standards documents, as far as I can tell, do not address key distribution in any way except for oblique and incidental references like "validating the certificate to a trust anchor." This seems odd except in the context of S/MIME's development by RSA Security and, as it became an IETF standard, Netscape—the birthplace of SSL. S/MIME was intended for use with the PKI, the Public Key Infrastructure, which has a nuanced and sometimes vague history but fell out of a combination of X.500 concepts and implementation work by companies including RSA Security and Netscape. In other words, S/MIME grew up alongside SSL/TLS, and uses the same key distribution architecture.

S/MIME emails are expected to provide the certificates that go along with the keys used (one of the reasons for the CMS container format), and those certificates are supposed to be signed by a Certificate Authority.

This is not exactly the same as saying that keys will be retrieved from a directory, but if you remember that the X.500 that gave us these concepts was itself supposed to be the directory, you can see that they are closely related ideas. In practice, they are often pretty much the same: a classic real-world application of S/MIME is in a Microsoft Active Directory environment, where the Domain Controller acts both as a directory (LDAP) and as a Certificate Authority, signing certificates for the users that populate that directory. This means that key retrieval from LDAP vs. certificate verification against the (AD) CA are pretty much the same thing, and you have the same assumptions around there being some sort of central authority to maintain the directory.

This means that S/MIME thrives in the same places that X.400 was more successful. The US military, birthplace of internet email, opts for S/MIME for encrypting it. They have a truly enormous Exchange and Active Directory environment to this end. Once those emails leave the military, the whole thing becomes a huge hassle, because properly using S/MIME requires establishing some form of federation with the military's Active Directory domain... or at least importing their CA as trusted, which is not all that easy with most mail clients.

Today

So here we have the classical email encryption landscape: there are two major options, PGP and S/MIME, each of which is unsuitable for most users for its own special reasons. Out of this frustrating stalemate, several other phenomenon have emerged.

First, we should briefly touch on "modern" email encryption, where "modern" can become a bit of a euphemism for "proprietary." There is a new set of email providers that offer some kind of built-in, managed E2E. Protonmail is the most prominent. Protonmail's dominant E2E encryption implementation is a proprietary, simplified system that relies on Protonmail as key directory and cryptographic implementation. This is not interoperable with other email providers, so I do not consider it a true form of "email encryption," but rather a proprietary encrypted messaging product offered alongside email. Protonmail does have two features for sending secure email to non-Protonmail recipients: the first is essentially a compatibility layer to PGP that allows Protonmail's encryption to get along with PGP tools. The second is a traditional compliance-driven "secure email" product where the recipient is actually sent a link to a website where they can enter a password to retrieve the message—once again, not really email but a separate product that loosely works with email.

Second, let's talk a bit about how email is really encrypted, in real reality, because there is this huge, grand-canyon-esque disconnect between the way that people discuss email encryption and what actually happens in the real world. Let's start taking that on by differentiating some terms: so far I have discussed end-to-end or E2E encryption, which is not exactly a well-defined technical term but generally means that a single encrypted payload travels unmodified and undecrypted from a human user to a human user. This can be distinguished from non-E2E encryption, where a message might be encrypted and decrypted several times, and using keys controlled by people other than the sender and recipient. We should also note the distinction between "encryption at rest," whether or not data is encrypted when stored (e.g. on disk), and "encryption in transit," or whether or not data is encrypted when sent over the network.

Email is exchanged between email servers using SMTP, and is exchanged between email servers and clients using IMAP ². Both of these protocols are now widely used over encryption, either using the STARTTLS method or "real" TLS, a distinction that I will not explain right now. While I lack solid data, it seems like a safe bet that the vast majority of real-world email is encrypted in transit, because at least the mail server logs that I have access to show the vast majority of mail servers using STARTTLS to secure their transfer connections. Client configuration is a more confusing topic but everything other than IMAP over TLS is now generally regarded as a big mistake, and modern clients should aggressively steer the user towards that route.

This is a confusing contradiction: we have had this big, long, boring discussion of how email encryption has failed to go anywhere. Then I tell you that almost all email is encrypted. What gives?

To start, remember our "in transit vs. at rest, and E2E vs. not-E2E, distinctions. The fact that the network connections between email participants are generally encrypted gives us a lot of security advantages, but it does not address the full scope of requirements that PGP and S/MIME do. Your email is not secure from your email provider, or the recipient's email provider for that matter, since email is usually not encrypted at rest and the provider has all the keys anyway. It also does not provide integrity protection, or non-repudiation, or any of a number of other useful security properties.

What's even worse is the technical reality. I pointed out that one of the big criticisms of PGP is that it is simply too complicated and provides too many options. This resulted in a litany of defects in PGP implementations over the years, and no small amount of insecure user behavior due to lack of usability. In-transit email encryption has very similar problems. STARTTLS, for example, is a fundamentally opportunistic mechanism. In its original form, an on-path attacker attempting to intercept an SMTP session can simply hijack it and suppress the STARTTLS offers from the two sides, preventing them ever negotiating a switch to encryption. This simple "downgrade attack" is a pervasive problem, and it's the main reason that I say that anything other than true TLS is a mistake. Unfortunately, STARTTLS is the most widespread option available for inter-provider email transfer.

That said, there are ways to address this problem. For example, DANE, the DNS-based scheme for TLS certificate distribution, has failed in the HTTP arena but has decent adoption (although very far from universal) in email transfer. It does a good job of providing mail servers with a way to discover that their peers are capable of encryption, without doing so over the SMTP channel itself. Even without DNSSEC to fully protect the DANE channel, this is an improvement that prevents many common attack scenarios. There is an even more popular approach that, while clumsy, avoids the dependence on DNSSEC to avoid a second on-path attack on the key channel: MTA-STS. MTA-STS is similar to DANE in that it provides an independent channel for a mail server to find if a peer supports TLS, but it uses HTTPS for that second channel, thus inheriting the protections of CA-rooted TLS.

Both DANE and MTA-STS are ways of "tipping" a mail server that a peer has STARTTLS support, at which point the mail server knows that it should only connect via STARTTLS, even if the mail server on the actual SMTP channel claims not to support it. While the extra moving parts are unfortunate, this does a lot to close the downgrade attack window, and email transferred between mail providers with DANE or MTA-STS support is really pretty well secured, at least on par with most instant messaging solutions.

This also implies that you can improve your email security a lot by just manually pinning major peers—e.g. configuring your mail server that, regardless of everything else, connections to major providers like gmail.com must be encrypted. This is actually a pretty common practice in some industries, including internal processes to maintain the list of "secure email destinations" so that it includes business partners and other frequent recipients.

In some industries, this is taken to an extreme: within the healthcare sector, for example, there are healthcare information exchanges that operate secure messaging platforms that are mostly just conventional email servers configured to refuse any non-TLS SMTP exchanges. This is sufficient to meet NIST 800-43 recommendations which is sufficient to meet the NIST security architecture for healthcare information. That might be surprising to you but it illustrates how this topic gets confusing: we all know that "email is not secure," but that's really a configuration problem with the "public" email system rather than a technical limitation. Few compliance standards actually require E2E, and securing email in transit and at rest, once you drop the E2E requirement, is so easy that it sometimes happens just by default when SMTP servers have STARTTLS support and disk encryption.

If you look at healthcare marketing materials, you'll sometimes see STARTTLS and S/MIME described as alternatives to each other. This doesn't make much technical sense, but you have to think about the context. If you have to communicate over mail servers whose configuration you do not control (or at least have insight into), you don't have a way of ensuring that your messages will be encrypted in transit unless you use E2E encryption such as S/MIME. If you are sending mail between servers that you do control, though, you can ensure encryption in transit by configuring the mail servers that way, and you do not need E2E encryption to meet that specific aim. Taken in this view, E2E encryption is almost a workaround for a defect of the design of email, and if you have taken anything from this longer than expected journey I hope it will be that: the history of encrypted email is so odd, and full of so many false starts, because it is basically the history of E2E encryption as an overlay "fix" for email's lack of the level of security provided by newer messaging solutions out of the box.

This also helps to explain the lack of adoption: it's not really that we can't figure out how to encrypt email, we do encrypt email. But as with most communications technology, the "security layer" has been pushed onto the service providers rather than the users. This has significant downsides, since it requires trust in those service providers, but one huge, conspicuous upside: if encryption is transparent to the user, people will actually use it.