Email is crazy

Email infrastructure, accurately depicted

I come across email systems routinely during my day job and often wonder how email works under the hood. This blog post is a result of my foray into that land.

Email is one of the most successful communication technologies ever built and Billions¹ of emails are sent each day. Almost half of them are spam, and a quarter of them are marketing emails. Nonetheless a big chunk is transactional emails like password resets, notifications, alerts etc. The beauty of email is that it just works despite the insane complexities that power the email infrastructure.

Email evolved organically over the years, tracing its origins to the academic labs of ARPANET in the ’70s. Its various components were designed by different people, at different times, facing different challenges. The email ecosystem is a very complex beast with more than 100 RFCs mandating its various protocols.

Let us take a deep dive into this 1970s technlogy to see how it works with a classic example of Alice and Bob.

1. Alice wants to send an email to Bob. She fires up her mail client and types in the message.
2. The mail client hands it over to the Mail transfer agent over the internet using SMTP.
3. The security and the spam filters kick in and check the email.
4. If the email is deemed safe, the email is forwarded to the Mail Access Agent of the recipient.
5. Bob’s Mailbox stores the email.
6. Bob’s Mail client polls from this mailbox and displays the email to Bob.
   

Simple, right? Well, not really. A lot of complexity and security potholes are hidden in plain sight here. Let us go deeper to see what’s actually happening under the hood.

Part 1: Your Mail Client is just a composer

When Alice hits send, the mail client doesn’t send the email, it simply submits it. The Mail Submission Agent, which is on a server listening on port 587 checks the authentication and stamps it with a message-ID. At this point Alice’s client is done and the email is someone else’s problem.

That someone else is the Mail Transfer Agent i.e. the postman. Its job is to find the address of the recipient and deliver the message there. To do this, the MTA uses the address book of the internet, the DNS system. It asks the question to the DNS, “What are the MX records of this domain?”.

MX i.e. Mail Exchanger are a special type of DNS records that are used to route emails. Every domain that wishes to receive email needs to publish them.

They look like this:

example.com.  IN  MX  10  mail.example.com.
example.com.  IN  MX  20  mail2.example.com.

Note: The numbers 10, 20 denote priority. If mail.example.com is down, the sending server automatically tries mail2.example.com.

Most people think email is real-time, but it is not. It is a queuing system with retry logic. If the destination server is down, the message stays on the queue and is retried with exponential backoff at 5 minutes, 30 minutes, 1 hour and so on up to 4-5 days. If Bob receives Alice’s email instantly, it’s probably because of fast queue drain. Email is an eventually consistent system that doesn’t care about speed, although modern email infrastructure makes it feel blazing fast.

Part 2: The signature is not the address

Once the sending server locates the destination after MX lookup, it opens up an SMTP connection. SMTP is just plain text. In fact anyone can type it in their terminal.

$ telnet smtp.example.com 587
220 smtp.example.com ESMTP
EHLO localhost
250 Hello
MAIL FROM:<[email protected]>
250 OK
RCPT TO:<[email protected]>
250 OK
DATA
354 Go ahead
From: [email protected]
Subject: Urgent: Verify Your Account

Click here immediately.
.
250 Message accepted

Note: How do the servers know when the messages end? By a single period ’.’ on its own line. Email is full of such tiny quirks.

Astute readers might have noticed the discrepancy in the previous email. The MAIL FROM: says [email protected] whereas the FROM: header inside the message says [email protected]. Unfortunately this is not an error and according to SMTP they can be different. The first one (MAIL FROM:) is the envelope, used by servers for routing, and the second one (From:) is the header meant for the human reading the email. The envelope is like an address written on the physical letter, whereas the header is like the sender’s signature on the letter. SMTP doesn’t check if they match.

You might be wondering why is it so. SMTP was designed in the late 70s² and early 80s³ for the academic cooperative networks⁴ which were based on trust. I’m sure nobody then had imagined internet banking would exist in the future. This glaring loophole is the reason phishing and spam work.

Part 3: Securing email AKA three kids in a trenchcoat

Good things rarely last and it didn’t take long for spammers to exploit these gaps. With the explosion of phishing and spam, the industry’s answer was not to fix SMTP but to add more duct tape. SMTP was deeply entrenched and deployed everywhere by then. So in classic software development fashion, three separate systems were invented over the years and bolted on top. Let us take a look at these three musketeers.

SPF (Sender Policy Framework):

The sending domain publishes a list of IP addresses authorised to send email on its behalf. If an email arrives from an unauthorized domain, it is marked as suspicious. SPF is fast and simple, but also hides a major flaw. If Bob forwards an email to Charlie, SPF fails because the forwarding domain is not in the original domain’s SPF record⁵. An innocent email forward turns into a spoofing attempt.

Here’s what an SPF record looks like (stored as a TXT record):

example.com.  IN  TXT  "v=spf1 ip4:203.0.113.1 include:_spf.google.com ~all"

DKIM (DomainKeys Identified Mail)

A few years later, to overcome the SPF shortcomings, DKIM was introduced. The sending server cryptographically signs every outgoing message using a private key. The public key is stored on the DNS as a DKIM record. Any receiving server can then fetch the public key and verify the signature. If the message has been tampered with, the verification fails. With this innovation, DKIM survives forwarding, however it adds another problem, DKIM signatures never expire. If an attacker got hold of a signed email from 2020 and replays it, the signature checks out. The protocol has zero concept of timeouts and expiry.

Here’s what a DKIM record looks like:

default._domainkey.example.com.  IN  TXT  "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC3v4yMx1C8WnSBsz0WMH..."

DMARC (Domain-based Message Authentication, Reporting, and Conformance)

DMARC stepped in to solve this problem and to tie SPF and DKIM together. SPF checks the envelope address, DKIM checks the signature, but neither of them checks the From: header that the user sees. DMARC introduces alignment: From: header must match the domain verified by SPF or DKIM. DMARC also tells the receiving servers what to do when the checks fail. The values it can take are:

• none: Do nothing, just report
• quarantine: Send to spam
• reject: Block entirely

DMARC also sends a daily XML report back to the domain owner with a list of all the servers that tried to impersonate them. Here’s a sample DMARC record:

_dmarc.example.com.  IN  TXT  "v=DMARC1; p=reject; adkim=s; aspf=s; rua=mailto:[email protected]; ruf=mailto:[email protected]; pct=100"

SPF, DKIM, DMARC are complicated systems, beyond the scope of this article. They work well together, but they are three different systems that need separate DNS configurations and fine grained maintenance. Each of them has lots of permutations & combinations which are prone to misconfiguration by humans. However the web of trust for email identity is built on these three. It’s impressive (and alarming) that this contraption works fine most of the time.

Part 4: The firewall

Even if the three authenticated checks pass, your email isn’t delivered yet. The exponential rise of spam and phishing means every receiving server needs to run the emails through a filtering layer. There is no standard and every provider has their own bespoke way of doing this. The system broadly consists of:

IP Reputation

Your sending IP has a persistent score maintained by 3rd parties like Spamhaus. Think of it as a historical score card of your IP. If you’re on a shared hosting platform and another customer sent spam last month, your IP inherits that reputation. In fact new IP addresses have no reputation at all and fresh new mail servers often take weeks to build some reputation. This is why most platforms like AWS are very cautious while approving requests for AWS SES.

Domain reputation

Apart from IP reputation, Google and Microsoft track how recipients interact with mail from your domain. High spam complaints and low engagement will tank your reputation.

Content analysis

Spam filters also scan your email content and score the message against thousands of rules⁶. Fighting spam is a race to the bottom and these rules have evolved massively over the decades. Even before the advent of LLMs, ML models were doing most of the heavy lifting. This means the logic is mostly opaque even to the people running these filters.

Virus scanning

When there is spam and phishing, viruses can’t be far behind. The firewall layer also scans attachments for malware signatures. Some providers have dedicated sandboxes to test these suspicious emails and study their blast radius.

URL Reputation

Nothing is left to chance and even the links in your email are checked against blocklists. Some providers rewrite every link present in the message to route the clicks through their security proxy.

Part 5: Wait, what about encryption?

If email servers are rewriting the URLs and scanning the text, are our emails are being sent in plain text over the wire? The good news: yes, email uses the TLS standard. The bad news: not in the way you think.

Email uses something called Opportunistic TLS. A sending server checks if the receiving server supports STARTTLS, which upgrades the plain text connection to an encrypted one. If the receiver supports it, they negotiate encryption and proceed securely. If the receiver doesn’t support it? Well, they fall back to plain text and proceed as if nothing happened.

For client-server (e.g. Gmail/Outlook), it is effectively mandatory, but for server-server connections it’s still opportunistic. The kicker is: TLS encrypts only the connection, not the content end-to-end. The mail servers themselves have always been reading your email, that’s how the system was designed. That’s why the Firewall layer I described earlier is able to function.

Standards like MTA-STS, DANE, S/MIME are the bandaids that are being proposed, but the adoption is very low. Remember we are talking about infra that’s processing billions of emails per day.

Part 6: So, are we home yet?

After running through all these layers and filters, one of three things happens to your email:

1. Your email is delivered to the inbox.
2. Your email is silently moved to spam. You get no notification. The sender gets no notification. The message exists but will almost certainly never be read.
3. Your email is rejected with a 5xx error. The sending server gets the rejection, generates a bounce message, and sends it back to the original sender.

Option 2 is the nightmare scenario for anyone sending transactional email. The server accepted the message, said nothing went wrong, and then buried it. RFC 5321 doesn’t prohibit this. It’s entirely legal behavior.

Parting thoughts

Email is like those creaking old Terminators from the ’70s which continue to function without complaining. Designed for a world that doesn’t exist anymore, it has optional encryption, no built-in auth, three⁺ retrofitted security layers bolted on top, an unstandardized filtering layer and many more quirks. Yet billions of emails arrive correctly every single day.

Email is not elegant but nonetheless it is Lindy⁷. In the new age of agentic AI⁸, we can only expect it to metamorphose into another dimension.