Nothing like a Cloudflare outage to inspire another blog post. I learned from the last incident and was able to turn off the CF proxy early on while it was still possible to do so. This kept my website up during most of the outage as requests were bypassing CF and hitting my origin servers directly.
However, in turning off the CF proxy, I noticed one flaw in my infrastructure. When I turned off the CF proxy for links.mydomain.com, which is the subdomain I use to wrap links in outgoing emails, any user who clicked a link in our emails saw a SSL certificate not valid error.
In this post, I describe how I've added a layer of redundancy (using Cloudfront) so that next time I have to turn off Cloudflare, not only will my website still work but email link clicks will too.
Reason for SSL Certificate Error
I relied on Cloudflare for two things:
- As a DNS with a CNAME record mapping
links.mydomain.comtosendgrid.net - As a proxy which presents a valid
*.mydomain.comcertificate to the browser.
All our email links are https links of the form https://links.mydomain.com/.... When I disabled the proxy during the outage, the browser wasn't presented with a valid SSL certificate for links.mydomain.com and showed the error.
Using Cloudfront As Backup
Instead of mapping links.mydomain.com to sendgrid.net directly, I now map it to mycdn.cloudfront.net. I've configured my Cloudfront distribution to proxy requests to sendgrid.net and also present a valid SSL certificate for *.mydomain.com.
Now Cloudfront provides the same function as Cloudflare. If Cloudflare goes down, I can turn off the CF proxy and rely exclusively on Cloudfront. If Cloudfront goes down, I can update my Cloudflare DNS to map links.mydomain.com to sendgrid.net. If both go down, I'll know it's time to finally retire.
Getting Cloudfront to Work with Sendgrid
This was not as straightforward as I expected it to be. Sendgrid's documentation on this is outdated and also erroneous. Below I share what I did instead.
After my Cloudfront distribution was deployed, I just had to change my Cloudflare DNS settings to point links.mydomain.com to mycdn.cloudfront.net.