Sharding Web Traffic with Subdomains to Better Leverage Cloudflare CDN and WAF

2 min read Original article ↗

At Listen Notes, we reverse-proxy our URLs through Cloudflare - just like 20%+ of websites on the internet. We've relied on Cloudflare for both security (WAF) and performance (CDN) since launching Listen Notes in 2017.

Overall, Cloudflare has been a reliable vendor in our tech stack. Yes, there have been occasional outages (which are understandable at their scale), but it has still been far better than the alternatives, whether that’s competing CDN/WAF vendors or running and maintaining our own infrastructure in an in-house data center.

One particularly effective strategy we’ve implemented is using separate subdomains for different types of traffic, even when they’re all served from the same web server (uwsgi / Django). This allows us to apply tailored firewall rules and caching strategies to each category of requests.

How We Do It

Here are a few example subdomains that all run on the same web server (uwsgi + Django):

www (dot) listennotes.com: Dedicated to human users.

feeds (dot) listennotes.com: Designed primarily for bots consuming RSS feeds.

audio (dot) listennotes.com: Serves both humans and bots for audio URL proxying.

By separating traffic in this way, we can fine-tune security and performance settings for each subdomain - stronger bot controls where needed, more aggressive caching where appropriate, and fewer compromises overall. It’s also easier and less error-prone to configure WAF and cache rules by subdomain rather than writing custom url path patterns.

A Simple & Obvious Lesson

Like many useful lessons in life, this approach is painfully simple and obvious in hindsight. That said, we didn’t arrive at it until at least the fifth year of the company’s history.

Better late than never :)

You can use our website to search 3,687,458 podcasts and 188,980,118 episodes by people, places, or topics.

An investment in knowledge pays the best interest, so I listen to podcasts.  ‐ Benjamin "Llamacorn" Franklin