Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all
I run https://next-episode.net and recently (a few months ago) - I noticed it started getting pages not indexed because of “Server error (5xx)” in Google Search Console (GSC):
https://next-episode.net/img/upload/5xx.png
On the website everything looked good and all reported links worked fine.
I tried validating these errors at GSC as fixed, but it would always report back that the issue is still present and new links would keep popping up to have 5xx errors (as seen on the screenshot).
This was worrisome because it indicated there was some kind of an issue I wasn’t aware of that may be affecting not only Google’s crawlers, but my users as well.
I did what everyone would do - checked my server, Cloudflare and analytics logs for anything suspicious and placed some additional logging to try and catch what was happening.
This turned out nothing - as far as I could tell - no requests returned any 5xx errors, so I decided it’s just a weird Google quirk and ignored it for a while.
With time though, Google kept reporting these problems and the count of 5xx URLs only grew larger so once again (about 2 weeks ago) I started investigating what was happening.
This time around, I tried to match the URLs reported by GSC with the analytics provided by Cloudflare and bingo - I found that all these requests had the Edge Status Code (and Origin Status Code) of “429 Too Many Requests”.
Now that was progress.
There is only one thing on my service that would return this status code and is my custom rate limiting which would be triggered if you do more than 30 requests in less than 10 seconds.
What changed so that Google suddenly decided to crawl so aggressively and hit that limit (something that never happened before, and Next Episode is online for more than 19 years now!) and why it’s reporting them 5xx in GSC when my server clearly returns 429 - I don’t know.
What I do know for sure is that Google is misreporting 429 server status as 5xx.
To fix this (at least as a quick fix for now) - I whitelisted in my rate limiter all Google Crawlers’ IPs (which I found through here: https://www.infidigit.com/blog/google-update-googlebots-ip-a... ) - listed in this JSON provided by Google: https://developers.google.com/search/apis/ipranges/googlebot...
For just in case, I also passed on the ASN in the request header (through a Cloudflare transform rule) and whitelisted the whole Google ASN (15169) as well.
After - I monitored for new 5xx errors popping up in GSC and new 429 statuses logged in Cloudflare from Google’s ASN and so far (for more than 2 weeks) - so good. Is there any possibility there is a problem with the 429 response? So something is converting to say a bad gateway error because it doesn’t like the 429? Not trying to criticize if this was already checked. Just something I’d try to double check out of being overly cautious. with spammed curl -i, I get It's usually used to prevent caching from older caching systems. Fun fact: this specific value of Expires is the birthday of Sasha Schumann, the person who added this code to PHP[1][2]. [1]: https://stackoverflow.com/a/8194500/2805120 [2]: https://github.com/php/php-src/blob/master/ext/session/sessi... Expires header in the past is typically used to prevent caching. The lower-case headers is part of the http2 specification. Lower case headers are required for http/2 requests and responses. I can't find a problem with the 429 ... How do you rate limit unidentified users? If you are treating all unidentified users, or at least all Google bots as 1 user and they consider each Google bot instances with a unique ip and no credentials a unique user and are hitting a 429 with a single request they may be treating that as a 503 instead. Simply identifying them by IP so that each Google bot instance is a unique user ... Sounds more like an issue on cloud flare or whatever server Google is hitting. It should return 429 so Google can reduce the requests. I'm not even sure why you would rate limit in the first place? IPS are not unique. One company gateway or university for example has plenty of users. Rate limit requests from users you know and make sure every public API is properly cached. Why do you say that? The 429 edge status code implies quite strongly that the client gets 429s as well. Not hard to test it from a vpn connection and verify. I rate limit because of aggressive bots (scrapers and ones trying random injection attacks etc). And yes, it's returning 429 ... Cloudflare doesn't offer rate limiting by user. Caching is completely useless as a replacement for rate limiting. Simply add a bunch of nonsense to the url, or change query parameters, and you will bypass the cache. Cloudflare does offer rate limiting - it's in Security / WAF / Rate limiting rules. On the free plan it is perhaps a bit limited in the rules you can create (match based on bot or URL regex, limit by # requests per IP per 10 seconds). But still pretty useful for slowing stupid requests. https://developers.cloudflare.com/waf/rate-limiting-rules/cr... And / or you can use Transform Rules to better normalize the Origin URLs so that the query string or other path info doesn't create new origin requests. Or more simply, enable Ignore Query String in the Cache Settings. That’s not proper caching then. I have gotten these emails as well for a Cloudflare-hosted site. I was surprised when I got them because they seemed to crop up out of the blue (a month or so ago?) and AFAIK I couldn't think of much that could actually cause a 500 status code on my end. I'll take a look for 429s. Cheers. Cool. Let me know if it turns out it's the same thing.
Expires header in the past seems suspect. And all the header names are lowercased. Too many requests, please try again later.HTTP/2 429
date: Wed, 14 Aug 2024 22:09:40 GMT
content-type: text/html;charset=UTF-8
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
retry-after: 3600
vary: Accept-Encoding
set-cookie: PHPSESSID=b25ece07c8923fa6b66b14599e1ed545; path=/
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=z4hhYJcawKduyEdIbsTqPNE6k1o7YSkmMFlcDvDfe%2BvU7V8B%2FFJj4mItPssnI96kp1Ot%2B7jadmhYVWC%2BpAKG4zrqN%2FG6cY7KXDJJtFH2gQwQCNwGfS6Rbsu4xOCnMhoyT7U%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8b3445445e51f93d-SJC