Using Proxies in Web Scraping
stackabuse.comThe issue that I struggled with is browser fingerprinting. Servers can detect if you're using a browser. If you are, then there are a number of flags, or parameters, in the browser that give away your identity. You can cycle through user agents and proxies. I don't think these things help. If you're not loading the page through a browser or if you try to hide information, you're already flagged. Scraping is a cat and mouse game now.
IIRC there are ways to give an authentic-looking fingerprint to browsers used for scraping. But as you said, it's a cat and mouse game