Ask HN: Web scraping in production?
Are any of you maintaining any web scrapers in production?
I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.
Curious what applications people have in industry and what sorts of techniques people use for reliability. excel web scraping[0] (vs. using python[1] and/or odbc/delimited files) A few 2025 use cases [2],[3]: [0] : excel scraping : https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk [1] : python scraping : https://www.youtube.com/watch?v=Oo8-nEuDBkk Neat - didn't realize there were affordances for scraping in Excel (but in hindsight I shouldn't be surprised). I didn't follow the connection between modern looming and scraping though? hint: silk spider webs & fabric threads. Guess 3d printing should have been clarified as linear, fused deposition. Melted plastic line gets scraped along plate/material. The 3d printed web reference, in this instance, being the in-fill pattern. [0] Robotic metal pinching / incremental sheet forming might be bit more clearer example. [1] ----- [0] : in-fill pattern : https://jlc3dp.com/blog/choosing-the-right-infill-structure-... Scraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years. IMHO, "untyped" format/delimited file yes. Directly placing/'compiling' in appropriate topological construct/environment works wonders. aka environment of database, spreadsheet, "reports" with information beyond raw data, etc
---- Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc
---- creative take on topic:
modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384
Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html
3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html