Show HN: POC to scrape and structure HTML into JSON for RAG
structured.pages.devHey all,
I built a quick PoC that scrapes a webpage, sends the content to Gemini Flash, and outputs a clean, structured JSON — ready for RAG workflows.
In my case, I’ll use this structured data to enhance models by integrating external knowledge sources during the generation process.
Curious if you think this has potential or if there are any use cases I might have missed.
Happy to share more details if there's interest! Do you mind sharing more about the implementation details? Any safeguards you have for the urls and all? Basically, I use a headless Chromium with Puppeteer to render the page. Then, some logic extracts and cleans the HTML content. Finally, I use Gemini with a specific schema to return a JSON response. Okay, thanks! How do you plan to use it? At the moment, I use it in client projects to build agents for their chat systems by adding RAG to models Okay