Ask HN: Is there a library or a package that converts a web page to JSON?
I am living in a country where open data is a very alien concept for the public and government. Except for a handful of civic hackers who find it very difficult to create apps or services due to lack of open APIs or datasets. Data is available on government websites but its not technically open (as in a RESTful API). I am thinking about writing a library that can crawl all over a page and convert the HTML into JSON that can be consumed over REST. Is there something like that out there already? What are some of the things I need to consider before I start writing this? Back in the day Yahoo Pipes would have been something I'd have recommended as being a tool in this space. Here's a blog post that lists some alternatives to Yahoo Pipes and some of them look interesting for your use case, though I don't know that any of them is exactly what you're looking for. http://www.makeuseof.com/tag/12-best-yahoo-pipes-alternative... Also, CloudScrape: