How I Spent Six Years Developing a Product

After reading the article just realised why didn't you use mechanize with beautifulsoup library in Python. I feel you could have succeeded in your first attempt within 3-6 months. I did it in 2008 and it worked great for me. Also look at another open source project called scrapy http://scrapy.org/