Project status
This repository is preserved for archival purposes only and is no longer actively developed. A new version of Alida, specifically tailored for crawling data for vector databases, is available as Alida Vector. See alida.dev for an overview.
A recording of the original talk is available from ClojureTV on YouTube.
The Alida project was started as companion code to my talk at EuroClojure 2012 on the topic of "Building a search engine with Clojure". The goal of this application is to provide the back-end for a simple search engine, while the front-end (i.e. the part that performs the actual searches and displays the results to the visitor) is separate. Alida provides the following functionality:
- Data retrieval (web crawling).
- Storage (storing the crawled pages).
- Scraping (extracting interesting bits and pieces from documents).
- Indexing (storing the end result in a searchable index).
The application depends on several external libraries and applications, including Apache CouchDB to store the crawled pages, Apache Lucene for indexing, clj-http as a wrapper around the Apache HttpClient, Clutch as a CouchDB library and both Enlive and Jsoup for scraping data from the retrieved documents. As an experimental project Alida isn't designed for immediate production use. Some technology choices reflect that. Apache CouchDB, for example, isn't the most obvious choice for storing crawled pages. For large scale document storage something like HDFS would be more efficient, but CouchDB is easier to set up in a small, experimental setting. If you need something more mature, I'd suggest looking at Apache Nutch, which also includes a web crawler. That being said, one of the goals for Alida is to be able to power a real-world search engine project.
Check the docs/ directory for the slides. I've also created two sub-projects. clojure-blog-search is a very basic example of how you could use Alida to crawl the web. Please remember that it does an actual crawl on the personal blogs of fellow Clojurians, so please don't run it unless you have a very good reason to do so. I've also created clojure-blog-search-www, which is a little front-end web application that also includes a pre-built Lucene index that you can experiment with.
You can use this code as a library in your own project by adding the following to the :dependencies in your project.clj file:
[alida "0.1.3"]
About and contact
You can read more about me on my personal homepage, which also includes contact information. Feel free to reach out if you have any questions or feedback!
License
Copyright 2012, F.M. de Waard / Vixu.com. All code is covered by the Apache License, version 2.0.