Pegasus is a robust, modular, multi-threaded web-crawler for clojure.
Crawl state saved to disk - can recover from crashes.
Parallelism achieved using core.async
Very easy to customize
Crawl my blog (http://blog.shriphani.com/) using enlive selectors
- Crawl my blog (http://blog.shriphani.com/) using XPaths selectors
- Github issues: [link].
- For other questions, you can send me email @ s...@gmail.com