web spidering

Anemone is a free, multi-threaded Ruby web spider framework from Chris Kite, which is useful for collecting information about websites.

With Anemone you can write tasks to generate some interesting statistics on a site just by giving it the URL.