Tools that support the capture of data from the world wide web, by "crawling" links between resources or other approaches.
The DeDuplicator (Heritrix add-on module)