Difference between revisions of "Heritrix"
(→Development Activity: Adding activity sources) |
Nullhandle (talk | contribs) |
||
| Line 37: | Line 37: | ||
Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags. | Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags. | ||
====Influence and take-up==== | ====Influence and take-up==== | ||
| − | Heritrix is extremely influential; as of March 2012 the | + | Heritrix is extremely influential; as of March 2012, the SourceForge site reports nearly 240,000 downloads. [https://webarchive.jira.com/wiki/display/Heritrix/Users+of+Heritrix Users] include the Internet Archive, the British Library, the United States Library of Congress, and the French National Library. The software powers [http://www.dcc.ac.uk/node/9380 NetarchiveSuite] and the [http://www.dcc.ac.uk/node/9394 Web Curator Tool]. |
Revision as of 17:06, 2 March 2016
Description
Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site. The software is most often used as a powerful back-end tool incorporated into a web archiving workflow.
Provider
Internet Archive
Licensing and cost
Apache License, Version 2.0 – free. Some individual source code files are subject to or offered under other licenses.
Development activity
Version 3.1.1 was released in May 2012. Heritrix powers the Internet Archive, and so receives ongoing support.
Platform and interoperability
As a Java application, Heritrix is theoretically platform agnostic; however, only Linux is supported. The software requires Java Runtime Environment 1.6 or higher, and at least 256MB of available RAM.
Functional notes
Web crawls are carried out by configuring a ‘job,’ which itself is an instance of a crawl template called a ‘profile.’ Although they contain the same configurations, these two entities have different functions; profiles record the set of configurations and act as a starting point for shaping a new job, but only the job itself can excecute a crawl. The software will crawl FTP sites in addition to HTTP. Users can examine the results of a crawl by opening its log files, which include information about crawl problems and errors, each URI that was collected, and statistics about the job as a whole. Users can also create reports showing a summary of the crawl’s activity. Heritrix stores the web resources it crawls in an Arc file. The software includes a command-line tool called arcreader which can be used to extract the contents.
Documentation and user support
The User Guide for versions 3.0 and 3.1 is in the form of a wiki, which at time of writing is not structured in any obvious narrative order; while detailed, it is very difficult to navigate. The User Manual for version 2.0 is structured and can be used as a reference for navigation. Extensive documentation is available, including release notes, Javadoc API documentation, and FAQs linking within the wiki. Heritrix’s website links to two active mailing lists: a yahoo discussion group and a sourceforge list distributing source code commits. The project also uses a public JIRA for bug, feature, and issue tracking.
Usability
Heritrix is installed via a command line interface, but once installed the user can launch a web-based interface for configuration. Setting up a crawl requires a significant number of adjustments.
Expertise required
Installation requires solid knowledge of Linux and command line interfaces. As with any web archiving software, deep understanding of the project’s scope and collections policy is essential in order to set up appropriate targets.
Standards compliance
Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags.
Influence and take-up
Heritrix is extremely influential; as of March 2012, the SourceForge site reports nearly 240,000 downloads. Users include the Internet Archive, the British Library, the United States Library of Congress, and the French National Library. The software powers NetarchiveSuite and the Web Curator Tool.
User Experiences
Development Activity
Mailing List(s)
See here for more information.
Release Feed
- 2024-11-20 08:12:08
- [tag:github.com,2008:Grit::Commit/13075ec4442c6a5352d509fd057c7dd3a3b101f7 Merge pull request #624 from kris-sigur/strict-if-url-matching]
- by ato https://github.com/ato
- 2024-11-20 08:11:55
- [tag:github.com,2008:Grit::Commit/29cc045c221a2b2747d5f4b884cdeb375ccf2cbd Merge pull request #623 from kris-sigur/manfest-links]
- by ato https://github.com/ato
- 2024-10-30 08:53:08
- [tag:github.com,2008:Grit::Commit/4a49557b8b28a7423867e2b0aea7e2b099585897 Force strict if URL matching regex.]
- by kris-sigur https://github.com/kris-sigur
- 2024-10-29 08:53:15
- [tag:github.com,2008:Grit::Commit/5ed6147c3e3f69441b3771c95b4dbd56645e1f06 Treat manifest hops same as navlink hops]
- by kris-sigur https://github.com/kris-sigur
- 2024-10-29 06:18:54
- [tag:github.com,2008:Grit::Commit/25c73da7e334e61e9c29dcb5a59045d3992c3a8c [maven-release-plugin] prepare for next development iteration]
- by ato https://github.com/ato
Issues Feed
Failed to load RSS feed from https://webarchive.jira.com/sr/jira.issueviews:searchrequest-rss/temp/SearchRequest.xml?jqlQuery=project+%3D+HER&tempMax=100: There was a problem during the HTTP request: 400 Bad Request