Heritrix

From COPTR
Jump to navigation Jump to search


Heritrix
Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site.
Homepage:http://crawler.archive.org
License:GNU Lesser General Public License 2.1
Platforms:Written in Java. Must have Java Runtime Environment (JRE, http://www.java.com/en/download/index.jsp) and at least Java version 5.0 installed. Default heap size is 256MB RAM.
Function:Web Crawl
Content type:Web


Mailing List(s)

See here for more information.

Issues Feed

2019-04-24 22:08:54
[HER-2097] Ambiguity between srcset urls and data:image base64 encoded image
<style type="text/css"> .tableBorder, .grid { background-color: #fff; width: 100%; border-collapse: collapse; } .tableBorder td, .grid td { vertical-align: top; padding:...
by Adam Miller
2017-08-31 17:31:02
[HER-2096] heritrix hitting non existent URLs in wix.com/app-market
<style type="text/css"> .tableBorder, .grid { background-color: #fff; width: 100%; border-collapse: collapse; } .tableBorder td, .grid td { vertical-align: top; padding:...
by Vangelis Banos
2016-10-28 19:55:16
[HER-2095] Crawl M3U8 files and capture resources they describe
<style type="text/css"> .tableBorder, .grid { background-color: #fff; width: 100%; border-collapse: collapse; } .tableBorder td, .grid td { vertical-align: top; padding:...
by Barbara Miller
2019-03-08 19:05:08
[HER-2094] Add support for extracting URLs from img srcset attribute
<style type="text/css"> .tableBorder, .grid { background-color: #fff; width: 100%; border-collapse: collapse; } .tableBorder td, .grid td { vertical-align: top; padding:...
by Adam Miller
2016-06-07 07:21:42
[HER-2093] appCtx.getBean() does no longer work in scripting console
<style type="text/css"> .tableBorder, .grid { background-color: #fff; width: 100%; border-collapse: collapse; } .tableBorder td, .grid td { vertical-align: top; padding:...
by Robert Jäschke

Description[edit]

Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site. The software is most often used as a powerful back-end tool incorporated into a web archiving workflow.

Provider[edit]

Internet Archive

Licensing and cost[edit]

Apache License, Version 2.0 – free. Some individual source code files are subject to or offered under other licenses.

Development activity[edit]

Version 3.1.1 was released in May 2012. Heritrix powers the Internet Archive, and so receives ongoing support.

Platform and interoperability[edit]

As a Java application, Heritrix is theoretically platform agnostic; however, only Linux is supported.  The software requires Java Runtime Environment 1.6 or higher, and at least 256MB of available RAM.

Functional notes[edit]

Web crawls are carried out by configuring a ‘job,’ which itself is an instance of a crawl template called a ‘profile.’ Although they contain the same configurations, these two entities have different functions; profiles record the set of configurations and act as a starting point for shaping a new job, but only the job itself can excecute a crawl. The software will crawl FTP sites in addition to HTTP. Users can examine the results of a crawl by opening its log files, which include information about crawl problems and errors, each URI that was collected, and statistics about the job as a whole. Users can also create reports showing a summary of the crawl’s activity. Heritrix stores the web resources it crawls in an Arc file. The software includes a command-line tool called arcreader which can be used to extract the contents.

Documentation and user support[edit]

The User Guide for versions 3.0 and 3.1 is in the form of a wiki, which at time of writing is not structured in any obvious narrative order; while detailed, it is very difficult to navigate.  The User Manual for version 2.0 is structured and can be used as a reference for navigation.  Extensive documentation is available, including release notes, Javadoc API documentation, and FAQs linking within the wiki. Heritrix’s website links to two active mailing lists: a yahoo discussion group and a sourceforge list distributing source code commits. The project also uses a public JIRA for bug, feature, and issue tracking.

Usability[edit]

Heritrix is installed via a command line interface, but once installed the user can launch a web-based interface for configuration. Setting up a crawl requires a significant number of adjustments.

Expertise required[edit]

Installation requires solid knowledge of Linux and command line interfaces. As with any web archiving software, deep understanding of the project’s scope and collections policy is essential in order to set up appropriate targets.

Standards compliance[edit]

Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags.

Influence and take-up[edit]

Heritrix is extremely influential; as of March 2012, the SourceForge site reports nearly 240,000 downloads. Users include the Internet Archive, the British Library, the United States Library of Congress, and the French National Library. The software powers NetarchiveSuite and the Web Curator Tool.


User Experiences[edit]

Development Activity[edit]

All development activity is visible on GitHub: http://github.com/internetarchive/heritrix3/commits


Release Feed[edit]

Below the last 3 release feeds:

2021-06-21 14:42:49
[tag:github.com,2008:Repository/2623406/3.4.0-20210621 3.4.0-20210621]
by anjackson
2021-06-18 08:41:27
[tag:github.com,2008:Repository/2623406/3.4.0-20210618 3.4.0-20210618]
by anjackson
2021-06-17 20:08:36
[tag:github.com,2008:Repository/2623406/3.4.0-20210617 2021-06-17 Interim Release]
by anjackson


Activity Feed[edit]

Below the last 5 commits:

2021-07-27 11:26:46
[tag:github.com,2008:Grit::Commit/466c10b7a4cb32288c708c6b2d42e6d4ede0cdf5 Merge pull request #416 from internetarchive/extractor-chrome-replay-…]
by ato https://github.com/ato
2021-07-24 08:25:21
[tag:github.com,2008:Grit::Commit/014fb2dcad9a5ce2b0dbaecf1b2d76a60439e1e7 ExtractorChrome: have frontier consider browser-fetched uris included]
by ato https://github.com/ato
2021-07-24 08:16:44
[tag:github.com,2008:Grit::Commit/bbae794b473beb69fb0b000d7875b3384a3a277f ExtractorChrome: run extractors on subresources captured by the browser]
by ato https://github.com/ato
2021-07-23 08:38:56
[tag:github.com,2008:Grit::Commit/8ec7ce33638b23bf02bf40ff5efb2809ccefb52f ExtractorChrome: replay the recorded CrawlURI response to the browser]
by ato https://github.com/ato
2021-07-22 02:21:35
[tag:github.com,2008:Grit::Commit/ab19efa9d6673cb819b819879aa874d77b8ade23 Merge pull request #414 from internetarchive/maven-assembly-plugin-3.3.0]
by ato https://github.com/ato