Difference between revisions of "Heritrix"

From COPTR
Jump to navigation Jump to search
(→‎Development Activity: Adding activity sources)
 
(4 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
{{Infobox_tool
+
{{Infobox tool
 +
|image=heritrix-logo.gif
 
|purpose=Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site.
 
|purpose=Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site.
−
|image=
+
|homepage=https://github.com/internetarchive/heritrix3/wiki
−
|homepage=http://crawler.archive.org
 
 
|license=GNU Lesser General Public License 2.1
 
|license=GNU Lesser General Public License 2.1
 
|platforms=Written in Java. Must have Java Runtime Environment (JRE, http://www.java.com/en/download/index.jsp) and at least Java version 5.0 installed. Default heap size is 256MB RAM.
 
|platforms=Written in Java. Must have Java Runtime Environment (JRE, http://www.java.com/en/download/index.jsp) and at least Java version 5.0 installed. Default heap size is 256MB RAM.
 +
|Wikidata ID=Q3097891
 +
|function=Web Capture
 +
|content=Web
 +
}}
 +
{{Infobox tool details
 +
|ohloh_id=Heritrix
 +
|mailing_lists=https://groups.yahoo.com/neo/groups/archive-crawler/info
 +
|issues_rss=https://webarchive.jira.com/sr/jira.issueviews:searchrequest-rss/temp/SearchRequest.xml?jqlQuery=project+%3D+HER&tempMax=100
 
}}
 
}}
−
 
−
<!-- Delete the Categories that do not apply -->
 
−
[[Category:Web Crawl]]
 
−
[[Category:Web]]
 
−
 
−
 
 
= Description =
 
= Description =
 
[https://webarchive.jira.com/wiki/display/Heritrix/Heritrix Heritrix] is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site. The software is most often used as a powerful back-end tool incorporated into a web archiving workflow.
 
[https://webarchive.jira.com/wiki/display/Heritrix/Heritrix Heritrix] is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site. The software is most often used as a powerful back-end tool incorporated into a web archiving workflow.
Line 37: Line 39:
 
Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags.
 
Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags.
 
====Influence and take-up====
 
====Influence and take-up====
−
Heritrix is extremely influential; as of March 2012 the sourceforge site reports nearly 240,000 downloads. [https://webarchive.jira.com/wiki/display/Heritrix/Users+of+Heritrix Users]&nbsp;include the Internet Archive, The British Library, the United States Library of Congress, and the French National Library. The software powers [http://www.dcc.ac.uk/node/9380 Netarchive Suite] and the [http://www.dcc.ac.uk/node/9394 Web Curator Tool].
+
Heritrix is extremely influential; as of March 2012, the SourceForge site reports nearly 240,000 downloads. [https://webarchive.jira.com/wiki/display/Heritrix/Users+of+Heritrix Users] include the Internet Archive, the British Library, the United States Library of Congress, and the French National Library. The software powers [http://www.dcc.ac.uk/node/9380 NetarchiveSuite] and the [http://www.dcc.ac.uk/node/9394 Web Curator Tool].
  
  
Line 44: Line 46:
  
 
= Development Activity =
 
= Development Activity =
−
 
+
<!-- Provide *evidence* of development activity of the tool. For example, RSS feeds for code issues or commits. -->
−
{{Infobox_tool_details
+
All development activity is visible on GitHub: http://github.com/internetarchive/heritrix3/commits
−
|releases_rss=https://github.com/internetarchive/heritrix3/commits/master.atom
+
−
|issues_rss=https://webarchive.jira.com/sr/jira.issueviews:searchrequest-rss/temp/SearchRequest.xml?jqlQuery=project+%3D+HER&tempMax=100
+
−
|mailing_lists=https://groups.yahoo.com/neo/groups/archive-crawler/info
+
=== Release Feed ===
−
|ohloh_id=Heritrix
+
Below the last 3 release feeds:
−
}}
+
<rss max=3>https://github.com/internetarchive/heritrix3/releases.atom</rss>
 +
 
 +
 +
=== Activity Feed ===
 +
Below the last 5 commits:
 +
<rss max=5>https://github.com/internetarchive/heritrix3/commits/master.atom</rss>

Latest revision as of 15:59, 26 November 2021



Heritrix
Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site.
Homepage:https://github.com/internetarchive/heritrix3/wiki
License:GNU Lesser General Public License 2.1
Platforms:Written in Java. Must have Java Runtime Environment (JRE, http://www.java.com/en/download/index.jsp) and at least Java version 5.0 installed. Default heap size is 256MB RAM.
Wikidata ID:Q3097891
Function:Web Capture
Content type:Web
Appears in COW:Quality Assurance: Iterative Seed Issue Decision Tree, Web Archiving Quality Assurance (QA) Workflow, Web Archiving Quality Assurance Lifecycle

Mailing List(s)

See here for more information.

Issues Feed

Failed to load RSS feed from https://webarchive.jira.com/sr/jira.issueviews:searchrequest-rss/temp/SearchRequest.xml?jqlQuery=project+%3D+HER&tempMax=100: There was a problem during the HTTP request: 400 Bad Request

Description

Heritrix is an open-source web crawler, allowing users to target websites they wish to include in a collection and to harvest an instance of each site. The software is most often used as a powerful back-end tool incorporated into a web archiving workflow.

Provider

Internet Archive

Licensing and cost

Apache License, Version 2.0 – free. Some individual source code files are subject to or offered under other licenses.

Development activity

Version 3.1.1 was released in May 2012. Heritrix powers the Internet Archive, and so receives ongoing support.

Platform and interoperability

As a Java application, Heritrix is theoretically platform agnostic; however, only Linux is supported.  The software requires Java Runtime Environment 1.6 or higher, and at least 256MB of available RAM.

Functional notes

Web crawls are carried out by configuring a ‘job,’ which itself is an instance of a crawl template called a ‘profile.’ Although they contain the same configurations, these two entities have different functions; profiles record the set of configurations and act as a starting point for shaping a new job, but only the job itself can excecute a crawl. The software will crawl FTP sites in addition to HTTP. Users can examine the results of a crawl by opening its log files, which include information about crawl problems and errors, each URI that was collected, and statistics about the job as a whole. Users can also create reports showing a summary of the crawl’s activity. Heritrix stores the web resources it crawls in an Arc file. The software includes a command-line tool called arcreader which can be used to extract the contents.

Documentation and user support

The User Guide for versions 3.0 and 3.1 is in the form of a wiki, which at time of writing is not structured in any obvious narrative order; while detailed, it is very difficult to navigate.  The User Manual for version 2.0 is structured and can be used as a reference for navigation.  Extensive documentation is available, including release notes, Javadoc API documentation, and FAQs linking within the wiki. Heritrix’s website links to two active mailing lists: a yahoo discussion group and a sourceforge list distributing source code commits. The project also uses a public JIRA for bug, feature, and issue tracking.

Usability

Heritrix is installed via a command line interface, but once installed the user can launch a web-based interface for configuration. Setting up a crawl requires a significant number of adjustments.

Expertise required

Installation requires solid knowledge of Linux and command line interfaces. As with any web archiving software, deep understanding of the project’s scope and collections policy is essential in order to set up appropriate targets.

Standards compliance

Heritrix does not offer metadata support. The software is designed to respect robots.txt exclusion directives and META robots tags.

Influence and take-up

Heritrix is extremely influential; as of March 2012, the SourceForge site reports nearly 240,000 downloads. Users include the Internet Archive, the British Library, the United States Library of Congress, and the French National Library. The software powers NetarchiveSuite and the Web Curator Tool.


User Experiences

Development Activity

All development activity is visible on GitHub: http://github.com/internetarchive/heritrix3/commits


Release Feed

Below the last 3 release feeds:

2025-12-11 05:08:49
[tag:github.com,2008:Repository/2623406/3.13.0 3.13.0]
by ato
2025-10-30 01:52:52
[tag:github.com,2008:Repository/2623406/3.12.0 3.12.0]
by ato
2025-09-22 05:04:22
[tag:github.com,2008:Repository/2623406/3.11.0 3.11.0]
by ato


Activity Feed

Below the last 5 commits:

2026-02-02 09:12:36
[tag:github.com,2008:Grit::Commit/cb1f36b92657de7f5453732173f099c0911aa538 Merge pull request #709 from internetarchive/dependabot/maven/minor-a…]
by ato https://github.com/ato
2026-02-02 09:11:23
[tag:github.com,2008:Grit::Commit/fe49a8ae58b731847a8947ad08e89375065cf29d Merge pull request #708 from mistydemeo/misty/yt-dlp_filter_-sr]
by ato https://github.com/ato
2026-02-01 23:44:24
[tag:github.com,2008:Grit::Commit/938c68dbd8a8a495316fd4bea30825375efc88ba Bump the minor-and-patch group with 25 updates]
by dependabot https://github.com/dependabot
2026-01-30 21:19:47
[tag:github.com,2008:Grit::Commit/3e19a2487963a8542dae181ea6af810eee3315d0 ExtractorYoutubeDL: configuration for skipping -sr]
by mistydemeo https://github.com/mistydemeo
2026-01-30 19:51:52
[tag:github.com,2008:Grit::Commit/49d6c5d830ccff9b8d411c845ce33d3dad48e8b9 ExtractorYoutubeDL: skip YouTube super resolution videos]
by mistydemeo https://github.com/mistydemeo

Retrieved from ‘https://coptr.digipres.org/index.php?title=Heritrix&oldid=5433’