Difference between revisions of "NetarchiveSuite"

From COPTR
Jump to navigation Jump to search
(Trial import from script.)
 
 
(8 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Infobox_tool
+
{{Infobox tool
|purpose=NetarchiveSuite is a web archiving software package which is used to plan, schedule and run web harvests of parts of the Internet.
+
|image=NAS.gif
|image=
+
|purpose=NetarchiveSuite is a web archiving software package designed to plan, schedule and run web harvests of parts of the Internet.
|homepage=http://netarchive.dk/suite/Welcome
+
|homepage=https://sbforge.org/display/NAS/NetarchiveSuite
|license=LGPL License
+
|license=[http://www.gnu.org/licenses/lgpl.html#translations GNU Lesser General Public License]
|platforms=n/a
+
|platforms=Linux
 +
|function=Web Capture
 +
|content=Web
 
}}
 
}}
 +
{{Infobox tool details
 +
|ohloh_id=NetarchiveSuite
 +
}}
 +
= Description =
 +
The [https://sbforge.org/display/NAS/NetarchiveSuite NetarchiveSuite] is a web archiving software package designed to plan, schedule and run web harvests of parts of the Internet. It is also serves as an archiving platform for the collected materials, storing and performing consistency checks of the data.
  
<!-- Delete the Categories that do not apply -->
+
====Provider====
[[Category:Metadata]]
+
The Royal Library of Denmark and the State and University Library of Denmark
[[Category:Lifecycle Management; Storage]]
+
====Platform and interoperability====
[[Category:Verification]]
+
Netarchive requires a computer running a Linux operating system with Sun Java 1.6, as well as Java Messaging Service. The software uses Heritrix as its web crawler.
[[Category:Retrieval]]
+
====Functional notes====
[[Category:Web]]
+
The NetarchiveSuite is split into four main modules: three modules corresponding to processes of harvesting, archiving and accessing materials, and one module to coordinate functions. The Harvester module can organise both snapshot and recurring harvests; it supports packaging metadata about the harvest together with the harvested data. The Archive module advertises bit consistency checks and the ability to support distributed batch jobs; it also supports storage. The Access Module uses a proxy solution to give access to the material. As the software uses Heritrix for its crawls, the materials collected take the form of arc files (not to be confused with ARC files).
 
+
====Documentation and user support====
 +
Netarchive&rsquo;s website includes extensive [https://sbforge.org/display/NAS/Documentation documentation], including an Overview and Quick Start Manual. Detailed guidance is found in the Configuration, Installation, and User Manuals. Developer guidance is found in the System Design and Additional Tools Manual. The website also points to a new Wiki that as of writing is in some places incomplete, and in general rather difficult to navigate. The project supports four mailing lists, all of which are currently active: -announce; -curator; -devel; and -users. The site also includes a contact page with email addresses for individuals at KB, SB, BnF, and ONB.
 +
====Usability====
 +
The Suite advertises its ability to be used by librarians rather than systems administrators; its Quickstart installation option is designed to take an hour, and the GUI for ongoing use is extremely simple.
 +
====Expertise required====
 +
With any web archiving project, deep understanding of the project&rsquo;s scope and collections policy is essential in order to set up appropriate targets.
 +
====Standards compliance====
 +
No standards compliance is obviously advertised.
 +
====Influence and take-up====
 +
The Royal Library of Denmark and the State and University Library have used NetarchiveSuite to harvest the Danish world wide web since 2005. The software is also used by the Bibliothèque nationale de France, and Österreichische Nationalbibliothek.
  
= Description =
 
NetarchiveSuite is a web archiving software package which is used to plan, schedule and run web harvests of parts of the Internet. It scales to a wide range of tasks, from small, thematic harvests (e.g. related to special events, or special domains) to harvesting and archiving the content of an entire national domain. Developed by The Royal Library, Denmark.
 
  
 
= User Experiences =
 
= User Experiences =
Line 22: Line 36:
  
 
= Development Activity =
 
= Development Activity =
 +
<!-- Provide *evidence* of development activity of the tool. For example, RSS feeds for code issues or commits. -->
 +
Version 4.1 was released 27th May 2013. Version 4.2 should be released at the end of June 2013.
 +
The software appears to be an integral part of the Libraries&rsquo; ongoing web archiving effort, indicating continuing support. The website includes a roadmap for the software.
 +
 +
All development activity is visible on GitHub: http://github.com/netarchivesuite/netarchivesuite/commits
 +
 +
 +
=== Release Feed ===
 +
Below the last 3 release feeds:
 +
<rss max=3>https://github.com/netarchivesuite/netarchivesuite/releases.atom</rss>
 +
 
 +
 +
=== Activity Feed ===
 +
Below the last 5 commits:
 +
<rss max=5>https://github.com/netarchivesuite/netarchivesuite/commits/master.atom</rss>

Latest revision as of 16:05, 26 November 2021



NetarchiveSuite
NetarchiveSuite is a web archiving software package designed to plan, schedule and run web harvests of parts of the Internet.
Homepage:https://sbforge.org/display/NAS/NetarchiveSuite
License:GNU Lesser General Public License
Platforms:Linux
Function:Web Capture
Content type:Web



Description[edit]

The NetarchiveSuite is a web archiving software package designed to plan, schedule and run web harvests of parts of the Internet. It is also serves as an archiving platform for the collected materials, storing and performing consistency checks of the data.

Provider[edit]

The Royal Library of Denmark and the State and University Library of Denmark

Platform and interoperability[edit]

Netarchive requires a computer running a Linux operating system with Sun Java 1.6, as well as Java Messaging Service. The software uses Heritrix as its web crawler.

Functional notes[edit]

The NetarchiveSuite is split into four main modules: three modules corresponding to processes of harvesting, archiving and accessing materials, and one module to coordinate functions. The Harvester module can organise both snapshot and recurring harvests; it supports packaging metadata about the harvest together with the harvested data. The Archive module advertises bit consistency checks and the ability to support distributed batch jobs; it also supports storage. The Access Module uses a proxy solution to give access to the material. As the software uses Heritrix for its crawls, the materials collected take the form of arc files (not to be confused with ARC files).

Documentation and user support[edit]

Netarchive’s website includes extensive documentation, including an Overview and Quick Start Manual. Detailed guidance is found in the Configuration, Installation, and User Manuals. Developer guidance is found in the System Design and Additional Tools Manual. The website also points to a new Wiki that as of writing is in some places incomplete, and in general rather difficult to navigate. The project supports four mailing lists, all of which are currently active: -announce; -curator; -devel; and -users. The site also includes a contact page with email addresses for individuals at KB, SB, BnF, and ONB.

Usability[edit]

The Suite advertises its ability to be used by librarians rather than systems administrators; its Quickstart installation option is designed to take an hour, and the GUI for ongoing use is extremely simple.

Expertise required[edit]

With any web archiving project, deep understanding of the project’s scope and collections policy is essential in order to set up appropriate targets.

Standards compliance[edit]

No standards compliance is obviously advertised.

Influence and take-up[edit]

The Royal Library of Denmark and the State and University Library have used NetarchiveSuite to harvest the Danish world wide web since 2005. The software is also used by the Bibliothèque nationale de France, and Österreichische Nationalbibliothek.


User Experiences[edit]

Development Activity[edit]

Version 4.1 was released 27th May 2013. Version 4.2 should be released at the end of June 2013. The software appears to be an integral part of the Libraries’ ongoing web archiving effort, indicating continuing support. The website includes a roadmap for the software.

All development activity is visible on GitHub: http://github.com/netarchivesuite/netarchivesuite/commits


Release Feed[edit]

Below the last 3 release feeds:

2021-08-19 10:03:38
[tag:github.com,2008:Repository/19818273/netarchivesuite-7.2 netarchivesuite-7.2]
by csrster
2021-07-06 10:50:26
[tag:github.com,2008:Repository/19818273/netarchivesuite-7.1 netarchivesuite-7.1]
by csrster
2021-03-19 08:54:06
[tag:github.com,2008:Repository/19818273/netarchivesuite-7.0 netarchivesuite-7.0]
by csrster


Activity Feed[edit]

Below the last 5 commits:

2022-01-25 14:37:53
[tag:github.com,2008:Grit::Commit/534e263a5f491acddaf2697a61d51df56cc6fda1 Revert "Switched field to search for domain in crawl log"]
by csrster https://github.com/csrster
2022-01-25 13:39:05
[tag:github.com,2008:Grit::Commit/f0e2fec464f219bbccc5ad1a8c47bb5ab34b5f79 Switched field to search for domain in crawl log]
by csrster https://github.com/csrster
2022-01-25 13:17:20
[tag:github.com,2008:Grit::Commit/6c7a5062550049c6f6aab16d58f275970c3f5ef1 Added debugging to crawllog searching]
by csrster https://github.com/csrster
2022-01-25 13:05:19
[tag:github.com,2008:Grit::Commit/31fc8d34137a414d5c746069437456712678d46b Fixed a tempfile name]
by csrster https://github.com/csrster
2022-01-25 09:14:21
[tag:github.com,2008:Grit::Commit/c5d6c5d38bb0cfe52a15493ccde9fa24a07e6a8d Fixed some issues with holding large hadoop result sets in memory]
by csrster https://github.com/csrster