Edit Tool: Web Archive Discovery

To edit this page, please answer the question that appears below (more info):

What short name does OAIS use for an information package that is used for dissemination?

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

Image: In another tab, use the "Upload file" link to upload a logo image. Paste the name of the image here:
Purpose* A single sentence describing the function of the tool. Keep it factual and concise:	Indexing and discovery tools for web archives.
Homepage The URL of the homepage:
Sourcecode The URL for the source code:
License:
Cost:
Platforms:
Language:
Wikidata ID:
Input formats Start typing and select from the available list. Only add a new format if necessary:
Output Formats: Start typing and select from the available list. Only add a new format if necessary:
Function*: Start typing and select one or more functions from the available list. Only add a new function if absolutely necessary:
Content: Start typing and select one or more content types from the available list. Only add a new content type if absolutely necessary:

OpenHub ID:
Mailing_lists:
Releases_rss:
Issues_rss:

Free text:

== Description ==  Full-text indexing system, using Apache Solr as the search back-end. Supports command-line or large-scale map-reduce (Hadoop) processing of ARC and WARC files. Also integrates file format analysis and scans for some known preservation risks. It runs format identification using both [[Apache Tika]] and [[DROID]], and also extracts filename extensions and the first few bytes of each resource, to enable analysis of formats unknown to those tools. If can also be configured to use [[Apache PDFBox]]'s Preflight tool to scan PDF's for PDF/A violations, and record them in the index for analysis. It also records any parsing errors or other access problems it discovers, which can help find problematic resources. For more information, see the [https://github.com/ukwa/webarchive-discovery/wiki/Features Features page of the tool's wiki]. == User Experiences ==  * Used by the [http://www.webarchive.org.uk/ UK Web Archive] to provide access to their collections. Some details [http://blogs.bl.uk/webarchive/2017/06/revitalising-the-uk-web-archive.html in this blog post]. Feel free to contact [https://twitter.com/UKWebArchive @UKWebArchive] for more information. * Also used by a number of other web archives, and by the [https://github.com/archivesunleashed/warclight WARCLight] project. = Development Activity =  All development activity is visible on GitHub: http://github.com/ukwa/webarchive-discovery There is also a #webarchive-discovery channel on the IIPC Slack service. Contact https://twitter.com/NetPreserve for details. === Release Feed === Below the last 3 release feeds: <rss max=3>https://github.com/ukwa/webarchive-discovery/releases.atom</rss> === Activity Feed === Below the last 5 commits: <rss max=5>https://github.com/ukwa/webarchive-discovery/commits/master.atom</rss>

Summary:

Cancel

Edit Tool: Web Archive Discovery

Navigation menu

Search