COPTR - User contributions [en-gb]

Workflow:Quality Assurance: Iterative Seed Issue Decision Tree

2023-06-15T19:21:02Z

Meghly: /* Purpose, Context and Content */

{{Infobox COW
|status=Production
|tools=Heritrix, Webrecorder, OpenWayback, Pywb, OutbackCDX, CDX
|input=Web Archives visual replay and crawl report data
|output=Adjustments to seed URLs and scopes; the results of a future crawl; documentation
|organisation=Library of Congress
|organisationurl=https://www.loc.gov
}}
==Workflow Description==


[[File:Seed-Issue-Decision-Tree-20230525-ScreenSized.jpeg|Quality Assurance: Seed Issue Decision Tree]] 



==Purpose, Context and Content==
This decision tree is meant to provide an outline for completing seed-by-seed quality assurance, beginning with data from Heritrix crawl reports and iterating on input, crawler, or other variables until either the captures improve or a seed URL is deemed non-archivable. At our organization, this workflow is conducted entirely by the Web Archiving Team, the technical team which facilitates the contracted crawling, the use of our curatorial workflow tool Digiboard, and the ingest and access to the web archives (this latter in conjunction with our Office of the Chief Information Officer or OCIO).

==Evaluation/Review==

This workflow or some version of it has been in place for a long time in our program. It is labor intensive and certainly cannot be completed on 100% of the materials going into the crawls or 100% of the materials coming out of the crawls. That said, close attention to detail may yield results, even if QA cannot be completed all the time on everything.

==Further Information==

Workflow:Quality Assurance: Iterative Seed Issue Decision Tree

2023-06-02T20:49:26Z

Meghly: Created page with "{{Infobox COW |status=Production |tools=Heritrix, Webrecorder, OpenWayback, Pywb, OutbackCDX, CDX |input=Web Archives visual replay and crawl report data |output=Adjustments t..."

{{Infobox COW
|status=Production
|tools=Heritrix, Webrecorder, OpenWayback, Pywb, OutbackCDX, CDX
|input=Web Archives visual replay and crawl report data
|output=Adjustments to seed URLs and scopes; the results of a future crawl; documentation
|organisation=Library of Congress
|organisationurl=https://www.loc.gov
}}
==Workflow Description==


[[File:Seed-Issue-Decision-Tree-20230525-ScreenSized.jpeg|Quality Assurance: Seed Issue Decision Tree]] 



==Purpose, Context and Content==
This decision tree is meant to provide an outline for completing seed-by-seed quality assurance, beginning with data from Heritrix crawl reports and iterating on input, crawler, or other variables until either the captures improve or a seed URL is deemed non-archivable.

==Evaluation/Review==

This workflow or some version of it has been in place for a long time in our program. It is labor intensive and certainly cannot be completed on 100% of the materials going into the crawls or 100% of the materials coming out of the crawls. That said, close attention to detail may yield results, even if QA cannot be completed all the time on everything.

==Further Information==

File:Seed-Issue-Decision-Tree-20230525-ScreenSized.jpeg

2023-06-02T20:43:19Z

Meghly: Decision Tree illustration of dealing with seed issues in Web Archiving.

== Summary ==
Decision Tree illustration of dealing with seed issues in Web Archiving.

Workflow:Web Archiving Capture Assessment Response Processing Workflow

2023-06-02T20:40:57Z

Meghly: Created page with "{{Infobox COW |status=Experimental |tools=Jira, Confluence, email |input=Visual curatorial assessments of web archives captures. |output=Jira tickets, QA on web archives, emai..."

{{Infobox COW
|status=Experimental
|tools=Jira, Confluence, email
|input=Visual curatorial assessments of web archives captures.
|output=Jira tickets, QA on web archives, emails.
|organisation=Library of Congress
|organisationurl=https://loc.gov
}}
==Workflow Description==

[[File:Capture-Assessment-Processing-20230525.jpeg|Capture Assessment Response Processes]] 



==Purpose, Context and Content==

Currently at the LC, our systems for web archives curatorial workflows (Digiboard), Web Archiving Team (WAT) documentation and work processes (Confluence/Jira), Web Archives Replay (OpenWayback/Pywb), and capture (Digiboard for curating, a vendor for at-scale crawling, other ticketing systems to submit seed lists, Heritrix and browser-based crawlers for capture, and subsequent crawl reports, storage, etc.) do not speak to each other. A process was designed whereby curatorial staff, digital technicians, acquisitions specialists, and other designated individuals could review captures of given sites and send responses in a structured way to the WAT. That process is called Capture Assessment. For our purposes, Quality Assurance (QA) is the technical process completed by WAT and the Library crawl vendor to iteratively adjust crawl parameters and other variables in order to improve capture quality over time. Capture Assessment response processing, featured in this workflow, is the workflow through which the WAT review capture assessments and complete QA as-needed.

==Evaluation/Review==

This workflow is relatively new and experimental in our program as of 6/2/2023. The workflow replaced a previous workflow where a single yes/no question was asked in the banner of Wayback, "is this a good capture?", and no notification went to the Web Archiving Team when curatorial staff submitted an answer. While the responses were recorded in a module in Digiboard, it was not a reliable tool and did not always yield detailed or usable results.

==Further Information==

File:Capture-Assessment-Processing-20230525.jpeg

2023-06-02T20:40:28Z

Meghly: Meghly uploaded a new version of File:Capture-Assessment-Processing-20230525.jpeg

== Summary ==
WAT Scrum/administrative/Quality Assurance processes for curatorial capture assessments.

File:Capture-Assessment-Processing-20230525.jpeg

2023-06-02T20:26:44Z

Meghly: WAT Scrum/administrative/Quality Assurance processes for curatorial capture assessments.

== Summary ==
WAT Scrum/administrative/Quality Assurance processes for curatorial capture assessments.

Workflow:Web Archiving Quality Assurance Lifecycle

2023-06-02T20:20:15Z

Meghly:

{{Infobox COW
|status=Production
|tools=Heritrix, AWS, Pywb, OpenWayback, CDX
|input=Input: Seed URLs, SURTs, Exclude lists
|output=WARCs, CDX files, Curatorial Data
|organisation=Library of Congress
|organisationurl=https://www.loc.gov/
}}
==Workflow Description==

[[File:QA-Life-Cycle-20230525.jpeg|Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.]] 



==Purpose, Context and Content==

This workflow is an illustration of the life cycle of the quality assurance (QA) process in place in the Web Archiving Program at the Library of Congress as of March 25th, 2023. This work flow is meant to be vendor-agnostic, but assumes cloud-based web archiving services and cloud-based transfer. It is designed for an iterative crawling environment whereby adjustments to seed URLs, scopes, SURTs, regex, etc. are done from crawl to crawl, rather than having missing elements patched in; and for a large scale operation. There is a mix of open source and non-open source technologies in play, and the QA itself does not rely on a single technology, but require Web Archives crawl and replay tech.

==Evaluation/Review==


==Further Information==

File:QA-Life-Cycle-20230525.jpeg

2023-06-02T20:19:38Z

Meghly: Meghly uploaded a new version of File:QA-Life-Cycle-20230525.jpeg

== Summary ==
Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.

Workflow:Web Archiving Quality Assurance Lifecycle

2023-06-02T20:16:50Z

Meghly: /* Workflow Description */

{{Infobox COW
|status=Production
|tools=Heritrix, AWS, Pywb, OpenWayback, CDX
|input=Input: Seed URLs, SURTs, Exclude lists
|output=WARCs, CDX files, Curatorial Data
|organisation=Library of Congress
|organisationurl=https://www.loc.gov/
}}
==Workflow Description==



[[File:QA-Life-Cycle-20230525.jpeg|Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.]] 



==Purpose, Context and Content==

This workflow is an illustration of the life cycle of the quality assurance (QA) process in place in the Web Archiving Program at the Library of Congress as of March 25th, 2023. This work flow is meant to be vendor-agnostic, but assumes cloud-based web archiving services and cloud-based transfer. It is designed for an iterative crawling environment whereby adjustments to seed URLs, scopes, SURTs, regex, etc. are done from crawl to crawl, rather than having missing elements patched in; and for a large scale operation. There is a mix of open source and non-open source technologies in play, and the QA itself does not rely on a single technology, but require Web Archives crawl and replay tech.

==Evaluation/Review==


==Further Information==

Workflow:Web Archiving Quality Assurance Lifecycle

2023-06-02T20:16:27Z

Meghly:

{{Infobox COW
|status=Production
|tools=Heritrix, AWS, Pywb, OpenWayback, CDX
|input=Input: Seed URLs, SURTs, Exclude lists
|output=WARCs, CDX files, Curatorial Data
|organisation=Library of Congress
|organisationurl=https://www.loc.gov/
}}
==Workflow Description==



[[File:577px-QA-Life-Cycle-20230525.jpeg|Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.]] 



==Purpose, Context and Content==

This workflow is an illustration of the life cycle of the quality assurance (QA) process in place in the Web Archiving Program at the Library of Congress as of March 25th, 2023. This work flow is meant to be vendor-agnostic, but assumes cloud-based web archiving services and cloud-based transfer. It is designed for an iterative crawling environment whereby adjustments to seed URLs, scopes, SURTs, regex, etc. are done from crawl to crawl, rather than having missing elements patched in; and for a large scale operation. There is a mix of open source and non-open source technologies in play, and the QA itself does not rely on a single technology, but require Web Archives crawl and replay tech.

==Evaluation/Review==


==Further Information==

Workflow:Web Archiving Quality Assurance Lifecycle

2023-06-02T20:16:01Z

Meghly:

{{Infobox COW
|status=Production
|tools=Heritrix, AWS, Pywb, OpenWayback, CDX
|input=Input: Seed URLs, SURTs, Exclude lists
|output=WARCs, CDX files, Curatorial Data
|organisation=Library of Congress
|organisationurl=https://www.loc.gov/
}}
==Workflow Description==



[[File:QA-Life-Cycle-20230525.jpeg/577px-QA-Life-Cycle-20230525.jpeg|Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.]] 



==Purpose, Context and Content==

This workflow is an illustration of the life cycle of the quality assurance (QA) process in place in the Web Archiving Program at the Library of Congress as of March 25th, 2023. This work flow is meant to be vendor-agnostic, but assumes cloud-based web archiving services and cloud-based transfer. It is designed for an iterative crawling environment whereby adjustments to seed URLs, scopes, SURTs, regex, etc. are done from crawl to crawl, rather than having missing elements patched in; and for a large scale operation. There is a mix of open source and non-open source technologies in play, and the QA itself does not rely on a single technology, but require Web Archives crawl and replay tech.

==Evaluation/Review==


==Further Information==

Workflow:Web Archiving Quality Assurance Lifecycle

2023-06-02T20:15:02Z

Meghly: Created page with "{{Infobox COW |status=Production |tools=Heritrix, AWS, Pywb, OpenWayback, CDX |input=Input: Seed URLs, SURTs, Exclude lists |output=WARCs, CDX files, Curatorial Data |organisa..."

{{Infobox COW
|status=Production
|tools=Heritrix, AWS, Pywb, OpenWayback, CDX
|input=Input: Seed URLs, SURTs, Exclude lists
|output=WARCs, CDX files, Curatorial Data
|organisation=Library of Congress
|organisationurl=https://www.loc.gov/
}}
==Workflow Description==



[[File:QA-Life-Cycle-20230525.jpeg|Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.]] 



==Purpose, Context and Content==

This workflow is an illustration of the life cycle of the quality assurance (QA) process in place in the Web Archiving Program at the Library of Congress as of March 25th, 2023. This work flow is meant to be vendor-agnostic, but assumes cloud-based web archiving services and cloud-based transfer. It is designed for an iterative crawling environment whereby adjustments to seed URLs, scopes, SURTs, regex, etc. are done from crawl to crawl, rather than having missing elements patched in; and for a large scale operation. There is a mix of open source and non-open source technologies in play, and the QA itself does not rely on a single technology, but require Web Archives crawl and replay tech.

==Evaluation/Review==


==Further Information==

File:QA-Life-Cycle-20230525.jpeg

2023-06-02T20:06:30Z

Meghly: Meghly uploaded a new version of File:QA-Life-Cycle-20230525.jpeg

== Summary ==
Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.

File:QA-Life-Cycle-20230525.jpeg

2023-06-02T19:52:41Z

Meghly: Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.

== Summary ==
Quality Assurance Life Cycle at Library of Congress as of March, 25 2023.