Editing Workflow:Browsertrix-crawler Workflow

Jump to navigation Jump to search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 20: Line 20:
 
<!-- Describe what your workflow is for - i.e. what it is designed to achieve, what the organisational context of the workflow is, and what content it is designed to work with -->
 
<!-- Describe what your workflow is for - i.e. what it is designed to achieve, what the organisational context of the workflow is, and what content it is designed to work with -->
  
The purpose of this workflow is determine whether a site is suitable for capture with [https://github.com/webrecorder/browsertrix-crawler Browsertrix Crawler] and if so, run a Browsertrix crawl. The crawl is then subject to Quality Assurance. If the crawl is found to be unsatisfactory, Browsertrix settings are adjusted and the crawl is run again, with this process potentially being repeated several times until a satisfactory crawl is completed. If no satisfactory crawl can be made in this way, the site will be captured with Conifer.  
+
The purpose of this workflow is determine whether a site is suitable for capture with [https://github.com/webrecorder/browsertrix-crawler Browsertrix Crawler] and if so, run a Browsertrix crawl. The crawl is then subject to Quality Assurance. If the crawl is found to be unsatisfactory Browsertrix settings are adjusted and the crawl is run again, with this process potentially being repeated several times until a satisfactory crawl is completed. If not satisfactory crawl can be made in this way, the site will be captured with Conifer.  
  
 
The steps are as follows:
 
The steps are as follows:
Line 28: Line 28:
 
2. The site is assessed to determine which capture method is suitable. At this point we look at:
 
2. The site is assessed to determine which capture method is suitable. At this point we look at:
  
* How large the site is
+
    * How large the site is
* Does the site contain interactive content?
+
    * Does the site contain interactive content?
* What is the planned capture frequency? (if the proposed capture is very frequent we may be more likely to use an in-house tool like Browsertrix to reduce costs)
+
    * What is the planned capture frequency? (if the proposed capture is very frequent we may be more likely to use an in-house tool like Browsertrix to reduce costs)
* What level of fidelity is required
+
    * What level of fidelity is required
* Have previous crawls of the site been attempted and what was the outcome
+
    * Have previous crawls of the site been attempted and what was the outcome
  
 
3. An initial decision of what capture technology to use is made
 
3. An initial decision of what capture technology to use is made
 
4. If Browsertrix is selected, an initial config will be generated based on our knowledge of the site (in our case this would be generated from our in-house database).
 
 
5. An initial Browsertrix is then run, using default settings.
 
 
6. The resulting WARC is uploaded to Conifer for Quality Assurance.
 
 
7. If the level of quality is found to be unsatisfactory, Browsertrix settings will be tweaked, in preparation for running another crawl. At this point we will review settings for:
 
 
* Scoping rules
 
* Seeds
 
* Depth
 
* Behaviours
 
 
8. A new Browsertrix crawl will then be run with the new configuration.
 
 
9. This process is repeated until the crawl is found to be satisfactory in QA. If we are unable to complete a satisfactory crawl, at this point we will look at capturing the site with Conifer.
 
 
10. If a satisfactory Browsertrix crawl has been completed, we download the WARC from Conifer.
 
 
11. The WARC is then prepared to be ingested to our collection. For us this involves renaming the file and compressing it.
 
 
12. The WARC is then uploaded to our collection and published.
 
  
 
==Evaluation/Review==
 
==Evaluation/Review==
 
<!-- How effective was the workflow? Was it replaced with a better workflow? Did it work well with some content but not others? What is the current status of the workflow? Does it relate to another workflow already described on the wiki? Link, explain and elaborate -->
 
<!-- How effective was the workflow? Was it replaced with a better workflow? Did it work well with some content but not others? What is the current status of the workflow? Does it relate to another workflow already described on the wiki? Link, explain and elaborate -->
 
This workflow is currently being used in production at the UK Government Web Archive. We have found it to be successful with a variety of content types. However, we are still currently refining our process and experimenting with different Browsertrix configurations.
 
  
 
==Further Information==
 
==Further Information==
Line 70: Line 45:
  
 
<!-- Note that your workflow will be marked with a CC3.0 licence -->
 
<!-- Note that your workflow will be marked with a CC3.0 licence -->
 
Information on getting started with Browsertrix Crawler can be found on the project's [https://github.com/webrecorder/browsertrix-crawler GitHub page].
 

Please note that all contributions to COPTR are considered to be released under the Attribution-ShareAlike 3.0 Unported (see COPTR:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To edit this page, please answer the question that appears below (more info):

Cancel Editing help (opens in new window)

Template used on this page: