Workflow:Digital archiving workflow (high-level)

Digital archiving workflow (high-level)
Status:	Experimental
Tools:	Archivematica Bagger BitCurator CSV Validator DROID (Digital Record Object Identification) JHOVE (Harvard Object Validation Environment) TeraCopy TreeSize PAIMAS Bulk Extractor
Input:	Request for deposit of digital materials by transfer, donation, or purchase to the University Archives.
Output:	AIPs stored in Archival storage and/or DIPs uploaded to an Access system; or a Re-evaluation process in cases where digital materials are not deemed appropriate for acquisition.
Organisation:	Archives & Special Collections (ASC), University of Glasgow

Workflow Description

START

A request for deposit of digital materials (by transfer, donation, or purchase) is submitted to the University Archives.

PRE-ACQUISITION APPRAISAL

Processes for evaluating whether a deposit request will be accepted by the University Archives.

1. Check the deposit request against the Archives & Special Collections collection development policy - does the request align with the core collecting areas?
2. Prepare a records survey and/or pre-accession assessment of the proposed deposit.
3. Evaluate the results of the records survey against the Appraisal & Retention policy, which:
   • addresses issues pertaining to selection and long-term retention of digital objects
   • extends the collections development policy
   • ensures that retention decisions are balanced between value and capacity to preserve for the long-term; and
   • provides clarity to avoid assumptions over digital storage costs and availability.
4. Decide whether the requested deposit aligns with policies:
   • If not, re-evaluate acquisition and/or reject deposit.
   • If yes, proceed to Acquisition.

ACQUISITION

Processes for acquiring digital materials by transfer, donation, or purchase.

1. Follow the methodology in the Space data and information transfer systems — Producer-archive interface — Methodology abstract standard (PAIMAS) ISO 20652:2006 standard. The standard "identifies, defines and provides structure to the relationships and interactions between an information producer and an archive. It defines the methodology for the structure of actions that are required from the initial time of contact between the producer and the archive until the objects of information are received and validated by the archive." (ISO). For more information, see Acquisition and appraisal, Digital Preservation Handbook (DPC).
2. Follow the Accepted file formats/media procedure, which:
   • Specifies decisions on file formats and/or storage media that the University Archives will accept.
   • Aligns with preservation planning decisions for format normalisation; and capability to access storage media (esp. legacy media, e.g. floppy or zip disks).
   • For a summary table of options, see the Acquisition workflow section in Acquisition and appraisal, Digital Preservation Handbook (DPC).
3. For acquisitions deposited in physical storage media:
   • Place all incoming items in quarantine area on arrival, inspect for pest infestation and mould; and follow handling and moving procedures.
   • Create physical conservation and preservation report, documenting all actions on the acquired media.
   • Proceed to Accessioning.
4. For acquisitions deposited digitally (e.g. file transfer):
   • Proceed to Accessioning.

ACCESSIONING

Process of formally registering deposit into the University Archives, which enables intellectual control over the digital materials.

1. Generate a unique accession number, based on the University Archives' archival processing guidelines.
2. Check digital materials for viruses. See Select and transfer workflow in the TNA guide (section 1.3) for a reasonable process. Depending on the results of virus checks:
   • if virus is found, quarantine and attempt removal; and/or request clean versions from source. If all these fail, prepare a report documenting actions and re-evaluate acquisition.
   • if virus-free, proceed to
3. Generate a file manifest that contains folder and file listings, as well as metadata for the acquired digital materials.
4. List the accession number into the University's Collections Management System for cataloguing. Cataloguing processes include decisions over the system of arrangement and level of description that will be used for the deposited materials; definition of access and reproduction conditions; and documentation via descriptive metadata.
5. Decide whether archival forensic processing is required:
   • If Yes, follow the Archival Forensics workflow.
6. Proceed to Transfer.

TRANSFER

Processes for transferring digital materials to the University Archives.

1. Choose a method for transferring files:
   • Copy files from source media. OR
   • Create a disk image from source storage media. OR
   • Request that digital materials are submitted as a BagIt container.
2. Generate checksums to verify data integrity during transmission and/or storage:
   • For digital acquisitions in situ, it might be appropriate to first store files in a temporary location for virus and/or integrity checks, before transferring to process store. Not applicable to all scenarios.
   • Transfer digital materials to Process store (e.g. network drive).
3. Use tools to identify file types and validate file formats (e.g. DROID, JHOVE), then proceed to Appraisal.

APPRAISAL

Processes for selecting digital materials from a transfer, which are deemed as appropriate for long-term preservation and archiving.

1. Perform appraisal actions that have been deemed necessary (if any), including at a minimum:
   • Selecting which files from a transfer should be added to a SIP and how these files should be arranged within the SIP.
   • Removing duplicates, e.g. by comparing checksums. See Ingest workflow in the TNA guide (section 2.5) for a reasonable process.
2. Complete legal checks, including:
   • Checks against GDPR principles and possible exemptions for archiving in the public interest. The National Archives provide key information in their Archives and GDPR: frequently asked questions page.
   • Checks for Personal Identifiable Information and other restricted/confidential information included in the deposited files that should not be made puclicly accessible.
3. Proceed to Ingest.

INGEST

Processes for generating SIPs from transferred digital objects, normalising for preservation and/or access; packaging into AIPs for archival storage; and/or DIPs for access systems.

1. Add any descriptive metadata (created in Cataloguing) and rights metadata (created during Appraisal) in the SIP.
2. Choose a normalisation strategy for file formats (based on preservation planning).
3. Archival storage solutions should follow the Principles for using IT storage systems for digital preservation defined in the DPC Handbook.
4. Periodic monitoring should include:
   • File integrity checks by monitoring checksum changes.
   • Format obsolescence checks, by monitoring changes to preservation planning and normalisation strategies.

Purpose, Context and Content

The workflow aims to formalise digital archiving activities, and incorporate digital preservation processing requirements. It has been designed as an experimental tool for testing production-level digital preservation processing at the University of Glasgow Archives. Production-level here describes in vivo test conditions in terms of:

• data volume per transfer and ingest
• file format types and variability
• acquisition and accessioning procedures
• currently available technical infrastructure, including computing, local/shared storage and network speed/capacity on campus.

Evaluation/Review

The workflow has been successful in guiding tests of the University Archives' current digital preservation capability; and evaluating the suitability of our current setup for production-level digital preservation. Currently in version 0.5, it is continuously amended until reaching version 1.0 after which it will be reviewed on an annual basis.

Further Information

• Digital Preservation policy, University of Glasgow
• Collections development policy, Archives & Special Collections, University of Glasgow
• Digital Preservation Handbook, DPC
• Archival Forensics workflow, University of Glasgow
• Digital preservation workflows, The National Archives

Leo Konstantelos (talk) 12:12, 26 May 2023 (UTC)