Workflow:Cloud-based preservation and access workflow for MXF and MPG video

Cloud-based preservation and access workflow for MXF and MPG video
Status:	Production
Tools:	Goobi AWS S3 AWS Lambda AWS MediaConvert Wellcome Storage Service DDS DLCS Bag Bagit IIIF IIIF manifest
Input:	.mxf + .mp4 with .jpg poster image,and optional .pdf transcript or .mpg with .jpg poster image and optional .pdf transcript
Output:	Low res .mp4 with .jpg poster image, optional .pdf transcript, and IIIF manifest
Organisation:	Wellcome Collection

Workflow Description

MXF Video Pre-Ingest Workflow

The film pre-ingest workflow converts the .mxf video to .mp4 and moves all files from a public bucket to a private bucket

1. Vendor uploads Film Batch X consisting of .mxf and .jpg post images to the wellcomecollection-digitisation-transfer bucket in AWS S3
2. The files arriving in the bucket activate two Lambdas simultaneously
3. The A/V Pre-Ingest Copy Lambda copies the .mxf and .jpg files over to a different bucket, the wellcomecollection-av-digitisation bucket, which can only be accessed by Wellcome Staff
4. The A/V Pre-Ingest Convert Lambda sends the .mxf to AWS MediaConvert to create an .mp4
5. The .mp4 is delivered to the wellcomecollection-av-digitisation bucket alongside the .mxf and .jpgs. The .mp4 will be QA’d and the files remain here until ingest.

MXF/MP4 or MPG Video Workflow

This workflow can be used for either and .mxf/.mp4 and .jpg poster image or an .mpg and .jpg poster image. An optional .pdf transcript can be added to either type of ingest.

1. Copy over an .mxf and .mp4 or an .mpg, with the accompanying .jpg poster image and .pdf (*optional as not always available) transcript from their original bucket into a folder created in the wellcomecollection-workflow-upload bucket. The folder name should match the name of the process title for the item in Goobi. The process title will have been created in Goobi by loading the marc.xml prior to ingest in the **bibliographic import step.
2. The upload of the files will trigger the Goobi Lambda which queries Goobi for a process title that matches the name of the folder and will send the files to the process if a match is found
3. In Goobi, the user can check that the files are copied over and release the video data import step
4. Goobi automatically moves the files into appropriate internal folders (access, preservation, poster, transcript) in preparation for writing the METS filegroups and usage attributes
5. At Edit METS, the user must select the license and access status for the film to be written to the METS
6. Goobi continues the workflow automatically
   • writing PREMIS data to the METS
   • checking if the item is a single item or multiple manifestation
   • creating a bag
7. The bag is sent to the Wellcome Storage Service where it is verified and stored. The .mxf files are automatically life cycled to Glacier Deep Store. The storage sends a callback to Goobi to verify the bag has been stored successfully
8. When Goobi gets the call back, it calls the DDS API. DDS then reads the METS in storage and starts writing a IIIF manifest while DLCS begins making an access copy of the high res MP4 or MPG
9. When the IIIF manifest and access copies are ready, wellcomecollection.org/collection starts displaying the video