DROID Siegfried Sqlite Analysis Engine

From COPTR
Jump to navigation Jump to search



Format Identification Analysis and Reporting
Homepage:https://github.com/exponential-decay/demystify
License:Open source (see URL above)
Platforms:sqlite + Python + text/html
Function:Metadata Extraction,Content Profiling,De-Duplication




Description[edit]

Now known as "Demystify" with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of DROID CSV export files, Siegfried YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with rsync to isolate these sets from one-another to be more flexible to work with.

User Experiences[edit]

Development Activity[edit]

All development activity is visible on GitHub: http://github.com/ross-spencer/droid-sqlite-analysis/commits

Release Feed[edit]

Below the last 3 release feeds:

2023-09-03 19:04:14
[tag:github.com,2008:Repository/15066530/v2.0.0rc4 v2.0.0rc4]
by ross-spencer
2022-09-08 12:59:39
[tag:github.com,2008:Repository/15066530/v2.0.0rc3 v2.0.0rc3]
by ross-spencer
2022-09-08 11:23:45
[tag:github.com,2008:Repository/15066530/v2.0.0rc2 v2.0.0rc2]
by ross-spencer

Activity Feed[edit]

Below the last 5 commits:

2023-09-10 15:34:40
[tag:github.com,2008:Grit::Commit/c80d5f2bd4d64e6189b04ecb025ab72d77c429dd Add logging to help users with encoding issues]
by ross-spencer https://github.com/ross-spencer
2023-09-05 06:29:26
[tag:github.com,2008:Grit::Commit/963927850ba8cf235ef91b9e5ad9d427c1f588b4 Move linting into its own workflow]
by ross-spencer https://github.com/ross-spencer
2023-09-05 06:19:43
[tag:github.com,2008:Grit::Commit/7b79eb78279a33600acdf080807d8ada3545eb44 Update linting tooling]
by ross-spencer https://github.com/ross-spencer
2023-09-05 06:07:21
[tag:github.com,2008:Grit::Commit/1ed059fba6e8c50cdf4d8835141df550e2005819 Update workflow to run linting on Ubuntu-latest]
by ross-spencer https://github.com/ross-spencer
2023-09-03 18:57:01
[tag:github.com,2008:Grit::Commit/056e3061f57fbe8b5fdad75bfc655fd147743bdf Remove support for py37 => py38]
by ross-spencer https://github.com/ross-spencer