Difference between revisions of "Demystify"
Ross-spencer (talk | contribs) (Update Github link) |
Ross-spencer (talk | contribs) (Updated summary information and introduction) |
||
| Line 1: | Line 1: | ||
{{Infobox tool | {{Infobox tool | ||
| − | |purpose=Analysis and | + | |purpose=Format Identification Analysis and Reporting |
|homepage=https://github.com/exponential-decay/demystify | |homepage=https://github.com/exponential-decay/demystify | ||
|license=Open source (see URL above) | |license=Open source (see URL above) | ||
| Line 9: | Line 9: | ||
== Description == | == Description == | ||
<!-- Describe the what the tool does, focusing on it's digital preservation value. Keep it factual. --> | <!-- Describe the what the tool does, focusing on it's digital preservation value. Keep it factual. --> | ||
| − | + | Now known as "Demystify" with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions. | |
| − | The tool provides archivist definitions for each of the sections output; these definitions are | + | The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx |
The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with. | The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with. | ||
Revision as of 11:54, 21 January 2022
Description
Now known as "Demystify" with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of DROID CSV export files, Siegfried YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.
The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx
The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with rsync to isolate these sets from one-another to be more flexible to work with.
User Experiences
- Blog entries from the tool author, Ross Spencer:
- [2014-06-03] Describing the creation and purpose of the tool.
- [2015-08-25] Using the output of the tool to create a digital preservation rogues gallery.
- [2016-05-23] The integration of Siegfried output for consistent and repeatable reporting.
- [2016-05-24] Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.
Development Activity
All development activity is visible on GitHub: http://github.com/ross-spencer/droid-sqlite-analysis/commits
Release Feed
Below the last 3 release feeds:
- 2024-05-05 15:18:04
- [tag:github.com,2008:Repository/15066530/v2.0.0 v2.0.0]
- by ross-spencer
- 2024-04-14 13:53:10
- [tag:github.com,2008:Repository/15066530/v2.0.0rc7 v2.0.0rc7]
- by ross-spencer
- 2024-03-25 20:52:06
- [tag:github.com,2008:Repository/15066530/v2.0.0rc6 v2.0.0rc6]
- by ross-spencer
Activity Feed
Below the last 5 commits:
- 2024-05-05 15:16:45
- [tag:github.com,2008:Grit::Commit/8433426905e657dd90df739878dbe12ee0761aeb Demystify 2.0.0]
- by ross-spencer https://github.com/ross-spencer
- 2024-04-14 13:51:43
- [tag:github.com,2008:Grit::Commit/9060b9a424f22d1d60ad15ee0f1f7a03aa511f2a Update National Archives URI]
- by ross-spencer https://github.com/ross-spencer
- 2024-03-25 20:51:14
- [tag:github.com,2008:Grit::Commit/edcadd0246fc17580880c6bedcda55ad065db410 Update black]
- by ross-spencer https://github.com/ross-spencer
- 2024-03-25 19:53:59
- [tag:github.com,2008:Grit::Commit/40c5a871d14e866d55a61649930a12c3c629a9bb Remove denylist restrictions]
- by ross-spencer https://github.com/ross-spencer
- 2024-03-24 19:08:08
- [tag:github.com,2008:Grit::Commit/fc9dcc15c2c06bccf3b25122597d6860aafa9e44 Update workflows]
- by ross-spencer https://github.com/ross-spencer