DROID Siegfried Sqlite Analysis Engine

Format Identification Analysis and Reporting
Homepage:	https://github.com/exponential-decay/demystify
License:	Open source (see URL above)
Platforms:	sqlite + Python + text/html
Function:	Metadata Extraction,Content Profiling,De-Duplication

Description

Now known as "Demystify" with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of DROID CSV export files, Siegfried YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with rsync to isolate these sets from one-another to be more flexible to work with.

User Experiences

• Blog entries from the tool author, Ross Spencer:
  • [2014-06-03] Describing the creation and purpose of the tool.
  • [2015-08-25] Using the output of the tool to create a digital preservation rogues gallery.
  • [2016-05-23] The integration of Siegfried output for consistent and repeatable reporting.
  • [2016-05-24] Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.

Development Activity

All development activity is visible on GitHub: http://github.com/ross-spencer/droid-sqlite-analysis/commits

Release Feed

Below the last 3 release feeds:

2022-01-16 22:53:00

[tag:github.com,2008:Repository/15066530/v1.0.0 v1.0.0 release candidate for Python 2 and 3 compatibility]

by ross-spencer

2022-01-16 22:20:19

[tag:github.com,2008:Repository/15066530/v0.6.7-BETA v0.6.7-BETA]

by ross-spencer

2016-09-30 02:52:29

[tag:github.com,2008:Repository/15066530/0.6.4-BETA A little more on internationalization...]

by ross-spencer

Activity Feed

Below the last 5 commits:

2022-04-10 13:32:03

[tag:github.com,2008:Grit::Commit/c8f527585f74d3cdeee71c112e4773e531df67ad Up sqlitefid version to v2.0.2]

by ross-spencer https://github.com/ross-spencer

2022-01-16 22:35:52

[tag:github.com,2008:Grit::Commit/3452ac7c77e8b67420708a84f331968b6d91f9b4 Fixup workflows]

by ross-spencer https://github.com/ross-spencer

2022-01-16 22:25:30

[tag:github.com,2008:Grit::Commit/0f83da16733431ee2e2c3e568ba70e654dfd4a0c Bump lxml from 4.6.3 to 4.6.5 in /requirements]

by dependabot https://github.com/dependabot

2022-01-16 22:02:17

[tag:github.com,2008:Grit::Commit/ab3016690e5e4edf3b957c105a1224641c9bd122 Make Demystify Python 2 and 3 compatible]

by ross-spencer https://github.com/ross-spencer

2020-01-25 03:56:13

[tag:github.com,2008:Grit::Commit/114b1d08ea0d4dcb2bcf25a37f6e22d60f07a271 Create FUNDING.yml]

by ross-spencer https://github.com/ross-spencer