COPTR - User contributions [en-gb]

Talk:PRONOM

2026-05-27T09:04:25Z

Ross-spencer: Question about this entry

== Why do we not have a PRONOM entry? ==

This feels like it might be a good entry to have (for completeness if nothing else), but it was deleted early on in 2014. Do we want to revisit the decision? [[User:Ross-spencer|Ross-spencer]] ([[User talk:Ross-spencer|talk]]) 09:04, 27 May 2026 (UTC)

Talk:Wish list of tools to add to COPTR

2026-05-14T21:21:51Z

Ross-spencer: Add signature

== This page and workflow ==

Is there a way to bring this page more toward the front-page or side-menus of COPTR? I don't see it on the front page or in the menus and so I'm probably just one discovering that it exists.

I am tempted to argue that the existence of this page suggests some residue difficulty adding entries as it should be as easy to create a new entry as adding to a todo list.

I am not sure it should be deleted once triaged but maybe there is a discussion how to work with it within COPTR's broader workflow? [[User:Ross-spencer|Ross-spencer]] ([[User talk:Ross-spencer|talk]]) 21:21, 14 May 2026 (UTC)

Talk:Wish list of tools to add to COPTR

2026-05-14T21:21:27Z

Ross-spencer: Question about workflow

JSONID

2026-04-23T07:44:08Z

Ross-spencer: Add NTTW

{{Infobox tool
|purpose=Identification of JSON, YAML, and TOML document types
|homepage=https://github.com/ffdev-info/jsonid
|sourcecode=https://github.com/ffdev-info/jsonid
|license=Apache 2.0
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=JSON, JSONL, YAML, TOML
|function=File Format Identification
}}
{{Infobox tool details}}
== Description ==


Identification of JSON, YAML, and TOML document types.

== Functionality ==

JSONID parses serialization/deserialization formats ([https://en.wikipedia.org/wiki/Serialization "serde"]) such as JSON, YAML, and TOML to provide unambiguous identification.
JSONID also introduces a declarative syntax for writing document type signatures to enable identification of specific serde document types.
Key-value attributes can be shared across formats, and so signatures for JSON and YAML, for example, need only be written once.

=== Registry ===

As a temporary placeholder JSONID signatures are available [https://ffdev-info.github.io/jsonid/registry/ in a registry]. The long-term goal of this project is to enable other registries to delivery JSONID compatible signatures, e.g. PRONOM, Wikidata, and remove the need for a centralized resource like this.

=== Technical characteristics ===

JSONID explores potential technical characteristics that can
be attributed to serde formats. An example for a basic JSON object might
look as follows:

<pre>
{
"content_length": 82,
"number_of_lines": 9,
"line_warning": false,
"top_level_keys_count": 1,
"top_level_keys": [
"key1"
],
"top_level_types": [
"map"
],
"depth": 4,
"heterogeneous_list_types": false,
"fingerprint": {
"unf": "UNF:6:YEKQWBGm75JsN6H+8SzYRg==",
"cid": "bafkreidexnd3r76r5h3invwvu554573px5z4fg4uglw4pextmqc765kz64"
},
"doctype": "JSON",
"encoding": "UTF-8",
"agent": "jsonid/0.12.0 (ffdev-info)"
}
</pre>

The use of these technical characteristics will be explored in the
documentation and future writing.

=== Universal fingerprint ===

JSONID exports two universal fingerprints enabling the assertion of equivalence between different data objects. Universal Numerical Fingerprint (UNF) is also used in the [https://guides.dataverse.org/en/latest/developers/unf/index.html Dataverse project].
Content Identifiers (CIDs) come from the [https://docs.ipfs.tech/concepts/content-addressing/ IPFS project] and enable content-addressed storage within that ecosystem and others.

The significance of these fingerprinting techniques is their application
to identical data structures stored in different file formats.

The checksum of different file formats will always evaluate differently,
but analysed as data structures, we can begin to appraise data
beyond its presentation.

==== Fingerprinting example ====

<table>
<tr>
<td>Content</td>
<td><pre>
{
"hello": "world",
"goodbye": false,
"values": [
1,
2,
3.142
]
}</pre></td>
<td><pre>goodbye: false
hello: world
values:
- 1
- 2
- 3.142

</pre></td>
<td><pre>hello = "world"
values = [1, 2, 3.142]
goodbye = false

</pre></td>
</tr>
<tr>
<td>Type</td>
<td>JSON</td>
<td>YAML</td>
<td>TOML</td>
</tr>
<tr>
<td>Checksum (MD5)</td>
<td><code>bcd5a37f36ada2e4b72144d90a1427d5</code></td>
<td><code>b5e75fdc100032f2744eff1e1bdf5b88</code></td>
<td><code>04b33e24e0cc208c6bd70fabaef3a9c5</code></td>
</tr>
<tr>
<td>UNF</td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
</tr>
<tr>
<td>CID</td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
</tr>
</table>
 

=== Example identification output ===

JSONID has currently settled on a line-by-line, [[Fine Free File Command]] (File) and MIME-Like output. Fields are tab delineated and are human readable.

<pre>
/media/govdocs/govdocs_selected/LOG_1/074500.log [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/HTML_24/009970.html [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/TEXT_6/131522.txt [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
</pre>

It is anticipated that users will help guide the ongoing development of JSONID's output to better support both user and repository workflows.

=== PRONOM signature development ===

JSONID provides a high-level language for output of PRONOM compatible
signatures. The feature set is still in its BETA phase but JSONID provides
two distinct capabilities:

==== Registry output ====

JSONID's registry can be output using the `--pronom` flag. A signature file
will be created under `jsonid_pronom.xml` which can be imported into DROID
for identification of document types registered with JSONID.

JSONID's registry is output alongisde a handful of baseline JSON signatures
designed to capture "plain"-JSON that is not yet encoded in the registry.

==== Signature development ====

A standalone `json2pronom` utility is provided for creation of potentially
robust DROID compatible signatures.

As a high-level language, signatures can be defined in easy to understand
syntax and then output consistently via the `json2pronom` utility. Signatures
include sensible defaults for whitespace and other aspects that are
difficult for signature developers to consistently anticipate when writing
JSON based signatures.

See the [https://github.com/ffdev-info/jsonid?tab=readme-ov-file#pronom JSONID docs] for more information.

== Sample files ==

Sample files used for the development of JSONID signatures can be found in their [https://github.com/ffdev-info/jsonid-integration-files own repository].

== User experiences ==

* [https://docs.google.com/presentation/d/11zQTMwiXRa3UNyUSKTLwcCMos0gg9SpMxdrWOXF0lr4/edit?usp=sharing Introducing JSONID @ NTTW].
* [https://openpreservation.org/blogs/on-the-lookout-looking-for-common-json-files-and-introducing-jsonid/ On the lookout for common JSON files].
* [https://www.youtube.com/watch?v=ZMVb3Bc_irc JSONID at NTTW9].

= Development Activity =

All development activity is visible on GitHub: https://github.com/ffdev-info/jsonid/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ffdev-info/jsonid/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ffdev-info/jsonid/commits/main.atom</rss>

Linguist

2026-03-30T12:16:53Z

Ross-spencer: Initial entry

{{Infobox tool
|purpose=Identify the breakdown of programming languages used in a GitHub repository, or the anticipated language of an individual file
|homepage=https://github.com/github-linguist/linguist/tree/main
|tool_status=Maintained
|sourcecode=https://github.com/github-linguist/linguist/tree/main
|license=MIT
|cost=Free and Open Source (FOSS)
|language=Ruby
|function=Content Profiling, File Format Identification
}}
== Description ==


Identify the breakdown of programming languages used in a GitHub repository, or the anticipated language of an individual file.

The most well-known output of linguist's process is the language breakdown graph shown on a repository on GitHub documenting the percentage of languages used along with a small chart visualizing that breakdown.

Unlike utilities like DROID, based on PRONOM, which rely on pattern matching to identify, in this case programming language, and thus, file format, GitHub's linguist is deterministic and uses a series of decision making strategies to determine the potential file format.

=== Algorithm ===

Linguist uses a number of strategies to reduce the amount of inputs it is dealing with including excluding binary objects and those identified as data before applying a series of identification strategies to try and determine the programming language used across a GitHub repository or an individual file.

Its algorithm is described in more detail [https://github.com/github-linguist/linguist/blob/537297cdae3ab05f8d5dd1c03627a5bd73707b19/docs/how-linguist-works.md on GitHub].

Among its strategies, linguist lists the following checks:

* Vim or Emacs modeline,
* commonly used filename,
* shell shebang,
* file extension,
* XML header,
* man page section,
* heuristics,
* naïve Bayesian classification

== User Experiences ==


* [https://blog.jakelee.co.uk/adding-github-language-with-linguist/ How to add a programming language to GitHub] Jake Lee on Software.
* [https://github.com/github-linguist/linguist/discussions Linguist Discussions] on GitHub.

= Development Activity =

All development activity is visible on GitHub: https://github.com/github-linguist/linguist/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/github-linguist/linguist/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/github-linguist/linguist/commits/main.atom</rss>

== References ==

* [https://github.com/github-linguist/linguist/blob/537297cdae3ab05f8d5dd1c03627a5bd73707b19/docs/how-linguist-works.md How Linguist Works] on GitHub.

== See also ==

* [https://coptr.digipres.org/Cloc CLOC] (Count Lines of Code) on COPTR.

{{Infobox tool details}}

Arweave

2026-03-10T12:09:31Z

Ross-spencer: Add example transactions

{{Infobox tool
|purpose=A blockchain using a proof-of-access protocol and theoretically enabling permanent storage of data
|homepage=https://www.arweave.org/
|tool_status=Maintained
|sourcecode=https://github.com/arweaveteam
|cost=Transaction dependent
|Wikidata ID=Q129196013
|function=Active Data Storage, Backup, Service, Storage
}}
{{Infobox tool details}}
== Description ==


A storage based blockchain that uses proof-of-access (succinct proofs of random access (SPoRA)) as a form of consensus to ensure access to historical blocks of data, and to enable creation of new blocks.

Arweave attempts to enable permanence for the storage of data on its chain by using a "storage endowment" theoretically ensuing the payment of storage for 200 years.

== Proof of access ==

From the 19-25 ArweaveHub weekly digest, SPoRA is designed to incentivize storing large parts of the blockchain's data:

<blockquote>
Under SPoRA’s consensus model, miners must prove they have access to a randomly selected piece of data from the network’s history to mine a new block. This creates a massive incentive to store as much of the weave as possible.
</blockquote>

A network continuously proving access to user data should also be reassuring to those making use of the network/blockchain's resources.

== Storage endowment ==

An endowment calculation made at the genesis of the protocol ensures perpetual payment for storage for 200 years based on the theory of ever-decreasing storage costs.

From the Arweave protocol:

<blockquote>
If a transaction does upload data, the required upload fee scales with the uploaded volume.

A part of the fee goes directly to the miner including this transaction into a block. Its purpose is to incentivize miners to enlarge the Arweave dataset.

The rest of the fee goes to the endowment.

The Arweave network's endowment removes tokens from circulation every time data is uploaded, creating a reserve to pay for data storage over time. The storage purchasing power of the endowment is elastic, changing with the volume of data committed, the cost of data storage, and token value over time. One of the main drivers of change in the value of the endowment is that a decreasing cost of storage creates a corresponding proportional increase in storage purchasing power, leading to fewer tokens needing to be released from the endowment in the future. We call the rate of decline in overall costs for storing a unit of data for a fixed period of time the Kryder+ rate. This rate incorporates the change in price of hardware, electricity, and operational costs surrounding data storage.

Users pay for 200 years' worth of replicated storage at present prices, such that only a 0.5% Kryder+ rate is sufficient to sustain the endowment for an indefinite term, in the absence of token price changes. Under these conditions, the storage purchasing power of the endowment at the end of each year would be equal to that at the beginning. The actual declining cost of storage over the last 50 years has been, on average, ≈38.5% per year.

To determine how much a single byte costs in AR at any given time, the protocol uses the current network difficulty (as an oracle of the total amount of mining resources currently maintaining the network) and the recent history of operations.
</blockquote>

A simpler rendition from the 19-25 ArweaveHub weekly digest:

<blockquote>
...when a user uploads data to Arweave, a fraction of the fee goes to the miner immediately. The majority is sent to an endowment fund that grows steadily over time.

The fees for this transaction use a calculation that covers 20 replicas of user data for 200 years. This creates a self-executing, sustainable engine that ensures miners are incentivized to keep your data accessible for centuries, not just years.
</blockquote>

== User Experiences ==


* [https://podcast.onlyarweave.com/episodes/institutional-archiving-and-more-with-peter-of-arklyio-Bst5mWEr Institutional archiving and more @ PermaWeb Pioneers].

== Development Activity ==

=== Arweave API ===

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave/releases.atom</rss>

==== Activity Feed ====
Below the last 3 commits:
<rss max=3>https://github.com/ArweaveTeam/arweave/commits/master.atom</rss>

==== Arweave standards ====

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/releases.atom</rss>

==== Activity Feed ====
Below the last 3 commits:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/commits/master.atom</rss>

== Tools using Arweave ==

* [https://github.com/arkly-io/arweave-api Arkly Arweave API] enabling storage and validation of Bagit compatible archival packages.
* [https://ar.io/ ar.io] tooling provides distributed gateways for fault-tolerant access and and a <code>ar://</code> namespace enabling dns like addressing.

== Example Transaction ==

* [https://viewblock.io/arweave/tx/XX3Si_YD30UVUqkdSp5m0BXEG_avs9tmOLv3UFB4T6Q RedStone Data Bundle] is an example of bundled data from February 2025. A bundle conforms to [https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md ANS-104].
* [https://viewblock.io/arweave/tx/8Dke8p30z2_RVIcyK5AGQBWpRLGQlISis_TsE0IA_DE Orcfax Bagit Package] is an example of a Bagit RFC8493 file stored March 2026.

== References ==

* [https://docs.arweave.org/developers/development/protocol Arweave Protocol].
* [https://arweavehub.com/weekly/fundamentals-of-arweave 19-25 ArweaveHub Weekly Digest: Fundamentals of Arweave].

== Informative References ==

* [https://www.arweave.org/ Arweave Homepage].
* [https://www.crimrxiv.com/pub/2ud0ee07/release/1 Continuum Arweave Ar.io] digital preservation initiative.
* [https://docs.orcfax.io/verify verification of Orcfax packages on Arweave]

Arweave

2026-03-10T09:47:32Z

Ross-spencer: Add ar.io

{{Infobox tool
|purpose=A blockchain using a proof-of-access protocol and theoretically enabling permanent storage of data
|homepage=https://www.arweave.org/
|tool_status=Maintained
|sourcecode=https://github.com/arweaveteam
|cost=Transaction dependent
|Wikidata ID=Q129196013
|function=Active Data Storage, Backup, Service, Storage
}}
{{Infobox tool details}}
== Description ==


A storage based blockchain that uses proof-of-access (succinct proofs of random access (SPoRA)) as a form of consensus to ensure access to historical blocks of data, and to enable creation of new blocks.

Arweave attempts to enable permanence for the storage of data on its chain by using a "storage endowment" theoretically ensuing the payment of storage for 200 years.

== Proof of access ==

From the 19-25 ArweaveHub weekly digest, SPoRA is designed to incentivize storing large parts of the blockchain's data:

<blockquote>
Under SPoRA’s consensus model, miners must prove they have access to a randomly selected piece of data from the network’s history to mine a new block. This creates a massive incentive to store as much of the weave as possible.
</blockquote>

A network continuously proving access to user data should also be reassuring to those making use of the network/blockchain's resources.

== Storage endowment ==

An endowment calculation made at the genesis of the protocol ensures perpetual payment for storage for 200 years based on the theory of ever-decreasing storage costs.

From the Arweave protocol:

<blockquote>
If a transaction does upload data, the required upload fee scales with the uploaded volume.

A part of the fee goes directly to the miner including this transaction into a block. Its purpose is to incentivize miners to enlarge the Arweave dataset.

The rest of the fee goes to the endowment.

The Arweave network's endowment removes tokens from circulation every time data is uploaded, creating a reserve to pay for data storage over time. The storage purchasing power of the endowment is elastic, changing with the volume of data committed, the cost of data storage, and token value over time. One of the main drivers of change in the value of the endowment is that a decreasing cost of storage creates a corresponding proportional increase in storage purchasing power, leading to fewer tokens needing to be released from the endowment in the future. We call the rate of decline in overall costs for storing a unit of data for a fixed period of time the Kryder+ rate. This rate incorporates the change in price of hardware, electricity, and operational costs surrounding data storage.

Users pay for 200 years' worth of replicated storage at present prices, such that only a 0.5% Kryder+ rate is sufficient to sustain the endowment for an indefinite term, in the absence of token price changes. Under these conditions, the storage purchasing power of the endowment at the end of each year would be equal to that at the beginning. The actual declining cost of storage over the last 50 years has been, on average, ≈38.5% per year.

To determine how much a single byte costs in AR at any given time, the protocol uses the current network difficulty (as an oracle of the total amount of mining resources currently maintaining the network) and the recent history of operations.
</blockquote>

A simpler rendition from the 19-25 ArweaveHub weekly digest:

<blockquote>
...when a user uploads data to Arweave, a fraction of the fee goes to the miner immediately. The majority is sent to an endowment fund that grows steadily over time.

The fees for this transaction use a calculation that covers 20 replicas of user data for 200 years. This creates a self-executing, sustainable engine that ensures miners are incentivized to keep your data accessible for centuries, not just years.
</blockquote>

== User Experiences ==


* [https://podcast.onlyarweave.com/episodes/institutional-archiving-and-more-with-peter-of-arklyio-Bst5mWEr Institutional archiving and more @ PermaWeb Pioneers].

== Development Activity ==

=== Arweave API ===

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave/releases.atom</rss>

==== Activity Feed ====
Below the last 3 commits:
<rss max=3>https://github.com/ArweaveTeam/arweave/commits/master.atom</rss>

==== Arweave standards ====

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/releases.atom</rss>

==== Activity Feed ====
Below the last 3 commits:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/commits/master.atom</rss>

== Tools using Arweave ==

* [https://github.com/arkly-io/arweave-api Arkly Arweave API] enabling storage and validation of Bagit compatible archival packages.
* [https://ar.io/ ar.io] tooling provides distributed gateways for fault-tolerant access and and a <code>ar://</code> namespace enabling dns like addressing.

== References ==

* [https://docs.arweave.org/developers/development/protocol Arweave Protocol].
* [https://arweavehub.com/weekly/fundamentals-of-arweave 19-25 ArweaveHub Weekly Digest: Fundamentals of Arweave].

== Informative References ==

* [https://www.arweave.org/ Arweave Homepage].
* [https://www.crimrxiv.com/pub/2ud0ee07/release/1 Continuum Arweave Ar.io] digital preservation initiative.
* [https://docs.orcfax.io/verify verification of Orcfax packages on Arweave]

Fine Free File Command

2026-02-25T17:36:00Z

Ross-spencer: Add link to syntax manpage

{{Infobox tool
|purpose=This is the home page for the open source implementation of the file(1) command that ships with every free operating system (OpenBSD, Linux, NetBSD, FreeBSD, etc.
|homepage=http://www.darwinsys.com/file/
|function=File Format Identification
}}
{{Infobox tool details
|ohloh_id=Fine Free File Command
}}
= Description =
This is the home page for the open source implementation of the file(1) command that ships with every free operating system (OpenBSD, Linux, NetBSD, FreeBSD, etc.) and has been ported to most systems where people use free software (including OS/2, DOS, MS Windows, etc.).

= User Experiences =

*'''[2019-11-16]''' [https://shapeshed.com/unix-file/ Linux and Unix file command tutorial with examples]

= Development Activity =

All development activity is visible on GitHub: https://github.com/file/file/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/file/file/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/file/file/commits/master.atom</rss>

== Informative references ==

* [https://linux.die.net/man/5/magic syntax of the File magic numbers]

Talk:Fine Free File Command

2026-02-25T17:34:57Z

Ross-spencer: Note of thanks!

== Is this name the most idiomatic? ==

I am wondering what folks think about the name convention adopted for this record. I have contributed to this one in the past, but even today I don't think to look under this name when I am thinking of the Unix File(1) Utility. Perhaps file (command), or another name is more appropriate? That being said, does anyone know of an aliasing mechanism on MediaWiki? An alias, or set of aliases back to this record might also make it easier to find/reference. [[User:Ross-spencer|Ross-spencer]] ([[User talk:Ross-spencer|talk]]) 10:25, 26 January 2026 (UTC)
: I think the MediaWiki convention is to make [https://www.mediawiki.org/wiki/Help:Redirects Redirects] that point aliases to the 'canonical' page. Adding a "file (command)" alias seems reasonable. The tool's homepage declares it to be the "fine free file command", but also refers to it as the "file(1) command", so maybe that's a better name and/or alias? [[User:Andrew N. Jackson|Andrew N. Jackson]] ([[User talk:Andrew N. Jackson|talk]]) 15:12, 5 February 2026 (UTC)
: Experimenting with adding a redirect from [[file(1) command]]... [[User:Andrew N. Jackson|Andrew N. Jackson]] ([[User talk:Andrew N. Jackson|talk]]) 20:09, 5 February 2026 (UTC)
:: Alas, I didn't get a ping about this discussion, I'll need to check my notification settings -- either way, I am here because I saw it worked through the search this time!! No having to remember the Free File Command!! Very good, thanks for doing that Andy. [[User:Ross-spencer|Ross-spencer]] ([[User talk:Ross-spencer|talk]]) 17:34, 25 February 2026 (UTC)
----

Ddrescue

2026-02-12T09:55:21Z

Ross-spencer: Add full-stop

{{Infobox tool
|purpose=data recovery tool for hard disks, CD-ROMs, floppy disks, etc.
|homepage=https://www.gnu.org/software/ddrescue/
|tool_status=Maintained
|license=GNU GPL
|cost=Free and Open Source (FOSS)
|Wikidata ID=Q15427600
|function=Data capture and Deposit, Disk Imaging, File Recovery, Forensic
|content=Disk Image
}}
{{Infobox tool details}}
== Description ==


ddrescue is a data recovery tool that copies data from one file or block device to another.

ddrescue attempts to recover data by enabling multiple retries and splitting recovery into five phases which it has termed:

* copying,
* trimming,
* sweeping,
* scraping,
* retrying.

More information about these phases can be found in the [https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html manual].

== Additional utilities ==

Software that supports ddrescue outputs or generates ddrescue inputs include:

Ddrescueview - A graphical viewer for ddrescue mapfiles.
Ddrutility - A set of tools designed to work with ddrescue to aid with data recovery.
Partclone - Partition Software able to create domain mapfiles for ddrescue.

== User Experiences ==


* [https://openpreservation.org/blogs/resurrecting-the-first-dutch-web-index-nl-menu-revisited/?order=relevance%3ADESC&search=ddrescue&q=18193 Resurrecting the first Dutch Web Index] by Johan van der Knijff

== References ==

* [https://www.gnu.org/software/ddrescue/ ddrescue home page]

== Informative references ==

* [https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html ddrescue manual].
* [https://linux.die.net/man/1/ddrescue ddrescue manpage.

Ddrescue

2026-02-12T09:54:59Z

Ross-spencer: Add manpage

Ddrescue

2026-02-12T09:45:01Z

Ross-spencer: Add new entry

Arweave

2026-02-09T22:00:36Z

Ross-spencer: Update feeds

{{Infobox tool
|purpose=A blockchain using a proof-of-access protocol and theoretically enabling permanent storage of data
|homepage=https://www.arweave.org/
|tool_status=Maintained
|sourcecode=https://github.com/arweaveteam
|cost=Transaction dependent
|Wikidata ID=Q129196013
|function=Active Data Storage, Backup, Service, Storage
}}
{{Infobox tool details}}
== Description ==


A storage based blockchain that uses proof-of-access (succinct proofs of random access (SPoRA)) as a form of consensus to ensure access to historical blocks of data, and to enable creation of new blocks.

Arweave attempts to enable permanence for the storage of data on its chain by using a "storage endowment" theoretically ensuing the payment of storage for 200 years.

== Proof of access ==

From the 19-25 ArweaveHub weekly digest, SPoRA is designed to incentivize storing large parts of the blockchain's data:

<blockquote>
Under SPoRA’s consensus model, miners must prove they have access to a randomly selected piece of data from the network’s history to mine a new block. This creates a massive incentive to store as much of the weave as possible.
</blockquote>

A network continuously proving access to user data should also be reassuring to those making use of the network/blockchain's resources.

== Storage endowment ==

An endowment calculation made at the genesis of the protocol ensures perpetual payment for storage for 200 years based on the theory of ever-decreasing storage costs.

From the Arweave protocol:

<blockquote>
If a transaction does upload data, the required upload fee scales with the uploaded volume.

A part of the fee goes directly to the miner including this transaction into a block. Its purpose is to incentivize miners to enlarge the Arweave dataset.

The rest of the fee goes to the endowment.

The Arweave network's endowment removes tokens from circulation every time data is uploaded, creating a reserve to pay for data storage over time. The storage purchasing power of the endowment is elastic, changing with the volume of data committed, the cost of data storage, and token value over time. One of the main drivers of change in the value of the endowment is that a decreasing cost of storage creates a corresponding proportional increase in storage purchasing power, leading to fewer tokens needing to be released from the endowment in the future. We call the rate of decline in overall costs for storing a unit of data for a fixed period of time the Kryder+ rate. This rate incorporates the change in price of hardware, electricity, and operational costs surrounding data storage.

Users pay for 200 years' worth of replicated storage at present prices, such that only a 0.5% Kryder+ rate is sufficient to sustain the endowment for an indefinite term, in the absence of token price changes. Under these conditions, the storage purchasing power of the endowment at the end of each year would be equal to that at the beginning. The actual declining cost of storage over the last 50 years has been, on average, ≈38.5% per year.

To determine how much a single byte costs in AR at any given time, the protocol uses the current network difficulty (as an oracle of the total amount of mining resources currently maintaining the network) and the recent history of operations.
</blockquote>

A simpler rendition from the 19-25 ArweaveHub weekly digest:

<blockquote>
...when a user uploads data to Arweave, a fraction of the fee goes to the miner immediately. The majority is sent to an endowment fund that grows steadily over time.

The fees for this transaction use a calculation that covers 20 replicas of user data for 200 years. This creates a self-executing, sustainable engine that ensures miners are incentivized to keep your data accessible for centuries, not just years.
</blockquote>

== User Experiences ==


* [https://podcast.onlyarweave.com/episodes/institutional-archiving-and-more-with-peter-of-arklyio-Bst5mWEr Institutional archiving and more @ PermaWeb Pioneers].

== Development Activity ==

=== Arweave API ===

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave/releases.atom</rss>

==== Activity Feed ====
Below the last 3 commits:
<rss max=3>https://github.com/ArweaveTeam/arweave/commits/master.atom</rss>

==== Arweave standards ====

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/releases.atom</rss>

==== Activity Feed ====
Below the last 3 commits:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/commits/master.atom</rss>

== Tools using Arweave ==

* [https://github.com/arkly-io/arweave-api Arkly Arweave API] enabling storage and validation of Bagit compatible archival packages.

== References ==

* [https://docs.arweave.org/developers/development/protocol Arweave Protocol].
* [https://arweavehub.com/weekly/fundamentals-of-arweave 19-25 ArweaveHub Weekly Digest: Fundamentals of Arweave].

== Informative References ==

* [https://www.arweave.org/ Arweave Homepage].
* [https://www.crimrxiv.com/pub/2ud0ee07/release/1 Continuum Arweave Ar.io] digital preservation initiative.
* [https://docs.orcfax.io/verify verification of Orcfax packages on Arweave]

Arweave

2026-02-09T21:57:26Z

Ross-spencer: Add user experience

{{Infobox tool
|purpose=A blockchain using a proof-of-access protocol and theoretically enabling permanent storage of data
|homepage=https://www.arweave.org/
|tool_status=Maintained
|sourcecode=https://github.com/arweaveteam
|cost=Transaction dependent
|Wikidata ID=Q129196013
|function=Active Data Storage, Backup, Service, Storage
}}
{{Infobox tool details}}
== Description ==


A storage based blockchain that uses proof-of-access (succinct proofs of random access (SPoRA)) as a form of consensus to ensure access to historical blocks of data, and to enable creation of new blocks.

Arweave attempts to enable permanence for the storage of data on its chain by using a "storage endowment" theoretically ensuing the payment of storage for 200 years.

== Proof of access ==

From the 19-25 ArweaveHub weekly digest, SPoRA is designed to incentivize storing large parts of the blockchain's data:

<blockquote>
Under SPoRA’s consensus model, miners must prove they have access to a randomly selected piece of data from the network’s history to mine a new block. This creates a massive incentive to store as much of the weave as possible.
</blockquote>

A network continuously proving access to user data should also be reassuring to those making use of the network/blockchain's resources.

== Storage endowment ==

An endowment calculation made at the genesis of the protocol ensures perpetual payment for storage for 200 years based on the theory of ever-decreasing storage costs.

From the Arweave protocol:

<blockquote>
If a transaction does upload data, the required upload fee scales with the uploaded volume.

A part of the fee goes directly to the miner including this transaction into a block. Its purpose is to incentivize miners to enlarge the Arweave dataset.

The rest of the fee goes to the endowment.

The Arweave network's endowment removes tokens from circulation every time data is uploaded, creating a reserve to pay for data storage over time. The storage purchasing power of the endowment is elastic, changing with the volume of data committed, the cost of data storage, and token value over time. One of the main drivers of change in the value of the endowment is that a decreasing cost of storage creates a corresponding proportional increase in storage purchasing power, leading to fewer tokens needing to be released from the endowment in the future. We call the rate of decline in overall costs for storing a unit of data for a fixed period of time the Kryder+ rate. This rate incorporates the change in price of hardware, electricity, and operational costs surrounding data storage.

Users pay for 200 years' worth of replicated storage at present prices, such that only a 0.5% Kryder+ rate is sufficient to sustain the endowment for an indefinite term, in the absence of token price changes. Under these conditions, the storage purchasing power of the endowment at the end of each year would be equal to that at the beginning. The actual declining cost of storage over the last 50 years has been, on average, ≈38.5% per year.

To determine how much a single byte costs in AR at any given time, the protocol uses the current network difficulty (as an oracle of the total amount of mining resources currently maintaining the network) and the recent history of operations.
</blockquote>

A simpler rendition from the 19-25 ArweaveHub weekly digest:

<blockquote>
...when a user uploads data to Arweave, a fraction of the fee goes to the miner immediately. The majority is sent to an endowment fund that grows steadily over time.

The fees for this transaction use a calculation that covers 20 replicas of user data for 200 years. This creates a self-executing, sustainable engine that ensures miners are incentivized to keep your data accessible for centuries, not just years.
</blockquote>

== User Experiences ==


* [https://podcast.onlyarweave.com/episodes/institutional-archiving-and-more-with-peter-of-arklyio-Bst5mWEr Institutional archiving and more @ PermaWeb Pioneers].

== Development Activity ==

=== Arweave API ===

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave/releases.atom</rss>

==== Activity Feed ====
Below the last 5 commits:
<rss max=5>https://github.com/ArweaveTeam/arweave/commits/main.atom</rss>

==== Arweave standards ====

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/releases.atom</rss>

==== Activity Feed ====
Below the last 5 commits:
<rss max=5>https://github.com/ArweaveTeam/arweave-standards/commits/main.atom</rss>

== Tools using Arweave ==

* [https://github.com/arkly-io/arweave-api Arkly Arweave API] enabling storage and validation of Bagit compatible archival packages.

== References ==

* [https://docs.arweave.org/developers/development/protocol Arweave Protocol].
* [https://arweavehub.com/weekly/fundamentals-of-arweave 19-25 ArweaveHub Weekly Digest: Fundamentals of Arweave].

== Informative References ==

* [https://www.arweave.org/ Arweave Homepage].
* [https://www.crimrxiv.com/pub/2ud0ee07/release/1 Continuum Arweave Ar.io] digital preservation initiative.
* [https://docs.orcfax.io/verify verification of Orcfax packages on Arweave]

Arweave

2026-02-09T21:51:12Z

Ross-spencer: Add information about proof of access and the endowment

{{Infobox tool
|purpose=A blockchain using a proof-of-access protocol and theoretically enabling permanent storage of data
|homepage=https://www.arweave.org/
|tool_status=Maintained
|sourcecode=https://github.com/arweaveteam
|cost=Transaction dependent
|Wikidata ID=Q129196013
|function=Active Data Storage, Backup, Service, Storage
}}
{{Infobox tool details}}
== Description ==


A storage based blockchain that uses proof-of-access (succinct proofs of random access (SPoRA)) as a form of consensus to ensure access to historical blocks of data, and to enable creation of new blocks.

Arweave attempts to enable permanence for the storage of data on its chain by using a "storage endowment" theoretically ensuing the payment of storage for 200 years.

== Proof of access ==

From the 19-25 ArweaveHub weekly digest, SPoRA is designed to incentivize storing large parts of the blockchain's data:

<blockquote>
Under SPoRA’s consensus model, miners must prove they have access to a randomly selected piece of data from the network’s history to mine a new block. This creates a massive incentive to store as much of the weave as possible.
</blockquote>

A network continuously proving access to user data should also be reassuring to those making use of the network/blockchain's resources.

== Storage endowment ==

An endowment calculation made at the genesis of the protocol ensures perpetual payment for storage for 200 years based on the theory of ever-decreasing storage costs.

From the Arweave protocol:

<blockquote>
If a transaction does upload data, the required upload fee scales with the uploaded volume.

A part of the fee goes directly to the miner including this transaction into a block. Its purpose is to incentivize miners to enlarge the Arweave dataset.

The rest of the fee goes to the endowment.

The Arweave network's endowment removes tokens from circulation every time data is uploaded, creating a reserve to pay for data storage over time. The storage purchasing power of the endowment is elastic, changing with the volume of data committed, the cost of data storage, and token value over time. One of the main drivers of change in the value of the endowment is that a decreasing cost of storage creates a corresponding proportional increase in storage purchasing power, leading to fewer tokens needing to be released from the endowment in the future. We call the rate of decline in overall costs for storing a unit of data for a fixed period of time the Kryder+ rate. This rate incorporates the change in price of hardware, electricity, and operational costs surrounding data storage.

Users pay for 200 years' worth of replicated storage at present prices, such that only a 0.5% Kryder+ rate is sufficient to sustain the endowment for an indefinite term, in the absence of token price changes. Under these conditions, the storage purchasing power of the endowment at the end of each year would be equal to that at the beginning. The actual declining cost of storage over the last 50 years has been, on average, ≈38.5% per year.

To determine how much a single byte costs in AR at any given time, the protocol uses the current network difficulty (as an oracle of the total amount of mining resources currently maintaining the network) and the recent history of operations.
</blockquote>

A simpler rendition from the 19-25 ArweaveHub weekly digest:

<blockquote>
...when a user uploads data to Arweave, a fraction of the fee goes to the miner immediately. The majority is sent to an endowment fund that grows steadily over time.

The fees for this transaction use a calculation that covers 20 replicas of user data for 200 years. This creates a self-executing, sustainable engine that ensures miners are incentivized to keep your data accessible for centuries, not just years.
</blockquote>

== User Experiences ==


== Development Activity ==

=== Arweave API ===

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave/releases.atom</rss>

==== Activity Feed ====
Below the last 5 commits:
<rss max=5>https://github.com/ArweaveTeam/arweave/commits/main.atom</rss>

==== Arweave standards ====

==== Release Feed ====
Below the last 3 release feeds:
<rss max=3>https://github.com/ArweaveTeam/arweave-standards/releases.atom</rss>

==== Activity Feed ====
Below the last 5 commits:
<rss max=5>https://github.com/ArweaveTeam/arweave-standards/commits/main.atom</rss>

== Tools using Arweave ==

* [https://github.com/arkly-io/arweave-api Arkly Arweave API] enabling storage and validation of Bagit compatible archival packages.

== References ==

* [https://docs.arweave.org/developers/development/protocol Arweave Protocol].
* [https://arweavehub.com/weekly/fundamentals-of-arweave 19-25 ArweaveHub Weekly Digest: Fundamentals of Arweave].

== Informative References ==

* [https://www.arweave.org/ Arweave Homepage].
* [https://www.crimrxiv.com/pub/2ud0ee07/release/1 Continuum Arweave Ar.io] digital preservation initiative.
* [https://docs.orcfax.io/verify verification of Orcfax packages on Arweave]

Arweave

2026-02-09T21:35:13Z

Ross-spencer: Adds a record for Arweave

Talk:Main Page

2026-01-27T06:38:28Z

Ross-spencer: Question about license and license text

==Use of Level 1 section headings ==
I just had a first stab at some editing an existing entry, and creating a new one. One small thing I noticed: all of the pages I've seen use Level 1 headings for the main sections, but this is discouraged by the MediaWiki documentation (esp. since Level 1 is already used for the main heading of each entry). See e.g. here:

http://www.mediawiki.org/wiki/Help_talk:Formatting#Level_1

And also:

http://www.mediawiki.org/wiki/Help:Formatting

So maybe it would be better to change those to level 2 (also in all cases I've seen the child sections use level 3 headings)? Not a big deal of course but it's a bit ugly and it's probably easier to fix this now than postponing it to a later stage (also existing entries will most likely be used as a template for creating new ones, so the problem will get progressively worse if left as it is). I suppose this is also something that might mess up attempts at automated text extraction. [[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 14:58, 20 November 2013 (UTC)

: Good point - I didn't know about this recommendation. It should be possible to automated this transformation using the [https://www.mediawiki.org/wiki/Manual:Pywikibot Pywikibot] framework and the [https://pypi.python.org/pypi/mwparserfromhell mwparserfromhell]. We could change the [[Template:Tool/Preload|tool template]] now and use a bot to modify the rest? [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 15:15, 20 November 2013 (UTC)

::Great! It would also help to mention the template on the main page (I don't have edit rights there), as I didn't even know there was any! Anyway, I just updated the template myself.[[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 16:07, 20 November 2013 (UTC)

::: I updated the main page to include an explicit link to the template that should be used when creating new Tool entries. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]])

==Purpose field in infobox==
Another thing I noticed is that for most entries the ''purpose'' field in the infobox more or less repeats what's already in the ''Description'' section. So maybe it's better to leave it out altogether (on the other hand it might be useful for automated discovery/analysis)? [[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 14:58, 20 November 2013 (UTC)
: The intention was the other way around - that the 'purpose' field was a brief description that might be published and re-used in other forms (due to being in the infobox), and the Description section should be more detailed. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 15:15, 20 November 2013 (UTC)
:: Yes, that's what I thought as well. For some existing entries both are identical though, so it wasn't immediately clear. [[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 15:56, 20 November 2013 (UTC)
::: I would say the goal is to improve the body Description section for those items. Is the Description in these cases exactly identical to the purpose? If so, I might be able to check for that automatically. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 21:18, 20 November 2013 (UTC)

==Perhaps rename Imaging category to Disk Imaging?==
Another thing: people might end up mixing up the ''Image'' and ''Imaging'' categories. Maybe rename the latter to ''Disk Imaging'' to avoid this?
[[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 16:15, 20 November 2013 (UTC)
: Good idea. Unfortunately, I think that due to the way MediaWiki works, this means editing all the items tagged with that category (unless there's a trick I'm unaware of). However, perhaps I should tweak [[User:COPTR Bot|COPTR Bot]] to do this too. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 21:22, 20 November 2013 (UTC)
:: Fixed manually --[[User:Prwheatley|Prwheatley]] ([[User talk:Prwheatley|talk]]) 17:51, 26 November 2013 (UTC)

==Problems with login procedure (VeriSign)==
Having created an OpenID with Verisign, signing in to COPTR is a bit hit and miss. It always takes multiple attempts before I can successfully log in, and I first have to work myself through multiple authorization errors. Not clear to me whether the error is caused by COPTR or VeriSign, and results seem to be a bit random. Also after some time of inactivity I seem to get logged out automatically. Confusingly, the COPTR login status at the top right then shows I'm still logged in (depending on which page I'm on, it seems), and I have to go through the whole login procedure (including failures) again.
[[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 11:16, 21 November 2013 (UTC)

: [http://meta.stackoverflow.com/a/1789 This StackOverflow answer] outlines some of the possible causes of problems with OpenID logins. I guess we could perhaps enable non-OpenID logins, although I wanted to avoid that for fear of making it even easier for the robots/spammers. Using Google as an OpenID provider is working fine for me. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 11:40, 21 November 2013 (UTC)

:: Hmm ... the StackOverflow page links to an OpenID checker, but I'm a bit scared of it because it exposes all the info you enter there to a publicly viewable log ... Not keen on using Google either, because they're, well ... Google. The main point of course is that the success of COPTR will depend completely on community involvement, and if the login procedure is already such a stumbling block that's not going to help there. Also the procedure to get the OpenID isn't terribly straightforward, and this will scare off potential contributors as well. So I would consider adding a non-OpenID login procedure, perhaps augmented with a CAPTCHA to keep the robots at bay (e.g. like the Archiveteam Formats Wiki). [[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 12:28, 21 November 2013 (UTC)

::: Okay, I've enabled plain logins and registration. To make sure you retain the same user account, you should ensure it knows the right email address so you can use password recovery to login. Alternatively, I think you should be able to reset your PW from your Preferences page. The current setup is that you should see CAPTCHAs blocking all edits etc. unless you have confirmed your email address. We'll see how it goes, and if we have to tighten up things to avoid spam, so be it. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 12:45, 21 November 2013 (UTC)

::::Great, thanks for the quick response! Apparently there's no way to set a pw from the Preferences page once you have an existing OpenID assigned to your user profile, and when I try to delete my OpenID I get an error that this isn't possible because no password is set, classic Catch 22 there! Anyway, I'll stick with the OpenID madness for a bit and if I get really fed up with it I may well just create a new COPTR account, probably the easiest solution all around. [[User:Johanvanderknijff|johanvanderknijff]] ([[User talk:Johanvanderknijff|talk]]) 13:14, 21 November 2013 (UTC)

::::: I'm pretty sure that, if you completely log out first, and if there is an email associated with your account, you can use the 'Password reset' option to get access. [[User:Andy Jackson|Andy Jackson]] ([[User talk:Andy Jackson|talk]]) 15:28, 21 November 2013 (UTC)

== changing Open Planets Foundation to Open Preservation Foundation ==

Hi. The Main Page need some update (changing Open Planets Foundation to Open Preservation Foundation). I have of course not the permission to do that :-) . Thanks. [[User:Chlara|Chlara ]] ([[User talk:Chlara|talk]]) 10:18, 27 November 2014 (UTC)
* Good call, this is now fixed. Thanks!

== DSPS (Digital Preservation Software Platform) is the same as Digital Preservation Software Platform ==

Hi. I have updated the Digital Preservation Software Platform. After that I saw that there is a nearly same Page DSPS (Digital Preservation Software Platform). Due to the fact, that "DSPS (Digital Preservation Software Platform)" is the better title I have copied the content from Digital Preservation Software Platform into DSPS (Digital Preservation Software Platform). In the Page Digital Preservation Software Platform I replaced the content with a link to DSPS (Digital Preservation Software Platform).

Can you please delete the page Digital Preservation Software Platform.

Thanks. [[User:Chlara|Chlara ]] ([[User talk:Chlara|talk]]) 11:30, 8 January 2015 (UTC)

== Merging duplicate entries ==

I noticed that there are two entries for the CONTENTdm digital asset management system:

* http://coptr.digipres.org/ContentDM
* http://coptr.digipres.org/CONTENTdm

I removed the limited content from the ContentDM page and made it redirect to the CONTENTdm page, using the MediaWiki #redirect option (see [http://www.mediawiki.org/wiki/Help:Redirects http://www.mediawiki.org/wiki/Help:Redirects]. Should this be the recommended practice or would you rather simply delete one of the pages? If the latter, how to we request removal of a page?

[[User:Danielle Plumer|Danielle Plumer]] ([[User talk:Danielle Plumer|talk]]) 03:00, 5 February 2015 (UTC)

== Check this entries ==
==== Workflow ====
Hi. I Think the entrie http://coptr.digipres.org/Workflow:Workflow_for_ingesting_digitized_books_into_a_digital_archive need to be changed or deleted. What do you think about it? Thanks. [[User:Chlara|Chlara ]] ([[User talk:Chlara|talk]]) 06:19, 20 April 2017 (UTC)

== Change to https ==
Hi. COPTR should change to https. Thanks. --[[User:Chlara|Chlara ]] ([[User talk:Chlara|talk]]) 09:48, 14 September 2017 (UTC)

== GitHub ATOM RSS feeds don't work ==

Pages with RSS feed embeds show error messages like:

Below the last 5 commits: Failed to load RSS feed from https://github.com/intranda/goobi/commits/master.atom: Error fetching URL: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

Is this a server configuration problem on COPTR's end? Can it be corrected or should the GitHub feeds be removed from pages?
--[[User:Jukervin|Jukervin]] ([[User talk:Jukervin|talk]]) 07:12, 3 July 2020 (UTC)

== License text and license out of date ==

The current licence text reads: <code>Content is available under Attribution-ShareAlike 3.0 Unported unless otherwise noted.</code> but I am wondering if this needs to be prefixed with Creative Commons so that it is clearer? (asking because there may be a reason it doesn't). Additionally, the links to the license lead to [Content is available under Attribution-ShareAlike 3.0 Unported unless otherwise noted. version three] of the CC-BY-SA licence and this needs to be 4.0. It'd be good to just clarify both these points before making the change. [[User:Ross-spencer|Ross-spencer]] ([[User talk:Ross-spencer|talk]]) 06:38, 27 January 2026 (UTC)

Talk:Fine Free File Command

2026-01-26T10:25:22Z

Ross-spencer: Question about naming

JSONID

2026-01-26T10:20:38Z

Ross-spencer: Add information about sample files

{{Infobox tool
|purpose=Identification of JSON, YAML, and TOML document types
|homepage=https://github.com/ffdev-info/jsonid
|sourcecode=https://github.com/ffdev-info/jsonid
|license=Apache 2.0
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=JSON, JSONL, YAML, TOML
|function=File Format Identification
}}
{{Infobox tool details}}
== Description ==


Identification of JSON, YAML, and TOML document types.

== Functionality ==

JSONID parses serialization/deserialization formats ([https://en.wikipedia.org/wiki/Serialization "serde"]) such as JSON, YAML, and TOML to provide unambiguous identification.
JSONID also introduces a declarative syntax for writing document type signatures to enable identification of specific serde document types.
Key-value attributes can be shared across formats, and so signatures for JSON and YAML, for example, need only be written once.

=== Registry ===

As a temporary placeholder JSONID signatures are available [https://ffdev-info.github.io/jsonid/registry/ in a registry]. The long-term goal of this project is to enable other registries to delivery JSONID compatible signatures, e.g. PRONOM, Wikidata, and remove the need for a centralized resource like this.

=== Technical characteristics ===

JSONID explores potential technical characteristics that can
be attributed to serde formats. An example for a basic JSON object might
look as follows:

<pre>
{
"content_length": 82,
"number_of_lines": 9,
"line_warning": false,
"top_level_keys_count": 1,
"top_level_keys": [
"key1"
],
"top_level_types": [
"map"
],
"depth": 4,
"heterogeneous_list_types": false,
"fingerprint": {
"unf": "UNF:6:YEKQWBGm75JsN6H+8SzYRg==",
"cid": "bafkreidexnd3r76r5h3invwvu554573px5z4fg4uglw4pextmqc765kz64"
},
"doctype": "JSON",
"encoding": "UTF-8",
"agent": "jsonid/0.12.0 (ffdev-info)"
}
</pre>

The use of these technical characteristics will be explored in the
documentation and future writing.

=== Universal fingerprint ===

JSONID exports two universal fingerprints enabling the assertion of equivalence between different data objects. Universal Numerical Fingerprint (UNF) is also used in the [https://guides.dataverse.org/en/latest/developers/unf/index.html Dataverse project].
Content Identifiers (CIDs) come from the [https://docs.ipfs.tech/concepts/content-addressing/ IPFS project] and enable content-addressed storage within that ecosystem and others.

The significance of these fingerprinting techniques is their application
to identical data structures stored in different file formats.

The checksum of different file formats will always evaluate differently,
but analysed as data structures, we can begin to appraise data
beyond its presentation.

==== Fingerprinting example ====

<table>
<tr>
<td>Content</td>
<td><pre>
{
"hello": "world",
"goodbye": false,
"values": [
1,
2,
3.142
]
}</pre></td>
<td><pre>goodbye: false
hello: world
values:
- 1
- 2
- 3.142

</pre></td>
<td><pre>hello = "world"
values = [1, 2, 3.142]
goodbye = false

</pre></td>
</tr>
<tr>
<td>Type</td>
<td>JSON</td>
<td>YAML</td>
<td>TOML</td>
</tr>
<tr>
<td>Checksum (MD5)</td>
<td><code>bcd5a37f36ada2e4b72144d90a1427d5</code></td>
<td><code>b5e75fdc100032f2744eff1e1bdf5b88</code></td>
<td><code>04b33e24e0cc208c6bd70fabaef3a9c5</code></td>
</tr>
<tr>
<td>UNF</td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
</tr>
<tr>
<td>CID</td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
</tr>
</table>
 

=== Example identification output ===

JSONID has currently settled on a line-by-line, [[Fine Free File Command]] (File) and MIME-Like output. Fields are tab delineated and are human readable.

<pre>
/media/govdocs/govdocs_selected/LOG_1/074500.log [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/HTML_24/009970.html [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/TEXT_6/131522.txt [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
</pre>

It is anticipated that users will help guide the ongoing development of JSONID's output to better support both user and repository workflows.

=== PRONOM signature development ===

JSONID provides a high-level language for output of PRONOM compatible
signatures. The feature set is still in its BETA phase but JSONID provides
two distinct capabilities:

==== Registry output ====

JSONID's registry can be output using the `--pronom` flag. A signature file
will be created under `jsonid_pronom.xml` which can be imported into DROID
for identification of document types registered with JSONID.

JSONID's registry is output alongisde a handful of baseline JSON signatures
designed to capture "plain"-JSON that is not yet encoded in the registry.

==== Signature development ====

A standalone `json2pronom` utility is provided for creation of potentially
robust DROID compatible signatures.

As a high-level language, signatures can be defined in easy to understand
syntax and then output consistently via the `json2pronom` utility. Signatures
include sensible defaults for whitespace and other aspects that are
difficult for signature developers to consistently anticipate when writing
JSON based signatures.

See the [https://github.com/ffdev-info/jsonid?tab=readme-ov-file#pronom JSONID docs] for more information.

== Sample files ==

Sample files used for the development of JSONID signatures can be found in their [https://github.com/ffdev-info/jsonid-integration-files own repository].

== User experiences ==

* [https://docs.google.com/presentation/d/11zQTMwiXRa3UNyUSKTLwcCMos0gg9SpMxdrWOXF0lr4/edit?usp=sharing Introducing JSONID @ NTTW].
* [https://openpreservation.org/blogs/on-the-lookout-looking-for-common-json-files-and-introducing-jsonid/ On the lookout for common JSON files].

= Development Activity =

All development activity is visible on GitHub: https://github.com/ffdev-info/jsonid/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ffdev-info/jsonid/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ffdev-info/jsonid/commits/main.atom</rss>

JSONID

2026-01-26T10:18:51Z

Ross-spencer: Add example output

{{Infobox tool
|purpose=Identification of JSON, YAML, and TOML document types
|homepage=https://github.com/ffdev-info/jsonid
|sourcecode=https://github.com/ffdev-info/jsonid
|license=Apache 2.0
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=JSON, JSONL, YAML, TOML
|function=File Format Identification
}}
{{Infobox tool details}}
== Description ==


Identification of JSON, YAML, and TOML document types.

== Functionality ==

JSONID parses serialization/deserialization formats ([https://en.wikipedia.org/wiki/Serialization "serde"]) such as JSON, YAML, and TOML to provide unambiguous identification.
JSONID also introduces a declarative syntax for writing document type signatures to enable identification of specific serde document types.
Key-value attributes can be shared across formats, and so signatures for JSON and YAML, for example, need only be written once.

=== Registry ===

As a temporary placeholder JSONID signatures are available [https://ffdev-info.github.io/jsonid/registry/ in a registry]. The long-term goal of this project is to enable other registries to delivery JSONID compatible signatures, e.g. PRONOM, Wikidata, and remove the need for a centralized resource like this.

=== Technical characteristics ===

JSONID explores potential technical characteristics that can
be attributed to serde formats. An example for a basic JSON object might
look as follows:

<pre>
{
"content_length": 82,
"number_of_lines": 9,
"line_warning": false,
"top_level_keys_count": 1,
"top_level_keys": [
"key1"
],
"top_level_types": [
"map"
],
"depth": 4,
"heterogeneous_list_types": false,
"fingerprint": {
"unf": "UNF:6:YEKQWBGm75JsN6H+8SzYRg==",
"cid": "bafkreidexnd3r76r5h3invwvu554573px5z4fg4uglw4pextmqc765kz64"
},
"doctype": "JSON",
"encoding": "UTF-8",
"agent": "jsonid/0.12.0 (ffdev-info)"
}
</pre>

The use of these technical characteristics will be explored in the
documentation and future writing.

=== Universal fingerprint ===

JSONID exports two universal fingerprints enabling the assertion of equivalence between different data objects. Universal Numerical Fingerprint (UNF) is also used in the [https://guides.dataverse.org/en/latest/developers/unf/index.html Dataverse project].
Content Identifiers (CIDs) come from the [https://docs.ipfs.tech/concepts/content-addressing/ IPFS project] and enable content-addressed storage within that ecosystem and others.

The significance of these fingerprinting techniques is their application
to identical data structures stored in different file formats.

The checksum of different file formats will always evaluate differently,
but analysed as data structures, we can begin to appraise data
beyond its presentation.

==== Fingerprinting example ====

<table>
<tr>
<td>Content</td>
<td><pre>
{
"hello": "world",
"goodbye": false,
"values": [
1,
2,
3.142
]
}</pre></td>
<td><pre>goodbye: false
hello: world
values:
- 1
- 2
- 3.142

</pre></td>
<td><pre>hello = "world"
values = [1, 2, 3.142]
goodbye = false

</pre></td>
</tr>
<tr>
<td>Type</td>
<td>JSON</td>
<td>YAML</td>
<td>TOML</td>
</tr>
<tr>
<td>Checksum (MD5)</td>
<td><code>bcd5a37f36ada2e4b72144d90a1427d5</code></td>
<td><code>b5e75fdc100032f2744eff1e1bdf5b88</code></td>
<td><code>04b33e24e0cc208c6bd70fabaef3a9c5</code></td>
</tr>
<tr>
<td>UNF</td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
</tr>
<tr>
<td>CID</td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
</tr>
</table>
 

=== Example identification output ===

JSONID has currently settled on a line-by-line, [[Fine Free File Command]] (File) and MIME-Like output. Fields are tab delineated and are human readable.

<pre>
/media/govdocs/govdocs_selected/LOG_1/074500.log [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/HTML_24/009970.html [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/TEXT_6/131522.txt [1] application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
</pre>

It is anticipated that users will help guide the ongoing development of JSONID's output to better support both user and repository workflows.

=== PRONOM signature development ===

JSONID provides a high-level language for output of PRONOM compatible
signatures. The feature set is still in its BETA phase but JSONID provides
two distinct capabilities:

==== Registry output ====

JSONID's registry can be output using the `--pronom` flag. A signature file
will be created under `jsonid_pronom.xml` which can be imported into DROID
for identification of document types registered with JSONID.

JSONID's registry is output alongisde a handful of baseline JSON signatures
designed to capture "plain"-JSON that is not yet encoded in the registry.

==== Signature development ====

A standalone `json2pronom` utility is provided for creation of potentially
robust DROID compatible signatures.

As a high-level language, signatures can be defined in easy to understand
syntax and then output consistently via the `json2pronom` utility. Signatures
include sensible defaults for whitespace and other aspects that are
difficult for signature developers to consistently anticipate when writing
JSON based signatures.

See the [https://github.com/ffdev-info/jsonid?tab=readme-ov-file#pronom JSONID docs] for more information.

== User experiences ==

* [https://docs.google.com/presentation/d/11zQTMwiXRa3UNyUSKTLwcCMos0gg9SpMxdrWOXF0lr4/edit?usp=sharing Introducing JSONID @ NTTW].
* [https://openpreservation.org/blogs/on-the-lookout-looking-for-common-json-files-and-introducing-jsonid/ On the lookout for common JSON files].

= Development Activity =

All development activity is visible on GitHub: https://github.com/ffdev-info/jsonid/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ffdev-info/jsonid/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ffdev-info/jsonid/commits/main.atom</rss>

JSONID

2026-01-04T22:47:25Z

Ross-spencer: Update information about technical characteristics

{{Infobox tool
|purpose=Identification of JSON, YAML, and TOML document types
|homepage=https://github.com/ffdev-info/jsonid
|sourcecode=https://github.com/ffdev-info/jsonid
|license=Apache 2.0
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=JSON, JSONL, YAML, TOML
|function=File Format Identification
}}
{{Infobox tool details}}
== Description ==


Identification of JSON, YAML, and TOML document types.

== Functionality ==

JSONID parses serialization/deserialization formats ([https://en.wikipedia.org/wiki/Serialization "serde"]) such as JSON, YAML, and TOML to provide unambiguous identification.
JSONID also introduces a declarative syntax for writing document type signatures to enable identification of specific serde document types.
Key-value attributes can be shared across formats, and so signatures for JSON and YAML, for example, need only be written once.

=== Registry ===

As a temporary placeholder JSONID signatures are available [https://ffdev-info.github.io/jsonid/registry/ in a registry]. The long-term goal of this project is to enable other registries to delivery JSONID compatible signatures, e.g. PRONOM, Wikidata, and remove the need for a centralized resource like this.

=== Technical characteristics ===

JSONID explores potential technical characteristics that can
be attributed to serde formats. An example for a basic JSON object might
look as follows:

<pre>
{
"content_length": 82,
"number_of_lines": 9,
"line_warning": false,
"top_level_keys_count": 1,
"top_level_keys": [
"key1"
],
"top_level_types": [
"map"
],
"depth": 4,
"heterogeneous_list_types": false,
"fingerprint": {
"unf": "UNF:6:YEKQWBGm75JsN6H+8SzYRg==",
"cid": "bafkreidexnd3r76r5h3invwvu554573px5z4fg4uglw4pextmqc765kz64"
},
"doctype": "JSON",
"encoding": "UTF-8",
"agent": "jsonid/0.12.0 (ffdev-info)"
}
</pre>

The use of these technical characteristics will be explored in the
documentation and future writing.

=== Universal fingerprint ===

JSONID exports two universal fingerprints enabling the assertion of equivalence between different data objects. Universal Numerical Fingerprint (UNF) is also used in the [https://guides.dataverse.org/en/latest/developers/unf/index.html Dataverse project].
Content Identifiers (CIDs) come from the [https://docs.ipfs.tech/concepts/content-addressing/ IPFS project] and enable content-addressed storage within that ecosystem and others.

The significance of these fingerprinting techniques is their application
to identical data structures stored in different file formats.

The checksum of different file formats will always evaluate differently,
but analysed as data structures, we can begin to appraise data
beyond its presentation.

==== Fingerprinting example ====

<table>
<tr>
<td>Content</td>
<td><pre>
{
"hello": "world",
"goodbye": false,
"values": [
1,
2,
3.142
]
}</pre></td>
<td><pre>goodbye: false
hello: world
values:
- 1
- 2
- 3.142

</pre></td>
<td><pre>hello = "world"
values = [1, 2, 3.142]
goodbye = false

</pre></td>
</tr>
<tr>
<td>Type</td>
<td>JSON</td>
<td>YAML</td>
<td>TOML</td>
</tr>
<tr>
<td>Checksum (MD5)</td>
<td><code>bcd5a37f36ada2e4b72144d90a1427d5</code></td>
<td><code>b5e75fdc100032f2744eff1e1bdf5b88</code></td>
<td><code>04b33e24e0cc208c6bd70fabaef3a9c5</code></td>
</tr>
<tr>
<td>UNF</td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
<td><code>UNF:6:97EfAWBIQlObVCVwa7kc0g==</code></td>
</tr>
<tr>
<td>CID</td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
<td><code>bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy</code></td>
</tr>
</table>
 

=== PRONOM signature development ===

JSONID provides a high-level language for output of PRONOM compatible
signatures. The feature set is still in its BETA phase but JSONID provides
two distinct capabilities:

==== Registry output ====

JSONID's registry can be output using the `--pronom` flag. A signature file
will be created under `jsonid_pronom.xml` which can be imported into DROID
for identification of document types registered with JSONID.

JSONID's registry is output alongisde a handful of baseline JSON signatures
designed to capture "plain"-JSON that is not yet encoded in the registry.

==== Signature development ====

A standalone `json2pronom` utility is provided for creation of potentially
robust DROID compatible signatures.

As a high-level language, signatures can be defined in easy to understand
syntax and then output consistently via the `json2pronom` utility. Signatures
include sensible defaults for whitespace and other aspects that are
difficult for signature developers to consistently anticipate when writing
JSON based signatures.

See the [https://github.com/ffdev-info/jsonid?tab=readme-ov-file#pronom JSONID docs] for more information.

== User experiences ==

* [https://docs.google.com/presentation/d/11zQTMwiXRa3UNyUSKTLwcCMos0gg9SpMxdrWOXF0lr4/edit?usp=sharing Introducing JSONID @ NTTW].
* [https://openpreservation.org/blogs/on-the-lookout-looking-for-common-json-files-and-introducing-jsonid/ On the lookout for common JSON files].

= Development Activity =

All development activity is visible on GitHub: https://github.com/ffdev-info/jsonid/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ffdev-info/jsonid/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ffdev-info/jsonid/commits/main.atom</rss>

JSONID

2026-01-04T17:34:43Z

Ross-spencer: Add information on technical characteristics

JSONID

2026-01-04T17:04:25Z

Ross-spencer: Add information on PRONOM output

JSONID

2026-01-03T12:48:53Z

Ross-spencer: Add new record for JSONID

OPF ODF Validator

2026-01-03T12:42:50Z

Ross-spencer: Add new record for OPF ODF Validator

{{Infobox tool
|purpose=Open Source Open Document Validator
|homepage=https://odf.openpreservation.org
|sourcecode=https://github.com/openpreserve/odf-validator
|license=BSD 3-Clause
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Java
|formats_in=
|function=File Format Validation
}}
{{Infobox tool details}}
== Description ==


Open source tool for validation of Open Document Format files (ODF). An earlier utility by the same name was written by [https://coptr.digipres.org/ODF_Validator The Document Foundation].

= Development Activity =

All development activity is visible on GitHub: https://github.com/openpreserve/odf-validator/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/openpreserve/odf-validator/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/openpreserve/odf-validator/commits/main.atom</rss>

ODF Validator

2026-01-03T12:39:25Z

Ross-spencer: Add references

{{Infobox tool
|purpose=ODF Validator is a tool that validates OpenDocument files and checks them for certain conformance criteria.
|homepage=https://odftoolkit.org/conformance/ODFValidator.html
|license=Apache License, Version 2.0
|function=Validation, Metadata Extraction
|content=Document
}}
{{Infobox tool details}}
== Description ==
ODF Validator is a tool from the [https://github.com/tdf Document Foundation] that validates OpenDocument files and checks them for certain conformance criteria. It is available as Java library for command line or embedded use, as well as a [https://odfvalidator.org/ web application archive (WAR)]. ODF Validator is part of the [Apache ODF Toolkit].

== User Experiences ==

* [http://www.openplanetsfoundation.org/comment/529#comment-529 Test by KEEP Solutions]:

<blockquote>Apache ODF Validator gives many false-negatives and false-positives (...) [b]ut (..) the schemas are publicly available and the method to validate is well defined by OASIS (...).</blockquote>

* News from the ODF toolkit at [https://archive.fosdem.org/2023/schedule/event/lotech_odftoolkit/attachments/slides/5687/export/events/attachments/lotech_odftoolkit/slides/5687/2023_FOSDEM_News_from_the_ODF_Toolkit_10min.pdf FOSDEM 2023]

= Development Activity =

All development activity is visible on GitHub: https://github.com/tdf/odftoolkit

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/tdf/odftoolkit/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/tdf/odftoolkit/commits/master.atom</rss>

== Informative references ==

* The Document Foundation [https://www.documentfoundation.org/ website].

ODF Validator

2026-01-03T12:27:16Z

Ross-spencer: Additional context to FOSDEM presentation

{{Infobox tool
|purpose=ODF Validator is a tool that validates OpenDocument files and checks them for certain conformance criteria.
|homepage=https://odftoolkit.org/conformance/ODFValidator.html
|license=Apache License, Version 2.0
|function=Validation, Metadata Extraction
|content=Document
}}
{{Infobox tool details}}
== Description ==
ODF Validator is a tool that validates OpenDocument files and checks them for certain conformance criteria. It is available as Java library for command line or embedded use, as well as a [https://odfvalidator.org/ web application archive (WAR)]. ODF Validator is part of the [Apache ODF Toolkit].

== User Experiences ==

* [http://www.openplanetsfoundation.org/comment/529#comment-529 Test by KEEP Solutions]:

<blockquote>Apache ODF Validator gives many false-negatives and false-positives (...) [b]ut (..) the schemas are publicly available and the method to validate is well defined by OASIS (...).</blockquote>

* News from the ODF toolkit at [https://archive.fosdem.org/2023/schedule/event/lotech_odftoolkit/attachments/slides/5687/export/events/attachments/lotech_odftoolkit/slides/5687/2023_FOSDEM_News_from_the_ODF_Toolkit_10min.pdf FOSDEM 2023]

= Development Activity =

All development activity is visible on GitHub: https://github.com/tdf/odftoolkit

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/tdf/odftoolkit/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/tdf/odftoolkit/commits/master.atom</rss>

ODF Validator

2026-01-01T22:52:15Z

Ross-spencer: Fixes info box and adds development info

{{Infobox tool
|purpose=ODF Validator is a tool that validates OpenDocument files and checks them for certain conformance criteria.
|homepage=https://odftoolkit.org/conformance/ODFValidator.html
|license=Apache License, Version 2.0
|function=Validation, Metadata Extraction
|content=Document
}}
{{Infobox tool details}}
== Description ==
ODF Validator is a tool that validates OpenDocument files and checks them for certain conformance criteria. It is available as Java library for command line or embedded use, as well as a [https://odfvalidator.org/ web application archive (WAR)]. ODF Validator is part of the [Apache ODF Toolkit].

== User Experiences ==

* [http://www.openplanetsfoundation.org/comment/529#comment-529 Test by KEEP Solutions]:

<blockquote>Apache ODF Validator gives many false-negatives and false-positives (...) [b]ut (..) the schemas are publicly available and the method to validate is well defined by OASIS (...).</blockquote>

* [https://archive.fosdem.org/2023/schedule/event/lotech_odftoolkit/attachments/slides/5687/export/events/attachments/lotech_odftoolkit/slides/5687/2023_FOSDEM_News_from_the_ODF_Toolkit_10min.pdf FOSDEM 2023 presentation]

= Development Activity =

All development activity is visible on GitHub: https://github.com/tdf/odftoolkit

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/tdf/odftoolkit/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/tdf/odftoolkit/commits/master.atom</rss>

Demystify

2025-11-28T15:50:37Z

Ross-spencer: Modifies infobox

{{Infobox tool
|purpose=Format Identification, Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=WebAssembly + sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" (formerly 'DROID Siegfried Sqlite Analysis Engine') with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a denylist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

=== Demystify Lite ===

[https://ross-spencer.github.io/demystify-lite/ Demystify Lite] provides a Pyscript/WASM implementation of Demystify's features and runs completely browser side for users with DROID or Siegfried reports that they would like to see analyzed.

=== Siegfried integration ===

Demystify lite integrates a WASM build of Siegfried enabling client-side identification of your digital records and analysis within one secure tool, for more information, see Ross Spencer's [https://exponentialdecay.co.uk/blog/client-side-identification-and-reporting-pipeline-with-siegfried-and-demystify-lite/ brief integration report].

=== Fractal in Detail ===

[https://journal.code4lib.org/articles/16351 Fractal in Detail] in the Code4Lib Journal examines some of the motivations behind Demystify, generalizing its context and considering its source data alongside similar tools like Brunnhilde, Freud, and FileDriller.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]
**'''[2022-05-09]''' [https://journal.code4lib.org/articles/16351 Fractal in detail: What information is in a file format identification report?]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Demystify

2025-11-28T15:47:32Z

Ross-spencer: Update with information about Demystify's Siegfried integration

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" (formerly 'DROID Siegfried Sqlite Analysis Engine') with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a denylist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

=== Demystify Lite ===

[https://ross-spencer.github.io/demystify-lite/ Demystify Lite] provides a Pyscript/WASM implementation of Demystify's features and runs completely browser side for users with DROID or Siegfried reports that they would like to see analyzed.

=== Siegfried integration ===

Demystify lite integrates a WASM build of Siegfried enabling client-side identification of your digital records and analysis within one secure tool, for more information, see Ross Spencer's [https://exponentialdecay.co.uk/blog/client-side-identification-and-reporting-pipeline-with-siegfried-and-demystify-lite/ brief integration report].

=== Fractal in Detail ===

[https://journal.code4lib.org/articles/16351 Fractal in Detail] in the Code4Lib Journal examines some of the motivations behind Demystify, generalizing its context and considering its source data alongside similar tools like Brunnhilde, Freud, and FileDriller.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]
**'''[2022-05-09]''' [https://journal.code4lib.org/articles/16351 Fractal in detail: What information is in a file format identification report?]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Demystify

2025-11-26T13:21:26Z

Ross-spencer: Add link to Fractal in Detail

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" (formerly 'DROID Siegfried Sqlite Analysis Engine') with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a denylist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

=== Demystify Lite ===

[https://ross-spencer.github.io/demystify-lite/ Demystify Lite] provides a Pyscript/WASM implementation of Demystify's features and runs completely browser side for users with DROID or Siegfried reports that they would like to see analyzed.

=== Fractal in Detail ===

[https://journal.code4lib.org/articles/16351 Fractal in Detail] in the Code4Lib Journal examines some of the motivations behind Demystify, generalizing its context and considering its source data alongside similar tools like Brunnhilde, Freud, and FileDriller.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]
**'''[2022-05-09]''' [https://journal.code4lib.org/articles/16351 Fractal in detail: What information is in a file format identification report?]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Demystify

2025-11-26T11:59:14Z

Ross-spencer: Teminology

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" (formerly 'DROID Siegfried Sqlite Analysis Engine') with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a denylist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

=== Demystify Lite ===

[https://ross-spencer.github.io/demystify-lite/ Demystify Lite] provides a Pyscript/WASM implementation of Demystify's features and runs completely browser side for users with DROID or Siegfried reports that they would like to see analyzed.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]
**'''[2022-05-09]''' [https://journal.code4lib.org/articles/16351 Fractal in detail: What information is in a file format identification report?]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Whatsmapper

2025-09-17T08:19:10Z

Ross-spencer: Provide a brief description of function

{{Infobox tool
|purpose=Preserve Whatsapp messages as a web page
|homepage=https://github.com/ross-spencer/whatsmapper
|sourcecode=https://github.com/ross-spencer/whatsmapper
|license=
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=
|function=File Format Migration, Rendering
}}
{{Infobox tool details}}
== Description ==


Whatsmapper is a utility to convert Whatsapp messages into other formats. Its primary function is to provide a demo of this functionality and for it to be built upon by interested parties in the community.

== Function ==

Whatsmapper outputs Whatsapp messages in HTML counting the number of attachments there are in the data along the way. Images and so on are output inline. It embeds a default CSS (Cascading Style Sheet) but this CSS can be easily overridden to provide different styles per user requirement.

= Development Activity =

All development activity is visible on GitHub: https://github.com/ross-spencer/whatsmapper/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ross-spencer/whatsmapper/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ross-spencer/whatsmapper/commits/main.atom</rss>

Whatsmapper

2025-09-16T17:37:03Z

Ross-spencer: Fix title

{{Infobox tool
|purpose=Preserve Whatsapp messages as a web page
|homepage=https://github.com/ross-spencer/whatsmapper
|sourcecode=https://github.com/ross-spencer/whatsmapper
|license=
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=
|function=File Format Migration, Rendering
}}
{{Infobox tool details}}
== Description ==


Whatsmapper is a utility to convert Whatsapp messages into other formats. Its primary function is to provide a demo of this functionality and for it to be built upon by interested parties in the community.

= Development Activity =

All development activity is visible on GitHub: https://github.com/ross-spencer/whatsmapper/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ross-spencer/whatsmapper/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ross-spencer/whatsmapper/commits/main.atom</rss>

Whatsmapper

2025-09-16T17:36:44Z

Ross-spencer: Update commits atom feed

{{Infobox tool
|purpose=Preserve Whatsapp messages as a webpage (Prototype)
|homepage=https://github.com/ross-spencer/whatsmapper
|sourcecode=https://github.com/ross-spencer/whatsmapper
|license=
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Python
|formats_in=
|function=File Format Migration, Rendering
}}
{{Infobox tool details}}
== Description ==


Whatsmapper is a utility to convert Whatsapp messages into other formats. Its primary function is to provide a demo of this functionality and for it to be built upon by interested parties in the community.

= Development Activity =

All development activity is visible on GitHub: https://github.com/ross-spencer/whatsmapper/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ross-spencer/whatsmapper/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ross-spencer/whatsmapper/commits/main.atom</rss>

Whatsmapper

2025-09-16T17:35:50Z

Ross-spencer: Add new entry

Ruffle

2025-09-04T07:06:04Z

Ross-spencer: Add new entry

{{Infobox tool
|purpose=A flash player emulator written in Rust
|homepage=https://github.com/ruffle-rs/ruffle
|sourcecode=https://github.com/ruffle-rs/ruffle
|license=Apache License 2.0
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS
|language=Rust
|function=Viewer
}}

== Description ==


Ruffle is a flash player emulator written in Rust. It is helpful to users with content such as SWF who would like to view the content and engage with it interactively.

As open source software the utility might also offer potential for developers to extract information about ''significant properties'' about Flash content by accessing the utility's underlying APIs.

= Development Activity =

All development activity is visible on GitHub: http://github.com/ruffle-rs/ruffle/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ruffle-rs/ruffle/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ruffle-rs/ruffle/commits/master.atom</rss>

Bitflip

2024-11-22T07:54:06Z

Ross-spencer: Update branch name

{{Infobox tool
|purpose=An interactive tool that can be used to demonstrate the effects of information decay ("bitrot") in digital files.
|homepage=https://github.com/aybabtme/bitflip
|sourcecode=https://github.com/aybabtme/bitflip
|license=MIT License
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS, FreeBSD
|language=Golang
|function=Teaching, Research, Experimentation
}}

== Description ==


Bitflip is a tool designed to flip bits in digital files. It can provide a didactic reference for students of digital preservation to understand the effects of information decay ("bitrot") in digital files.

Its functionality similar to Manfred Thaller's [https://web.archive.org/web/20090323121026/http://hki.uni-koeln.de/material/shotGun/ Digital Preservation shotGun].

== User Experiences ==


* [https://www.youtube.com/watch?v=tdnyrk5m5vQ Example of an emulated video game glitched using Bitflip].

= Development Activity =

All development activity is visible on GitHub: http://github.com/aybabtme/bitflip/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/aybabtme/bitflip/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/aybabtme/bitflip/commits/master.atom</rss>

Bitflip

2024-11-22T07:53:10Z

Ross-spencer: Improve display of record

{{Infobox tool
|purpose=An interactive tool that can be used to demonstrate the effects of information decay ("bitrot") in digital files.
|homepage=https://github.com/aybabtme/bitflip
|sourcecode=https://github.com/aybabtme/bitflip
|license=MIT License
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS, FreeBSD
|language=Golang
|function=Teaching, Research, Experimentation
}}

== Description ==


Bitflip is a tool designed to flip bits in digital files. It can provide a didactic reference for students of digital preservation to understand the effects of information decay ("bitrot") in digital files.

Its functionality similar to Manfred Thaller's [https://web.archive.org/web/20090323121026/http://hki.uni-koeln.de/material/shotGun/ Digital Preservation shotGun].

== User Experiences ==


* [https://www.youtube.com/watch?v=tdnyrk5m5vQ Example of an emulated video game glitched using Bitflip].

= Development Activity =

All development activity is visible on GitHub: http://github.com/aybabtme/bitflip/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/aybabtme/bitflip/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/aybabtme/bitflip/commits/main.atom</rss>

Bitflip

2024-11-22T07:50:18Z

Ross-spencer: Add bitflip

{{Infobox tool
|purpose=An interactive tool that can be used to demonstrate the effects of information decay ("bitrot") in digital files.
|homepage=https://github.com/aybabtme/bitflip
|sourcecode=https://github.com/aybabtme/bitflip
|license=MIT License
|cost=Free and Open Source (FOSS)
|platforms=Windows, Linux, MacOS, FreeBSD
|language=Golang
|function=Teaching, Research, Experimentation
}}
{{Infobox tool details
|releases_rss=https://github.com/aybabtme/bitflip/releases.atom
}}
== Description ==


Bitflip is a tool designed to flip bits in digital files. It can provide a didactic reference for students of digital preservation to understand the effects of information decay ("bitrot") in digital files.

Its functionality similar to Manfred Thaller's [https://web.archive.org/web/20090323121026/http://hki.uni-koeln.de/material/shotGun/ Digital Preservation shotGun].

== User Experiences ==


* [https://www.youtube.com/watch?v=tdnyrk5m5vQ Example of an emulated video game glitched using Bitflip].

ArchiveBox

2024-03-27T15:33:01Z

Ross-spencer: Fixup headings

{{Infobox tool
|purpose=ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data.
|homepage=https://archivebox.io/
|sourcecode=https://github.com/ArchiveBox/ArchiveBox
|license=MIT License
|cost=Free
|platforms=Docker + Python
|language=English
|Wikidata ID=Q63097629
|formats_out=PDF, WARC, JSON
|function=Web Capture, Personal Archiving
}}
{{Infobox tool details}}
== Description ==


ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. ArchiveBox was first released in 2017.

== User Experiences ==


**'''[2024-03-27]''' [https://demo.archivebox.io/public/ ArchiveBox Online Demo.]

== Development activity ==

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ArchiveBox/ArchiveBox/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ArchiveBox/ArchiveBox/commits/main.atom</rss>

Demystify

2024-03-27T07:09:09Z

Ross-spencer: Fixup date

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" (formerly 'DROID Siegfried Sqlite Analysis Engine') with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

=== Demystify Lite ===

[https://ross-spencer.github.io/demystify-lite/ Demystify Lite] provides a Pyscript/WASM implementation of Demystify's features and runs completely browser side for users with DROID or Siegfried reports that they would like to see analyzed.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]
**'''[2022-05-09]''' [https://journal.code4lib.org/articles/16351 Fractal in detail: What information is in a file format identification report?]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Demystify

2024-03-27T07:08:29Z

Ross-spencer: Record updates including previous name; demystify-lite additions; more user-experiences

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" (formerly 'DROID Siegfried Sqlite Analysis Engine') with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

=== Demystify Lite ===

[https://ross-spencer.github.io/demystify-lite/ Demystify Lite] provides a Pyscript/WASM implementation of Demystify's features and runs completely browser side for users with DROID or Siegfried reports that they would like to see analyzed.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]
**'''[2022-05-0]''' [https://journal.code4lib.org/articles/16351 Fractal in detail: What information is in a file format identification report?]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Demystify

2024-03-27T07:03:49Z

Ross-spencer: Ross-spencer moved page DROID Siegfried Sqlite Analysis Engine to Demystify: The name of the application is no longer 'DROID Siegfried Sqlite Analysis Engine'

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

Demystify

2024-03-27T06:59:02Z

Ross-spencer: Update release links

{{Infobox tool
|purpose=Format Identification Analysis and Reporting
|homepage=https://github.com/exponential-decay/demystify
|license=Open source (see URL above)
|platforms=sqlite + Python + text/html
|function=Metadata Extraction, Content Profiling, De-Duplication
}}
{{Infobox tool details}}
== Description ==

Now known as "Demystify" with thanks to Joshua Ng for the suggestion to rename it. Demystify is an engine for the analysis of [https://github.com/digital-preservation/droid DROID] CSV export files, [https://github.com/richardlehane/siegfried Siegfried] YAML export files, and Siegfried 'DROID compatible' output. The tool has three purposes, break the exports into their components and store them within a table in a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions.

The tool provides archivist definitions for each of the sections output; these definitions are customizable. The tool also supports output of statistics about files that may require further triage or may not be appropriate for long-term preservation based on institutional rules, in the form of a blacklist. The tool also analyses file names and directory names for non-ascii characters, and also characteristics that may present problems cross-file-system based on known Microsoft rules: http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

The engine can be used to generate a list of file paths for files that may present digital preservation risks (Rogues) or files which on the surface i.e. via identification alone, look okay (Heroes) and these listings can be used in conjunction with [http://manpages.ubuntu.com/manpages/trusty/man1/rsync.1.html rsync] to isolate these sets from one-another to be more flexible to work with.

== User Experiences ==

*Blog entries from the tool author, Ross Spencer:
**'''[2014-06-03]''' [http://www.openplanetsfoundation.org/blogs/2014-06-03-analysis-engine-droid-csv-export Describing the creation and purpose of the tool.]
**'''[2015-08-25]''' [http://openpreservation.org/blog/2015/08/25/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery/ Using the output of the tool to create a digital preservation rogues gallery.]
**'''[2016-05-23]''' [http://openpreservation.org/blog/2016/05/23/whats-in-a-namespace-the-marriage-of-droid-and-siegfried-analysis/ The integration of Siegfried output for consistent and repeatable reporting.]
**'''[2016-05-24]''' [http://openpreservation.org/blog/2016/05/24/while-were-on-the-subject-a-few-more-points-of-interest-about-the-siegfrieddroid-analysis-tool/ Creating a multi-lingual consistent, digital preservation dialect and exploring alternative methods of format identification using Siegfried's capabilities.]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/demystify/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/demystify/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/demystify/commits/main.atom</rss>

ArchiveBox

2024-03-27T06:57:54Z

Ross-spencer: Initial draft of ArchiveBox record

{{Infobox tool
|purpose=ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data.
|homepage=https://archivebox.io/
|sourcecode=https://github.com/ArchiveBox/ArchiveBox
|license=MIT License
|cost=Free
|platforms=Docker + Python
|language=English
|Wikidata ID=Q63097629
|formats_out=PDF, WARC, JSON
|function=Web Capture, Personal Archiving
}}
{{Infobox tool details}}
== Description ==


ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. ArchiveBox was first released in 2017.

== User Experiences ==


**'''[2024-03-27]''' [https://demo.archivebox.io/public/ ArchiveBox Online Demo.]

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ArchiveBox/ArchiveBox/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ArchiveBox/ArchiveBox/commits/main.atom</rss>

Sumfolder1

2024-03-26T20:25:23Z

Ross-spencer: Update cost

{{Infobox tool
|purpose=sumfolder1 is a utility for use within the archival and digital preservation community to generate checksums for file system directories, and to generate an overall "collection" checksum for a given set of files. The utility may be used in support of de-duplication at a directory/folder level.
|homepage=https://pypi.org/project/sumfolder1/
|sourcecode=https://github.com/ross-spencer/sumfolder1
|license=GPL-3.0
|cost=Free as in kittens (or a donation of 1 million dollars to an offshore account if you have the funding)
|platforms=Python 3
|function=Appraisal, De-Duplication, Fixity
}}
{{Infobox tool details}}
== Description ==


sumfolder1 uses a DROID CSV output to generate checksums for file system directories and to generate an overall "collection" checksum for a given set of files. It can be used for fixity and de-duplication at the folder level.

== User Experiences ==


**'''[2023-01-16]''' [https://openpreservation.org/blogs/what-is-the-checksum-of-a-directory/ What is the checksum of a directory? Using DROID reports and the concepts behind Merkle Trees to generate directory and collection checksums.]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/sumfolder1/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ross-spencer/sumfolder1/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ross-spencer/sumfolder1/commits/main.atom</rss>

Sumfolder1

2024-03-26T20:22:33Z

Ross-spencer: Update dev links

{{Infobox tool
|purpose=sumfolder1 is a utility for use within the archival and digital preservation community to generate checksums for file system directories, and to generate an overall "collection" checksum for a given set of files. The utility may be used in support of de-duplication at a directory/folder level.
|homepage=https://pypi.org/project/sumfolder1/
|sourcecode=https://github.com/ross-spencer/sumfolder1
|license=GPL-3.0
|cost=Open-source
|platforms=Python 3
|function=Appraisal, De-Duplication, Fixity
}}
{{Infobox tool details}}
== Description ==


sumfolder1 uses a DROID CSV output to generate checksums for file system directories and to generate an overall "collection" checksum for a given set of files. It can be used for fixity and de-duplication at the folder level.

== User Experiences ==


**'''[2023-01-16]''' [https://openpreservation.org/blogs/what-is-the-checksum-of-a-directory/ What is the checksum of a directory? Using DROID reports and the concepts behind Merkle Trees to generate directory and collection checksums.]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/sumfolder1/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/ross-spencer/sumfolder1/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/ross-spencer/sumfolder1/commits/main.atom</rss>

Sumfolder1

2024-03-26T20:21:31Z

Ross-spencer: After a year no one in the community uploaded this to the registry so I thought I should...

{{Infobox tool
|purpose=sumfolder1 is a utility for use within the archival and digital preservation community to generate checksums for file system directories, and to generate an overall "collection" checksum for a given set of files. The utility may be used in support of de-duplication at a directory/folder level.
|homepage=https://pypi.org/project/sumfolder1/
|sourcecode=https://github.com/ross-spencer/sumfolder1
|license=GPL-3.0
|cost=Open-source
|platforms=Python 3
|function=Appraisal, De-Duplication, Fixity
}}
{{Infobox tool details}}
== Description ==


sumfolder1 uses a DROID CSV output to generate checksums for file system directories and to generate an overall "collection" checksum for a given set of files. It can be used for fixity and de-duplication at the folder level.

== User Experiences ==


**'''[2023-01-16]''' [https://openpreservation.org/blogs/what-is-the-checksum-of-a-directory/ What is the checksum of a directory? Using DROID reports and the concepts behind Merkle Trees to generate directory and collection checksums.]

= Development Activity =

All development activity is visible on GitHub: http://github.com/ross-spencer/sumfolder1/commits

=== Release Feed ===
Below the last 3 release feeds:
<rss max=3>https://github.com/exponential-decay/sumfolder1/releases.atom</rss>

=== Activity Feed ===
Below the last 5 commits:
<rss max=5>https://github.com/exponential-decay/sumfolder1/commits/main.atom</rss>

User:Ross-spencer

2023-03-06T12:32:58Z

Ross-spencer: Added link to user contributions

'''Digital Preservation Quartermaster''', information about me can be found via my Twitter: https://twitter.com/beet_keeper

My contributions to this wiki:

* [https://coptr.digipres.org/index.php?title=Special%3AContributions&tagfilter=&contribs=user&target=ross-spencer&namespace=&year=&month=-1 User Contributions]