JSONID

From COPTR
Jump to navigation Jump to search
Identification of JSON, YAML, and TOML document types
Homepage:https://github.com/ffdev-info/jsonid
Source Code:https://github.com/ffdev-info/jsonid
License:Apache 2.0
Cost:Free and Open Source (FOSS)
Platforms:Windows, Linux, MacOS
Language:Python
Input Formats:JSON, JSONL, YAML, TOML
Function:File Format Identification



Description

Identification of JSON, YAML, and TOML document types.

Functionality

JSONID parses serialization/deserialization formats ("serde") such as JSON, YAML, and TOML to provide unambiguous identification. JSONID also introduces a declarative syntax for writing document type signatures to enable identification of specific serde document types. Key-value attributes can be shared across formats, and so signatures for JSON and YAML, for example, need only be written once.

Registry

As a temporary placeholder JSONID signatures are available in a registry. The long-term goal of this project is to enable other registries to delivery JSONID compatible signatures, e.g. PRONOM, Wikidata, and remove the need for a centralized resource like this.

Technical characteristics

JSONID explores potential technical characteristics that can be attributed to serde formats. An example for a basic JSON object might look as follows: 

{
  "content_length": 82,
  "number_of_lines": 9,
  "line_warning": false,
  "top_level_keys_count": 1,
  "top_level_keys": [
    "key1"
  ],
  "top_level_types": [
    "map"
  ],
  "depth": 4,
  "heterogeneous_list_types": false,
  "fingerprint": {
    "unf": "UNF:6:YEKQWBGm75JsN6H+8SzYRg==",
    "cid": "bafkreidexnd3r76r5h3invwvu554573px5z4fg4uglw4pextmqc765kz64"
  },
  "doctype": "JSON",
  "encoding": "UTF-8",
  "agent": "jsonid/0.12.0 (ffdev-info)"
}

The use of these technical characteristics will be explored in the documentation and future writing.

Universal fingerprint

JSONID exports two universal fingerprints enabling the assertion of equivalence between different data objects. Universal Numerical Fingerprint (UNF) is also used in the Dataverse project. Content Identifiers (CIDs) come from the IPFS project and enable content-addressed storage within that ecosystem and others.

The significance of these fingerprinting techniques is their application to identical data structures stored in different file formats.

The checksum of different file formats will always evaluate differently, but analysed as data structures, we can begin to appraise data beyond its presentation.

Fingerprinting example

Content 
{
    "hello": "world",
    "goodbye": false,
    "values": [
        1,
        2,
        3.142
    ]
}
 
goodbye: false
hello: world
values:
- 1
- 2
- 3.142



hello = "world"
values = [1, 2, 3.142]
goodbye = false
Type JSON YAML TOML
Checksum (MD5) bcd5a37f36ada2e4b72144d90a1427d5 b5e75fdc100032f2744eff1e1bdf5b88 04b33e24e0cc208c6bd70fabaef3a9c5
UNF UNF:6:97EfAWBIQlObVCVwa7kc0g== UNF:6:97EfAWBIQlObVCVwa7kc0g== UNF:6:97EfAWBIQlObVCVwa7kc0g==
CID bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy bafkreiawsimwdn4blnb7scz2cfwtdksifrayccsl3z6gmxam6uxddctkoy


Example identification output

JSONID has currently settled on a line-by-line, Fine Free File Command (File) and MIME-Like output. Fields are tab delineated and are human readable. 

/media/govdocs/govdocs_selected/LOG_1/074500.log	[1]	application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/HTML_24/009970.html	[1]	application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON
/media/govdocs/govdocs_selected/TEXT_6/131522.txt	[1]	application/json; charset=UTF-8; doctype="JavaScript Object Notation (JSON)"; ref=jrid:JSON

It is anticipated that users will help guide the ongoing development of JSONID's output to better support both user and repository workflows.

PRONOM signature development

JSONID provides a high-level language for output of PRONOM compatible signatures. The feature set is still in its BETA phase but JSONID provides two distinct capabilities:

Registry output

JSONID's registry can be output using the `--pronom` flag. A signature file will be created under `jsonid_pronom.xml` which can be imported into DROID for identification of document types registered with JSONID.

JSONID's registry is output alongisde a handful of baseline JSON signatures designed to capture "plain"-JSON that is not yet encoded in the registry.

Signature development

A standalone `json2pronom` utility is provided for creation of potentially robust DROID compatible signatures.

As a high-level language, signatures can be defined in easy to understand syntax and then output consistently via the `json2pronom` utility. Signatures include sensible defaults for whitespace and other aspects that are difficult for signature developers to consistently anticipate when writing JSON based signatures.

See the JSONID docs for more information.

Sample files

Sample files used for the development of JSONID signatures can be found in their own repository.

User experiences

Development Activity

All development activity is visible on GitHub: https://github.com/ffdev-info/jsonid/commits

Release Feed

Below the last 3 release feeds:

2026-04-08 07:17:21
[tag:github.com,2008:Repository/964720703/0.12.4 0.12.4]
by github-actions[bot]
2026-01-15 15:59:23
[tag:github.com,2008:Repository/964720703/0.12.3 0.12.3]
by github-actions[bot]
2026-01-04 23:05:32
[tag:github.com,2008:Repository/964720703/0.12.2 0.12.2]
by github-actions[bot]

Activity Feed

Below the last 5 commits:

2026-08-08 17:04:58
[tag:github.com,2008:Grit::Commit/2e6c20c880045c199364ac9ebb9076e1c5014a0b Bump tox from 4.55.0 to 4.58.0]
by dependabot https://github.com/dependabot
2026-08-08 17:02:11
[tag:github.com,2008:Grit::Commit/8c2ebf31220d94ed3f7ac16c25f060757de7474b Bump pre-commit from 4.6.0 to 4.6.1]
by dependabot https://github.com/dependabot
2026-08-08 16:35:59
[tag:github.com,2008:Grit::Commit/bdc5025f62e3c437ad0f7c1de512321644355a25 Bump actions/setup-python from 6 to 7]
by dependabot https://github.com/dependabot
2026-08-08 16:34:48
[tag:github.com,2008:Grit::Commit/4ad01afc9a1d6522ac0c935c6c4244d1b72f2e5a Bump pylint from 4.0.5 to 4.0.6]
by dependabot https://github.com/dependabot
2026-08-08 16:32:10
[tag:github.com,2008:Grit::Commit/121a1e6d120b7006785420245544c7c5eba8db4a Bump pytest from 9.0.3 to 9.1.1]
by dependabot https://github.com/dependabot

Retrieved from ‘https://coptr.digipres.org/index.php?title=JSONID&oldid=6664’