Persistent Identifiers & Discovery (PIDs, DIDs, GUIDs & Scoped IDs)

Persistent Identifiers & Discovery (PIDs, DIDs, GUIDs & Scoped IDs)#

Findability and Discovery are fundamental pillars of the FAIR Principles (F1: (Meta)data are assigned a globally unique and persistent identifier) and CDIF (Cross-Domain Interoperability Framework).

This guide explains how to document primary canonical identifiers ($id, $anchor), web-resolvable persistent identifiers (DOIs, w3ids, DIDs, ARKs, Handles), random synthetic identifiers (GUIDs/UUIDs), and secondary identifiers across administrative uniqueness scopes (global, regional, national, institutional, project, local, other).

1. Primary Identifiers: `$id` and `$anchor`#

Standard JSON Schema Draft 2020-12 uses $id to declare the canonical base URI of a schema object. In FAIR Data JSON Schema, $id serves as the primary Persistent Identifier (PID) or Decentralized Identifier (DID) for a dataset or data product.

Persistent Web URIs as `$id`#

Whenever possible, use a web-resolvable persistent URI as the $id of your root dataset schema:

{
  "$schema": "https://highvaluedata.net/fair-data-schema",
  "$id": "https://doi.org/10.5281/zenodo.1234567",
  "title": "Global Longitudinal Demographic Survey (2024)"
}

Other valid persistent $id schemes include:

w3ids: https://w3id.org/my-project/dataset/v1
ARKs: https://n2t.net/ark:/12345/fk36108
Handles: https://hdl.handle.net/10222/3456
Decentralized Identifiers (DIDs): did:example:123456789abcdefghi

Fragment Identifiers (`$anchor`)#

To reference specific tables or individual variables within a larger schema document, use $anchor. Combining $id with $anchor produces a globally resolvable URI fragment PID:

"properties": {
  "age": {
    "$anchor": "var_age",
    "type": "integer",
    "fair:label": "Respondent Age"
  }
}

The resulting PID for the age variable is https://doi.org/10.5281/zenodo.1234567#var_age.

Benefits of Root Anchors (`$anchor` at Root Level)#

While $id provides the canonical base URI for the document, defining a root $anchor (e.g., "$anchor": "dataset_demographics_2024") provides key advantages:

Decoupling Opaque PIDs (DOIs) from Readable Slugs: PIDs like DOIs (https://doi.org/10.5281/zenodo.1234567) are numeric and opaque. A root $anchor creates a human-readable alias URI (https://doi.org/10.5281/zenodo.1234567#dataset_demographics_2024), making $ref statements in external schemas self-documenting.
Bundled Catalogs & Multi-Schema Documents: When multiple dataset schemas are packaged into a single master file (e.g., inside $defs or an API catalog payload), sub-schemas share the master file’s $id. The root $anchor allows tools to target and pull out that specific dataset schema from the bundle via $ref.
Offline & Local Registry Resolution: If schemas are resolved locally (where https://doi.org/... cannot be fetched over the network), internal references using #dataset_demographics_2024 remain valid within the local schema scope regardless of base URI rewriting.
Uniform Identifier Hierarchy: Establishes a consistent naming pattern across the entire data product hierarchy:
- Dataset Root: https://doi.org/10.5281/zenodo.1234567#dataset_demographics_2024
- Table / Resource: https://doi.org/10.5281/zenodo.1234567#table_respondents
- Variable / Property: https://doi.org/10.5281/zenodo.1234567#var_age

2. Resolvable PIDs vs. Random GUIDs / UUIDs#

Data stewards and software systems deal with two main classes of identifiers:

Identifier Category	Examples	Resolution Mechanism	Primary Use Case
Resolvable PIDs	DOI, w3id, DID, ARK, Handle	Direct HTTP GET / DID resolver	Canonical dataset identification, publishing, external citation
Random / Synthetic GUIDs	UUID v4 (`urn:uuid:...`)	Non-resolvable (offline hash)	Local database keys, transaction logs, immutable payload snapshots

Resolvable PIDs#

Resolvable PIDs provide active resolution services that redirect human users to dataset landing pages and machine agents (LLMs, FAIR tools, MCP servers) to machine-readable JSON Schema payloads.

Random Identifiers (GUIDs / UUIDs)#

Random synthetic identifiers like RFC 4122 UUID v4 (urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6) guarantee mathematical uniqueness (global scope) without relying on a centralized domain authority. They are ideal for tagging offline data extracts, local primary keys, or runtime message payloads.

3. Secondary Identifiers (`fair:identifiers`)#

Datasets and variables often carry legacy accession numbers, institutional database keys, or project-level grant numbers. The fair:identifiers annotation keyword follows progressive disclosure: start with simple string shorthands and add richer metadata only when needed.

🟢 MVP Shorthand: Simple String List (Option A)#

For 90% of everyday datasets, simply list your identifier strings:

"fair:identifiers": [
  "10.5281/zenodo.1234567",
  "ICPSR-38492",
  "urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
]

🟢 MVP Shorthand: Clean 2-Key Objects (Option B)#

Add light category or scope context using clean key aliases (type and scope):

"fair:identifiers": [
  { "identifier": "10.5281/zenodo.1234567", "type": "DOI" },
  { "identifier": "ICPSR-38492", "type": "Accession", "scope": "institutional" },
  { "identifier": "GRANT-2024-EU-8842", "type": "Custom", "scope": "project" }
]

🔵 Advanced Stewardship: Extended Objects (Optional)#

When institutional data stewards require active resolution links, formal scheme references, or historical notes, expand to full objects:

"fair:identifiers": [
  {
    "identifier": "10.5281/zenodo.1234567",
    "identifierType": "DOI",
    "identifierTypeRef": "https://highvaluedata.net/fair-data-schema/cv/identifier-types-v1#DOI",
    "uniquenessScope": "global",
    "scheme": "DataCite",
    "isResolvable": true,
    "resolverUrl": "https://doi.org/",
    "description": "Canonical DOI published on Zenodo."
  }
]

Uniqueness Scopes#

The uniquenessScope property specifies the administrative or spatial scope within which the identifier is guaranteed unique:

global: Globally unique across all systems worldwide (DOIs, w3ids, DIDs, UUIDs).
regional: Unique within a supranational geopolitical region (e.g. Eurostat series keys, EU NUTS codes).
national: Unique within a national statistical agency or country (e.g. US Census ID, INSEE commune code).
subnational: Unique within a subnational jurisdiction, state, province, canton, city, municipality, or district (e.g. US State FIPS code, city parcel ID).
institutional: Unique within a specific university, data bank, or repository (e.g. ICPSR study number, Dataverse accession).
project: Unique within a research project or grant (e.g. PROJ-2024-VAR01).
local: Unique only within a single table, database schema, or runtime instance (e.g. primary key integer).
other: Custom or unlisted administrative uniqueness scope.

Controlled vocabulary reference: Identifier Scopes CV. Controlled vocabulary reference: Identifier Types CV.

Primer on Identifiers#

Understanding the core properties, trade-offs, and ecosystem tooling of digital identifiers helps data stewards and software engineers choose the right identifier strategy for datasets, tables, and variables.

📌 1. Persistent Identifiers (PIDs)#

Concept: A long-lasting, stable digital reference to a resource that remains constant even if the hosting institution, web server URL, or backend database changes over time.
How it Works: PIDs rely on institutional commitment and central resolution infrastructure (e.g. DataCite, Crossref, ORCID, ROR) to update target URLs behind the scenes without breaking citations.
Key Standards & Schemes: Digital Object Identifiers (DOI), Archival Resource Keys (ARK), Handles (Handle), Permanent Web URLs (w3id).
Primary Benefits: Ensures long-term research reproducibility, FAIR Findability (F1), and citation integrity across decades.
Drawbacks & Challenges: Some persistent identifiers may involve registration and fees (e.g. formal DataCite or Crossref memberships). All PIDs require ongoing commitment to keep target redirection URLs updated.
Common Tools & Services:
- Free / Open Access / Self-Hosted: Zenodo, Figshare, Harvard Dataverse, ORCID, ROR Registry, w3id.org, N2T.net (ARK resolver), self-hosted PURLs/ARKs.
- Fee-Based / Institutional Membership Registrars: DataCite, Crossref, EZID (requires institutional subscription/account for direct minting).

🌐 2. Resolvable Identifiers#

Concept: An identifier that can be actively dereferenced over a network protocol (HTTP/HTTPS, DNS, or DID resolution protocol) to retrieve metadata, documentation, or payload data.
How it Works: When a human browser or machine agent (LLM, MCP server, API tool) sends an HTTP GET request (or content negotiation header Accept: application/schema+json), the resolver returns machine-actionable metadata.
Key Standards & Schemes: Web PIDs (https://doi.org/..., https://w3id.org/...), HTTP URIs, resolvable DIDs (did:web:...).
Primary Benefits: Enables automated machine-to-machine discovery, API tool calling, and programmatic FAIR data access without human intervention.
Drawbacks & Challenges: Subject to network outages, resolver server downtime, or link rot if web domain registrations expire or HTTP redirection rules break.
Common Tools & Services: w3id.org (community redirector), Handle.net, DOI.org, purl.org, cURL / HTTP client libraries with content negotiation support (httpx, fetch).

🔐 3. Decentralized Identifiers (DIDs)#

Concept: A W3C standard for globally unique, self-sovereign digital identifiers that operate without relying on a centralized registration authority, DNS provider, or single database operator.
How it Works: DIDs use public-key cryptography, distributed ledgers, or peer-to-peer networks to cryptographically verify data authenticity, origin, and ownership.
Key Standards & Schemes: W3C DID Specification (did:key:..., did:ion:..., did:web:...).
Primary Benefits: Enables tamper-proof, cryptographically verifiable data products across decentralized ecosystems and multi-party research networks.
Drawbacks & Challenges: Higher technical complexity, evolving standards across different DID methods (did:key vs did:web vs did:ion), and potential blockchain gas fees or indexer latency depending on the underlying DID method.
Common Tools & Services: W3C Universal Resolver (resolver.identity.foundation), Veramo Framework, ION / Sidetree, Spruce ID, did-cli, DIF (Decentralized Identity Foundation) SDKs.

🎲 4. Random / Synthetic Identifiers (GUIDs & UUIDs)#

Concept: Mathematically generated, non-semantic strings created algorithmically without central coordination, designed to guarantee collision prevention.
How it Works: Uses 128-bit pseudo-random generators (e.g. RFC 4122 UUID v4) to ensure that the probability of generating identical strings across different systems worldwide is effectively zero.
Key Standards & Schemes: Universally Unique Identifiers (UUID v4), Globally Unique Identifiers (GUID).
Primary Benefits: Fast, offline generation for internal primary keys, payload hashes, or message transaction logs without needing web domain setup or registrar fees.
Drawbacks & Challenges: Typically non-resolvable over HTTP natively—a raw UUID (f81d4fae-7dec-11d0-a765-00a0c91e6bf6) cannot be pasted into a browser to view metadata unless wrapped in a resolver URL. Opaque and non-semantic (contains no human title or domain context).
Common Tools & Services: uuidgen CLI tool, Python uuid module, JavaScript crypto.randomUUID(), PostgreSQL gen_random_uuid(), Rust uuid crate.

CDIF Metadata Publication, Signposting & RDA PID Kernel Alignment#

The Cross-Domain Interoperability Framework (CDIF) Handbook on Publishing Metadata defines mechanisms for making digital objects discoverable by search engines, machine crawlers, and autonomous agents across domains.

🌐 1. Signposting & FAIR Digital Object (FDO) Discovery (IETF RFC 8288)#

CDIF recommends Signposting—using typed HTTP web links (IETF RFC 8288) and IANA-registered link relation types—to allow software agents to discover a resource’s metadata when resolving its PID URI ($id).

When hosting or serving FAIR Data JSON Schemas over HTTP, web servers and API endpoints should return typed Signposting link headers:

HTTP/1.1 200 OK
Content-Type: application/schema+json
Link: <https://doi.org/10.5281/zenodo.1234567>; rel="cite-as",
      <https://highvaluedata.net/fair-data-schema/dev/schema.json>; rel="describedby"; type="application/schema+json",
      <https://spdx.org/licenses/CC-BY-4.0>; rel="license",
      <https://ror.org/007qwym43>; rel="author"

In human-readable landing pages, Signposting links can also be embedded directly in HTML <head> tags:

<link rel="cite-as" href="https://doi.org/10.5281/zenodo.1234567" />
<link rel="describedby" href="https://example.org/dataset-schema.json" type="application/schema+json" />
<link rel="license" href="https://spdx.org/licenses/CC-BY-4.0" />

📋 2. RDA PID Kernel & FAIR Digital Object (FDOF) Attribute Mapping#

The Research Data Alliance (RDA) Recommendation on PID Kernel Information and the FAIR Digital Object Framework (FDOF) specify a minimal metadata record returned when resolving a PID. The table below illustrates how FAIR Data JSON Schema keywords map to RDA PID Kernel attributes and CDIF schema.org recommendations:

RDA PID Kernel Attribute	CDIF schema.org Element	FAIR Data JSON Schema Keyword	Mapping & Implementation Notes
FDO Canonical PID	`@id`	`$id`	Primary web-resolvable PID / base URI (`DOI`, `w3id`, `ARK`, `DID`).
FDO Secondary PIDs	`identifier`	`fair:identifiers`	List of secondary, institutional accessions, or legacy database keys.
FDO Resource Type	`@type`	`fair:resourceType`	Object role (`dataset`, `data-product`, `variable`).
FDO Creator / Agent	`creator`	`fair:contributors` (`role="Creator"`)	Agent responsible for creating the object / registering the PID.
FDO Responsible Org	`provider` / `publisher`	`fair:contributors` (`role="Provider"`)	Responsible organization holding custody (supports ROR URIs).
Rights & License	`license`	`fair:licenseRef` / `fair:license`	SPDX license URI or license terms string.
Bit Checksum / Integrity	`spdx:checksum` / `etag`	`fair:checksum` (or via `fair:version`)	Hash digest (e.g. SHA-256) verifying payload bit-sequence integrity.
Persistency Policy	`publishingPrinciples`	`fair:version` / `fair:description`	Maintenance intentions and archiving policies.