Missing Values & Clamped Data (Sentinels & Censoring)

Missing Values & Clamped Data (Sentinels & Censoring)#

This recipe explains how to handle user-defined missing values (sentinels, refusal codes, error intervals) and clamped data (top-coded and bottom-coded privacy thresholds) in FAIR Data JSON Schema.

See the companion schema file: ../../../examples/missing-and-clamped-values.json

How-to: Implement Missing & Clamped Values

  1. Use oneOf: Partition the property domain into distinct subschemas representing valid values, boundary brackets, and missing codes.

  2. Flag Missing Codes with fair:sentinel: true: Apply to non-substantive codes (const: 99, const: -1) or error intervals (minimum: -99, maximum: -90) to instruct parsers to mask them as NaN.

  3. Flag Boundary Brackets with fair:clamped: true: Apply to substantive measurements that have been capped for confidentiality (e.g. Age 99+ or Hours <= 5).

  4. Rely on Standard Range Keywords: Use minimum for top-coding (upper clamp) and maximum for bottom-coding (lower clamp)—no custom condition strings needed.

  5. Prevent Overlaps: Use negative numbers (e.g. -1) for sentinels to allow open-ended top-coding without mutual exclusivity collisions.

1. The Core Dilemma: Sentinels vs. Clamped Observations#

In microdata, censuses, administrative registers, and health databases, variables frequently contain special boundary numbers. In standard JSON Schema, declaring "type": "integer" allows any integer through validation, but downstream software and AI agents must know how to treat these numbers:

Dimension

fair:sentinel (Missing / Refused / Error)

fair:clamped (Top-Coded / Bottom-Coded)

Data Nature

Non-substantive: Refusal, don’t know, processing error, system null

Substantive observation: Real measurement bracketed for confidentiality or outlier control

Statistical Action

Mask as NaN / None; exclude from means, sums, standard deviations, correlations

Retain in sample; model as censored observations (e.g. Tobit regression, survival models)

Typical Origins

Survey refusal codes (99, -1), negative error ranges (-99..-90), SPSS user-missing

Census anonymization (Age: 99+, Age: 5 and under, Income: $250k+)

AI / Agent Behavior

LLM recognizes non-response without calculating averages over codes like 999

LLM knows observations at boundary represent bracketed populations ($\ge 99$ or $\le 5$)

2. Terminology Equivalence Across Domains, Sectors, and Audiences#

Different disciplines, sectors, and software ecosystems use completely different vocabularies to describe the exact same underlying mathematical and data phenomena. FAIR Data JSON Schema bridges these worlds with a unified, cross-domain syntax:

Audience / Sector

Missing & Non-Response

Top-Coded / Upper Boundary

Bottom-Coded / Lower Boundary

Bounded Interval

Official Statistics & Census

User-missing codes, Refusal / Non-response, Suppressed codes

Top-coding (Upper-coding, disclosure avoidance)

Bottom-coding (Lower-coding, disclosure avoidance)

Classification bracket, uncompressed range

Econometrics & Biostatistics

Non-response (MCAR, MAR, MNAR)

Right-censoring (Tobit regression, survival analysis)

Left-censoring (Limit of Detection / LoD in assays)

Interval-censoring

Computer Science & Software

Sentinel values, in-band error signaling (-1, null)

Upper clamp, ceiling cap

Lower clamp, floor cap

Clamping (std::clamp(x, min, max))

Data Engineering & ML

Null masking (pd.NA, np.nan), indicator vectors

Outlier Winsorization, upper feature clipping (clip(max))

Lower Winsorization, lower feature clipping (clip(min))

Feature bounds, min-max scaling box

AI Agents & LLMs (MCP)

Masked non-evaluable sentinels (excluded from summaries)

Bounded parameter threshold (upper)

Bounded parameter threshold (lower)

Valid continuous tool argument domain

Metadata Standards (DDI-CDI)

SentinelValueDomain / InvalidValueRepresentation

MeasurementDomain with right-censoring rule

MeasurementDomain with left-censoring rule

Bounded continuous / interval domain

FAIR Data JSON Schema

"fair:sentinel": true

"fair:clamped": true + minimum

"fair:clamped": true + maximum

"minimum" + "maximum"

Understanding the Disciplinary Perspectives#

  1. Statistical Disclosure Control (SDC) & Census Stewardship:

    • Statistical agencies (e.g. US Census Bureau, Eurostat, Statistics Canada, IPUMS) routinely top-code and bottom-code microdata to protect respondent confidentiality.

    • For example, high incomes are top-coded at $$250,000+$ and older ages at $99+$ to prevent outlier identification.

    • Knowing an observation is top-coded is vital: running ordinary least squares (OLS) regression on top-coded income yields biased estimates unless modeled appropriately.

  2. Econometrics, Biostatistics, & Clinical Trials:

    • Econometricians and clinical researchers refer to top-coding as right-censoring and bottom-coding as left-censoring (e.g. assay Limit of Detection, where viral load $< 20$ copies/mL).

    • In statistical modeling, censored observations are modeled using Tobit regression, Kaplan-Meier estimators, or Cox proportional hazards models rather than being treated as literal point measurements.

  3. Computer Science & Data Engineering:

    • Software engineers use the universal term clamping (std::clamp in C++, clamp() in CSS and Rust) to restrict an expression between upper and lower thresholds while preserving substantive validity.

    • Machine learning pipelines perform Winsorization or feature clipping (torch.clamp(), numpy.clip()) to prevent extreme values from distorting model weights.

  4. Why fair:clamped Was Selected:

    • It is universally recognized across software engineering and numerical computing.

    • It avoids calling continuous physical measurements or ages a “code” (which falsely implies categorical classification).

    • It leaves the direction to be inferred naturally from the schema keyword (minimum for upper clamp, maximum for lower clamp).

3. Standard JSON Schema Validation Mechanics#

Instead of inventing custom condition strings (such as ">=" or "<=") that break standard JSON Schema validators, FAIR Data JSON Schema relies strictly on native Draft 2020-12 range keywords:

  • Discrete code: "const": 99 or "const": -1

  • Closed interval $[6, 98]$: "minimum": 6, "maximum": 98

  • Upper boundary $[99, \infty)$: "minimum": 99

  • Lower boundary $(-\infty, 5]$: "maximum": 5

Direction is 100% Inferred#

When paired with "fair:clamped": true:

  • "minimum": 99 is inherently an upper clamp (top-coded / right-censored).

  • "maximum": 5 is inherently a lower clamp (bottom-coded / left-censored).

No redundant directional flags (floor/ceiling or top/bottom) are required. Data types declared on the parent property (e.g. type: "integer") are inherited by oneOf branches automatically.

4. Universal Cross-Ecosystem Mapping#

The distinction between missing sentinels and censored boundaries is a universal challenge across statistical software, programming ecosystems, metadata standards, and AI systems:

Environment

Sentinel / Missing Value Mechanism

Clamped / Censored Mechanism

Equivalent in FAIR Data JSON Schema

SPSS (.sav)

n_missing = 1..3 (discrete codes)
n_missing = -2 (closed range $[low, high]$)
n_missing = -3 (range + discrete)

Handled via recoding or value labels (e.g. 99 "99 and older")

fair:sentinel: true on const or minimum/maximum branches

Stata

Extended missing values (.a, .b, …, .z, .)

Censored regression models (tobit, cnreg)

fair:sentinel: true for missing codes; fair:clamped: true for bounds

SAS

Special missing values (.A through .Z, ._)

Model statements in PROC LIFEREG, PROC QLIM

fair:sentinel: true for special missing; fair:clamped: true for thresholds

Python (pandas)

read_csv(na_values=[...])
df.mask() / df.replace()

Censored modeling in statsmodels, clipping via pandas.clip()

Inspect schema: mask fair:sentinel to NaN; retain fair:clamped

R (haven / labelled)

labelled_spss(..., na_values=..., na_range=...)
dplyr::na_if()

Censored regression in survival::Surv(), censReg

Inspect schema: map fair:sentinel to user-missing or NA

DDI-CDI / DDI

SentinelValueDomain / InvalidValueRepresentation

MeasurementDomain with censoring attributes

Direct 1:1 conceptual mapping

AI Agents (MCP / OpenAPI)

Agent parses fair:sentinel to avoid computing averages over error codes

Agent recognizes capped distributions for tool queries

Machine-actionable metadata directly in OpenAPI / JSON Schema

5. Practical Implementation Patterns#

Pattern A: Categorical Non-Response (fair:sentinel with const)#

Standard survey questions often contain discrete substantive responses alongside refusal or “don’t know” codes:

{
  "type": "integer",
  "title": "Employment Status",
  "oneOf": [
    { "const": 1, "title": "Employed full-time" },
    { "const": 2, "title": "Employed part-time" },
    { "const": 3, "title": "Unemployed seeking work" },
    { "const": 4, "title": "Retired" },
    { "const": 98, "title": "Don't know", "fair:sentinel": true },
    { "const": 99, "title": "Refused / No answer", "fair:sentinel": true }
  ]
}

Pattern B: The Demographic Age Bracket (5–99)#

In census and survey microdata, age is commonly bottom-coded for early childhood and top-coded for older adults to prevent respondent identification:

{
  "type": "integer",
  "title": "Age of Respondent",
  "oneOf": [
    { "minimum": 0, "maximum": 5, "title": "5 years and under (bottom-coded for confidentiality)", "fair:clamped": true },
    { "minimum": 6, "maximum": 98, "title": "Recorded age in single years (6 to 98)" },
    { "minimum": 99, "title": "99 years and older (top-coded for confidentiality)", "fair:clamped": true },
    { "const": -1, "title": "Refused / Not reported", "fair:sentinel": true }
  ]
}

Why Negative Sentinels & Bounded Zero Floors Are Recommended

Bounding the bottom-coded bracket with minimum: 0 ensures valid toddler ages (0 to 5) are captured while preventing negative sentinel codes like -1 from satisfying maximum: 5. Because all valid substantive ages are $\ge 0$, the top-coded threshold minimum: 99 can remain completely open-ended without matching missing codes or needing an artificial upper ceiling.

Pattern C: Top-Coded Earnings & Negative Processing Range#

Capturing continuous monetary amounts with a high-earner privacy cap ($250,000+) alongside closed machine error codes:

{
  "type": "number",
  "title": "Annual Gross Income (Thousands USD)",
  "oneOf": [
    { "minimum": 0, "maximum": 249, "title": "Recorded earnings under $250k" },
    { "minimum": 250, "title": "$250,000 and above (top-coded for privacy)", "fair:clamped": true },
    { "minimum": -99, "maximum": -90, "title": "Negative processing error codes (-99 to -90)", "fair:sentinel": true },
    { "const": -1, "title": "Refused / Not reported", "fair:sentinel": true }
  ]
}

Pattern D: Closed Suppressed Non-Response Range#

When surveys flag suppressed or invalid observations with a designated positive range (common in legacy SPSS files with n_missing = -2):

{
  "type": "integer",
  "title": "Weekly Hours Worked",
  "oneOf": [
    { "minimum": 0, "maximum": 5, "title": "5 hours or fewer (bottom-coded)", "fair:clamped": true },
    { "minimum": 6, "maximum": 89, "title": "Recorded hours worked (6 to 89)" },
    { "minimum": 90, "maximum": 99, "title": "Suppressed non-response range (90 to 99)", "fair:sentinel": true }
  ]
}

6. Technical Validation & Mutual Exclusivity in oneOf#

Standard JSON Schema validators (ajv, jsonschema) require that each valid instance matches exactly one branch in a oneOf array:

Value: 92  --> Matches: [6, 98]               --> VALID (1 match)
Value: 99  --> Matches: [99, inf) (clamped)   --> VALID (1 match)
Value: -1  --> Matches: const: -1 (sentinel)  --> VALID (1 match)
Value: 4   --> Matches: [0, 5] (clamped)      --> VALID (1 match)

Rule for Positive Sentinel Codes

If your dataset uses positive sentinel codes higher than a top-code threshold (e.g. 999: Refused alongside an Age 99+ top-code), you must bound the top-code branch (e.g. minimum: 99, maximum: 120). Otherwise, 999 would satisfy both minimum: 99 and const: 999, triggering a oneOf multi-match validation error.

7. Automated Pipeline Ingest Example#

Here is a complete Python snippet demonstrating how an automated ETL script or AI agent uses the schema annotations to prepare tabular data:

import numpy as np
import pandas as pd


def classify_value(val: float, branches: list[dict]) -> tuple[float, str]:
    """Inspects schema oneOf branches to mask sentinels and preserve clamped data."""
    for branch in branches:
        matches = True
        if "const" in branch and val != branch["const"]:
            matches = False
        if "minimum" in branch and val < branch["minimum"]:
            matches = False
        if "maximum" in branch and val > branch["maximum"]:
            matches = False

        if matches:
            if branch.get("fair:sentinel"):
                # Mask non-substantive codes as NaN
                return np.nan, "missing"
            elif branch.get("fair:clamped"):
                # Retain substantive measurement, tag censoring direction
                direction = "top" if "minimum" in branch else "bottom"
                return val, f"clamped_{direction}"
            else:
                return val, "substantive"

    return val, "unmatched"


# Example: Respondent ages evaluated against Pattern B
raw_ages = [4, 25, 72, 99, 104, -1]
age_branches = [
    {"minimum": 0, "maximum": 5, "fair:clamped": True},
    {"minimum": 6, "maximum": 98},
    {"minimum": 99, "fair:clamped": True},
    {"const": -1, "fair:sentinel": True},
]

results = [classify_value(age, age_branches) for age in raw_ages]
df = pd.DataFrame(results, columns=["clean_age", "status"])
print(df)
#    clean_age          status
# 0        4.0  clamped_bottom
# 1       25.0     substantive
# 2       72.0     substantive
# 3       99.0     clamped_top
# 4      104.0     clamped_top
# 5        NaN         missing

Notice how -1 is cleanly converted to NaN, preventing calculation distortions, while 4 and 99 are preserved with their clamping indicators ready for survival analysis or Tobit regression.