# Missing Values & Clamped Data (Sentinels & Censoring)

This recipe explains how to handle **user-defined missing values** (sentinels, refusal codes, error intervals) and **clamped data** (top-coded and bottom-coded privacy thresholds) in **FAIR Data JSON Schema**.

See the companion schema file: {download}`../../../examples/missing-and-clamped-values.json`

:::{admonition} How-to: Implement Missing & Clamped Values
:class: tip

1. **Use `oneOf`**: Partition the property domain into distinct subschemas representing valid values, boundary brackets, and missing codes.
2. **Flag Missing Codes with `fair:sentinel: true`**: Apply to non-substantive codes (`const: 99`, `const: -1`) or error intervals (`minimum: -99, maximum: -90`) to instruct parsers to mask them as `NaN`.
3. **Flag Boundary Brackets with `fair:clamped: true`**: Apply to substantive measurements that have been capped for confidentiality (e.g. `Age 99+` or `Hours <= 5`).
4. **Rely on Standard Range Keywords**: Use `minimum` for top-coding (upper clamp) and `maximum` for bottom-coding (lower clamp)—no custom condition strings needed.
5. **Prevent Overlaps**: Use negative numbers (e.g. `-1`) for sentinels to allow open-ended top-coding without mutual exclusivity collisions.
:::

---

## 1. The Core Dilemma: Sentinels vs. Clamped Observations

In microdata, censuses, administrative registers, and health databases, variables frequently contain special boundary numbers. In standard JSON Schema, declaring `"type": "integer"` allows any integer through validation, but downstream software and AI agents must know how to treat these numbers:

| Dimension | `fair:sentinel` (Missing / Refused / Error) | `fair:clamped` (Top-Coded / Bottom-Coded) |
| :--- | :--- | :--- |
| **Data Nature** | **Non-substantive**: Refusal, don't know, processing error, system null | **Substantive observation**: Real measurement bracketed for confidentiality or outlier control |
| **Statistical Action** | **Mask as `NaN` / `None`**; exclude from means, sums, standard deviations, correlations | **Retain in sample**; model as censored observations (e.g. Tobit regression, survival models) |
| **Typical Origins** | Survey refusal codes (`99`, `-1`), negative error ranges (`-99..-90`), SPSS user-missing | Census anonymization (`Age: 99+`, `Age: 5 and under`, `Income: $250k+`) |
| **AI / Agent Behavior** | LLM recognizes non-response without calculating averages over codes like `999` | LLM knows observations at boundary represent bracketed populations ($\ge 99$ or $\le 5$) |

---

## 2. Terminology Equivalence Across Domains, Sectors, and Audiences

Different disciplines, sectors, and software ecosystems use completely different vocabularies to describe the exact same underlying mathematical and data phenomena. FAIR Data JSON Schema bridges these worlds with a unified, cross-domain syntax:

| Audience / Sector | Missing & Non-Response | Top-Coded / Upper Boundary | Bottom-Coded / Lower Boundary | Bounded Interval |
| :--- | :--- | :--- | :--- | :--- |
| **Official Statistics & Census** | User-missing codes, Refusal / Non-response, Suppressed codes | **Top-coding** (Upper-coding, disclosure avoidance) | **Bottom-coding** (Lower-coding, disclosure avoidance) | Classification bracket, uncompressed range |
| **Econometrics & Biostatistics** | Non-response (MCAR, MAR, MNAR) | **Right-censoring** (Tobit regression, survival analysis) | **Left-censoring** (Limit of Detection / LoD in assays) | Interval-censoring |
| **Computer Science & Software** | **Sentinel values**, in-band error signaling (`-1`, `null`) | **Upper clamp**, ceiling cap | **Lower clamp**, floor cap | Clamping (`std::clamp(x, min, max)`) |
| **Data Engineering & ML** | Null masking (`pd.NA`, `np.nan`), indicator vectors | Outlier Winsorization, upper feature clipping (`clip(max)`) | Lower Winsorization, lower feature clipping (`clip(min)`) | Feature bounds, min-max scaling box |
| **AI Agents & LLMs (MCP)** | Masked non-evaluable sentinels (excluded from summaries) | Bounded parameter threshold (upper) | Bounded parameter threshold (lower) | Valid continuous tool argument domain |
| **Metadata Standards (DDI-CDI)** | `SentinelValueDomain` / `InvalidValueRepresentation` | `MeasurementDomain` with right-censoring rule | `MeasurementDomain` with left-censoring rule | Bounded continuous / interval domain |
| **FAIR Data JSON Schema** | **`"fair:sentinel": true`** | **`"fair:clamped": true`** + **`minimum`** | **`"fair:clamped": true`** + **`maximum`** | **`"minimum"`** + **`"maximum"`** |

### Understanding the Disciplinary Perspectives

1. **Statistical Disclosure Control (SDC) & Census Stewardship**:
   - Statistical agencies (e.g. US Census Bureau, Eurostat, Statistics Canada, IPUMS) routinely **top-code** and **bottom-code** microdata to protect respondent confidentiality.
   - For example, high incomes are top-coded at $\$250,000+$ and older ages at $99+$ to prevent outlier identification.
   - Knowing an observation is top-coded is vital: running ordinary least squares (OLS) regression on top-coded income yields biased estimates unless modeled appropriately.

2. **Econometrics, Biostatistics, & Clinical Trials**:
   - Econometricians and clinical researchers refer to top-coding as **right-censoring** and bottom-coding as **left-censoring** (e.g. assay Limit of Detection, where viral load $< 20$ copies/mL).
   - In statistical modeling, censored observations are modeled using **Tobit regression**, **Kaplan-Meier estimators**, or **Cox proportional hazards models** rather than being treated as literal point measurements.

3. **Computer Science & Data Engineering**:
   - Software engineers use the universal term **clamping** (`std::clamp` in C++, `clamp()` in CSS and Rust) to restrict an expression between upper and lower thresholds while preserving substantive validity.
   - Machine learning pipelines perform **Winsorization** or feature clipping (`torch.clamp()`, `numpy.clip()`) to prevent extreme values from distorting model weights.

4. **Why `fair:clamped` Was Selected**:
   - It is universally recognized across software engineering and numerical computing.
   - It avoids calling continuous physical measurements or ages a "code" (which falsely implies categorical classification).
   - It leaves the direction to be inferred naturally from the schema keyword (`minimum` for upper clamp, `maximum` for lower clamp).

---

## 3. Standard JSON Schema Validation Mechanics

Instead of inventing custom condition strings (such as `">="` or `"<="`) that break standard JSON Schema validators, FAIR Data JSON Schema relies strictly on **native Draft 2020-12 range keywords**:

- **Discrete code**: `"const": 99` or `"const": -1`
- **Closed interval $[6, 98]$**: `"minimum": 6, "maximum": 98`
- **Upper boundary $[99, \infty)$**: `"minimum": 99`
- **Lower boundary $(-\infty, 5]$**: `"maximum": 5`

### Direction is 100% Inferred
When paired with `"fair:clamped": true`:
- `"minimum": 99` is **inherently an upper clamp** (top-coded / right-censored).
- `"maximum": 5` is **inherently a lower clamp** (bottom-coded / left-censored).

No redundant directional flags (`floor`/`ceiling` or `top`/`bottom`) are required. Data types declared on the parent property (e.g. `type: "integer"`) are inherited by `oneOf` branches automatically.

---

## 4. Universal Cross-Ecosystem Mapping

The distinction between missing sentinels and censored boundaries is a universal challenge across statistical software, programming ecosystems, metadata standards, and AI systems:

| Environment | Sentinel / Missing Value Mechanism | Clamped / Censored Mechanism | Equivalent in FAIR Data JSON Schema |
| :--- | :--- | :--- | :--- |
| **SPSS (`.sav`)** | `n_missing = 1..3` (discrete codes)<br>`n_missing = -2` (closed range $[low, high]$)<br>`n_missing = -3` (range + discrete) | Handled via recoding or value labels (e.g. `99 "99 and older"`) | `fair:sentinel: true` on `const` or `minimum`/`maximum` branches |
| **Stata** | Extended missing values (`.a`, `.b`, ..., `.z`, `.`) | Censored regression models (`tobit`, `cnreg`) | `fair:sentinel: true` for missing codes; `fair:clamped: true` for bounds |
| **SAS** | Special missing values (`.A` through `.Z`, `._`) | Model statements in `PROC LIFEREG`, `PROC QLIM` | `fair:sentinel: true` for special missing; `fair:clamped: true` for thresholds |
| **Python (`pandas`)** | `read_csv(na_values=[...])`<br>`df.mask()` / `df.replace()` | Censored modeling in `statsmodels`, clipping via `pandas.clip()` | Inspect schema: mask `fair:sentinel` to `NaN`; retain `fair:clamped` |
| **R (`haven` / `labelled`)** | `labelled_spss(..., na_values=..., na_range=...)`<br>`dplyr::na_if()` | Censored regression in `survival::Surv()`, `censReg` | Inspect schema: map `fair:sentinel` to user-missing or `NA` |
| **DDI-CDI / DDI** | `SentinelValueDomain` / `InvalidValueRepresentation` | `MeasurementDomain` with censoring attributes | Direct 1:1 conceptual mapping |
| **AI Agents (MCP / OpenAPI)** | Agent parses `fair:sentinel` to avoid computing averages over error codes | Agent recognizes capped distributions for tool queries | Machine-actionable metadata directly in OpenAPI / JSON Schema |

---

## 5. Practical Implementation Patterns

### Pattern A: Categorical Non-Response (`fair:sentinel` with `const`)
Standard survey questions often contain discrete substantive responses alongside refusal or "don't know" codes:

```json
{
  "type": "integer",
  "title": "Employment Status",
  "oneOf": [
    { "const": 1, "title": "Employed full-time" },
    { "const": 2, "title": "Employed part-time" },
    { "const": 3, "title": "Unemployed seeking work" },
    { "const": 4, "title": "Retired" },
    { "const": 98, "title": "Don't know", "fair:sentinel": true },
    { "const": 99, "title": "Refused / No answer", "fair:sentinel": true }
  ]
}
```

### Pattern B: The Demographic Age Bracket (`5–99`)
In census and survey microdata, age is commonly bottom-coded for early childhood and top-coded for older adults to prevent respondent identification:

```json
{
  "type": "integer",
  "title": "Age of Respondent",
  "oneOf": [
    { "minimum": 0, "maximum": 5, "title": "5 years and under (bottom-coded for confidentiality)", "fair:clamped": true },
    { "minimum": 6, "maximum": 98, "title": "Recorded age in single years (6 to 98)" },
    { "minimum": 99, "title": "99 years and older (top-coded for confidentiality)", "fair:clamped": true },
    { "const": -1, "title": "Refused / Not reported", "fair:sentinel": true }
  ]
}
```

:::{admonition} Why Negative Sentinels & Bounded Zero Floors Are Recommended
:class: tip
Bounding the bottom-coded bracket with `minimum: 0` ensures valid toddler ages (0 to 5) are captured while preventing negative sentinel codes like `-1` from satisfying `maximum: 5`. Because all valid substantive ages are $\ge 0$, the top-coded threshold `minimum: 99` can remain completely open-ended without matching missing codes or needing an artificial upper ceiling.
:::

### Pattern C: Top-Coded Earnings & Negative Processing Range
Capturing continuous monetary amounts with a high-earner privacy cap ($250,000+) alongside closed machine error codes:

```json
{
  "type": "number",
  "title": "Annual Gross Income (Thousands USD)",
  "oneOf": [
    { "minimum": 0, "maximum": 249, "title": "Recorded earnings under $250k" },
    { "minimum": 250, "title": "$250,000 and above (top-coded for privacy)", "fair:clamped": true },
    { "minimum": -99, "maximum": -90, "title": "Negative processing error codes (-99 to -90)", "fair:sentinel": true },
    { "const": -1, "title": "Refused / Not reported", "fair:sentinel": true }
  ]
}
```

### Pattern D: Closed Suppressed Non-Response Range
When surveys flag suppressed or invalid observations with a designated positive range (common in legacy SPSS files with `n_missing = -2`):

```json
{
  "type": "integer",
  "title": "Weekly Hours Worked",
  "oneOf": [
    { "minimum": 0, "maximum": 5, "title": "5 hours or fewer (bottom-coded)", "fair:clamped": true },
    { "minimum": 6, "maximum": 89, "title": "Recorded hours worked (6 to 89)" },
    { "minimum": 90, "maximum": 99, "title": "Suppressed non-response range (90 to 99)", "fair:sentinel": true }
  ]
}
```

---

## 6. Technical Validation & Mutual Exclusivity in `oneOf`

Standard JSON Schema validators (`ajv`, `jsonschema`) require that each valid instance matches **exactly one** branch in a `oneOf` array:

```
Value: 92  --> Matches: [6, 98]               --> VALID (1 match)
Value: 99  --> Matches: [99, inf) (clamped)   --> VALID (1 match)
Value: -1  --> Matches: const: -1 (sentinel)  --> VALID (1 match)
Value: 4   --> Matches: [0, 5] (clamped)      --> VALID (1 match)
```

:::{admonition} Rule for Positive Sentinel Codes
:class: warning
If your dataset uses positive sentinel codes higher than a top-code threshold (e.g. `999: Refused` alongside an `Age 99+` top-code), you must bound the top-code branch (e.g. `minimum: 99, maximum: 120`). Otherwise, `999` would satisfy both `minimum: 99` and `const: 999`, triggering a `oneOf` multi-match validation error.
:::

---

## 7. Automated Pipeline Ingest Example

Here is a complete Python snippet demonstrating how an automated ETL script or AI agent uses the schema annotations to prepare tabular data:

```python
import numpy as np
import pandas as pd


def classify_value(val: float, branches: list[dict]) -> tuple[float, str]:
    """Inspects schema oneOf branches to mask sentinels and preserve clamped data."""
    for branch in branches:
        matches = True
        if "const" in branch and val != branch["const"]:
            matches = False
        if "minimum" in branch and val < branch["minimum"]:
            matches = False
        if "maximum" in branch and val > branch["maximum"]:
            matches = False

        if matches:
            if branch.get("fair:sentinel"):
                # Mask non-substantive codes as NaN
                return np.nan, "missing"
            elif branch.get("fair:clamped"):
                # Retain substantive measurement, tag censoring direction
                direction = "top" if "minimum" in branch else "bottom"
                return val, f"clamped_{direction}"
            else:
                return val, "substantive"

    return val, "unmatched"


# Example: Respondent ages evaluated against Pattern B
raw_ages = [4, 25, 72, 99, 104, -1]
age_branches = [
    {"minimum": 0, "maximum": 5, "fair:clamped": True},
    {"minimum": 6, "maximum": 98},
    {"minimum": 99, "fair:clamped": True},
    {"const": -1, "fair:sentinel": True},
]

results = [classify_value(age, age_branches) for age in raw_ages]
df = pd.DataFrame(results, columns=["clean_age", "status"])
print(df)
#    clean_age          status
# 0        4.0  clamped_bottom
# 1       25.0     substantive
# 2       72.0     substantive
# 3       99.0     clamped_top
# 4      104.0     clamped_top
# 5        NaN         missing
```

Notice how `-1` is cleanly converted to `NaN`, preventing calculation distortions, while `4` and `99` are preserved with their clamping indicators ready for survival analysis or Tobit regression.