Data Model & Contract Document

System: Integration Platform (IIP) Related docs: Architecture · Use Cases · Implementation Plan · Generalization Strategy

This document defines the canonical event contract, its evolution rules, and how each target's storage model derives from it.

What changed, and why it's safe. This document previously declared one fixed canonical schema as the single source of truth for shape. It now declares a registry of schemas sharing a common envelope (AD-11). The canonical model is still the single source of truth — no service invents fields that aren't defined first — but "defined first" now means defined in a contract, which may live in the registry rather than in this file.
The counterintuitive part, stated precisely because it's the whole basis for doing this: the idempotency guarantee never depended on the schema being fixed. ON CONFLICT (record_id) DO NOTHING works because recordId is an envelope field, and the envelope is universal. The same is true of ordering (naturalKey) and of retry/DLQ (which operate on the message, not the payload). Only two things genuinely generalize — the natural-key upsert and the field mapping — and the contract already declares both. Everything that makes IIP trustworthy is envelope-level and comes along for free. The full argument is in 06 §2.2.

1. Canonical Model

The canonical model has two parts: a fixed envelope, identical for every schema and enforced by the Schema Registry, and a per-contract payload, defined in the Contract Registry and validated against it at the source.

┌─ envelope (fixed, universal, crosses every boundary) ──────┐
│  recordId · contractId · recordType · schemaVersion        │
│  naturalKey · occurredAt · traceId                         │
│  ┌─ payload (per contract, opaque in transit) ──────────┐  │
│  │  … whatever the contract declares …                  │  │
│  └──────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────┘

1a. Envelope (fixed — the same for every contract)

Field	Type	Required	Notes
`recordId`	UUID (string)	yes	Unique per event (not per entity). Generated server-side by the Source Service. Primary idempotency key for every adapter, for every contract.
`contractId`	string	yes	Which contract this record conforms to (e.g. `interns`, `forms`). Determines payload validation, topic routing, and which adapter attachments apply.
`recordType`	string	yes	The event type within the contract (e.g. `intern.created`). Must be one of the contract's declared record types.
`schemaVersion`	int	yes	The contract version this payload was validated against — what makes a per-contract compatibility check meaningful (§5).
`naturalKey`	string	yes	Business identifier, derived per the contract's declared key strategy (for `interns`: `internId`). Kafka partition key — this is what guarantees ordering per entity.
`occurredAt`	timestamp (ISO-8601 UTC)	yes	Set server-side at publish time. Replaces the old `createdAt`/`updatedAt` pair — the record type says what happened, so the timestamp only has to say when.
`traceId`	UUID (string)	yes (Release 8+)	Correlates all events belonging to one logical user action across the pipeline for end-to-end log reconstruction.
`payload`	object	yes	The per-contract body. Opaque to everything except the source-service (which validates it) and the adapter (which maps it).

Nothing in that table mentions an intern. That is the point: every service boundary, every idempotency guard, every retry decision, and every DLQ entry operates on these fields alone.

1b. Payload (per contract — defined in the registry)

The payload is whatever the contract declares. Below is the interns contract's payload — contract #1, not "the" schema. It is exactly the field list this document used to present as the canonical record, moved down one level.

Field	Type	Required	Notes
`internId`	string	yes	Business identifier (e.g., `INT001`). Stable across the intern's lifecycle. The contract's declared natural key.
`firstName`	string	yes
`lastName`	string	yes
`email`	string (email format)	yes
`college`	string	yes
`department`	string	yes
`mentor`	string	no
`startDate`	date (`YYYY-MM-DD`)	yes
`status`	enum: `ACTIVE`, `COMPLETED`, `WITHDRAWN`	yes	Extendable via contract evolution (§5).

Example (contractId: interns, recordType: intern.created):

json

{
  "recordId":      "5c1f2e4a-3b7d-4e9a-9c2f-8a6d7e1b0c3f",
  "contractId":    "interns",
  "recordType":    "intern.created",
  "schemaVersion": 1,
  "naturalKey":    "INT001",
  "occurredAt":    "2026-07-21T14:10:00Z",
  "traceId":       "b6e2a9d0-1c4f-4a2e-8e7a-2f9d3c5b1a70",
  "payload": {
    "internId":   "INT001",
    "firstName":  "John",
    "lastName":   "Doe",
    "email":      "john@example.com",
    "college":    "MIT",
    "department": "Data Engineering",
    "mentor":     "Alice",
    "startDate":  "2026-08-01",
    "status":     "ACTIVE"
  }
}

No service other than the Source Service (producer) and the adapters (consumers) ever sees this record in a target-specific form — the canonical shape is what crosses every service boundary.

Migration note (Release 3). Releases 1–2 shipped the flat form: internId, firstName, … at top level, with createdAt and no contractId. Release 3's first step is purely mechanical — wrap the existing fields in payload and add the envelope around them (Phased Rollout 3.1). No field is lost, renamed, or retyped in that step.

1c. Contract Definition

A contract is the data that makes a schema real to the platform. In Release 3 it is a file baked into the image; from Release 4 it is a row in the Contract Registry. Same shape either way — that identity is what keeps Path A a deployment choice rather than a redesign (AD-12).

yaml

contractId: interns
title: Intern Records
schemaVersion: 1
naturalKey:
  strategy: field
  fields: [internId]            # composite keys join with '|', e.g. [formId, questionId]
recordTypes:
  - intern.created
  - intern.updated              # update-style: upsert on naturalKey
  - intern.deleted              # tombstone
fields:
  - { name: internId,   type: string, required: true,  queryable: true }
  - { name: firstName,  type: string, required: true }
  - { name: lastName,   type: string, required: true }
  - { name: email,      type: email,  required: true }
  - { name: college,    type: string, required: true }
  - { name: department, type: string, required: true }
  - { name: mentor,     type: string, required: false }
  - { name: startDate,  type: date,   required: true }
  - { name: status,     type: enum,   required: true, values: [ACTIVE, COMPLETED, WITHDRAWN], default: ACTIVE, queryable: true }

default is applied when a field is absent or blank in the submission, before the required check — so a field can be both required and omissible by clients, which is how a server-derived value stays server-derived. It exists because of a concrete case: Release 1's CanonicalMapper hardcoded "a new intern always starts ACTIVE" in Java, and the UI has never sent status. Phase 3.5 deletes that mapper, so without default the choice would have been to make status optional (weakening the contract) or to make every client send it (changing the HTTP intake shape). Declaring it here keeps Release 1's behaviour exactly, with no Java that knows what an intern is. A default must fall inside the field's own domain — default: PENDING against the enum above is a startup error, not a runtime surprise.

What moved out of code and into this file: field names, types, required-ness, server-side defaults; the natural-key strategy; the enum domain for status; the target mapping (in the attachment, below). What stayed in code: recordId/occurredAt generation, envelope construction and publish, retry/DLQ/idempotency machinery, offset-commit discipline — all envelope-level, all universal.

Registry schema (Release 4) — the same definition, persisted:

sql

CREATE TABLE contracts (
  contract_id     TEXT PRIMARY KEY,
  title           TEXT NOT NULL,
  schema_version  INT  NOT NULL,
  definition      JSONB NOT NULL,        -- the full contract (fields, key strategy, record types)
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE adapter_attachments (
  attachment_id   UUID PRIMARY KEY,
  contract_id     TEXT NOT NULL REFERENCES contracts(contract_id),
  adapter_type    TEXT NOT NULL,          -- 'postgres' | 'csv' | 'webhook'
  config          JSONB NOT NULL,         -- target-specific: table, endpoint, auth, mapping
  enabled         BOOLEAN NOT NULL DEFAULT true,
  UNIQUE (contract_id, adapter_type, config)
);

"Define a schema" (UC-13) is an insert into contracts. "Attach an adapter" (UC-14) is an insert into adapter_attachments. Both are runtime registry writes the control-plane UI performs — no build, no redeploy.

2. Record Types (Lifecycle)

Record types are declared per contract, not fixed by the platform. What the platform fixes is the three classes of record type an adapter knows how to handle — that classification is what lets one generic write path serve every contract.

Class	Topic	Introduced	Semantics
Create	`{contractId}.created`	Release 1	A new record. Idempotency guard: `ON CONFLICT (record_id) DO NOTHING`.
Update-style	`{contractId}.updated`	Release 7	Carries the full current state of the record (not a diff) — simplest for consumers to apply, avoids reconstructing state from partial deltas. Upserts on the contract's declared natural key.
Tombstone	`{contractId}.deleted`	Release 7	Minimal payload: the envelope plus enough to identify the record. No need to carry full record data for a deletion.

A contract declares which of its record types fall into which class. For interns: intern.created (create), intern.updated (update-style), intern.deleted (tombstone). A contract may declare create-only — an append-only event log with no update path is a legitimate, and simpler, contract.

This is the one guarantee that genuinely generalized. The old rule "intern.updated → ON CONFLICT (intern_id) DO UPDATE" was never really about interns; it was "upsert on the declared natural key," with exactly one declaration in existence. Generalizing it required no new mechanism, only a place to put the declaration.

Above: the interns contract's lifecycle as a worked example. The state names come from that contract's status enum — the platform has no opinion about them, and another contract's diagram would look entirely different while using the same three classes.

3. DLQ Message Envelope

Messages routed to iip.dlq wrap the original payload rather than replacing it, so nothing is lost and the failure is fully diagnosable. One DLQ serves every contract; contractId is what makes that a filter rather than a mess.

Field	Type	Description
`originalTopic`	string	Topic the message originated from (e.g., `interns.created`).
`originalPartition`	int	Partition of the original message.
`originalOffset`	long	Offset of the original message, for correlation with broker logs.
`originalKey`	string	The original `naturalKey`.
`contractId`	string	Which contract the failed record belongs to — the primary grouping dimension for DLQ triage (UC-11). Read off the envelope, so it survives even when the payload is what's broken.
`originalPayload`	bytes / JSON	The unmodified original message, envelope included (preserved exactly).
`errorType`	string	Classification, e.g. `SCHEMA_VIOLATION`, `UNMAPPABLE_DATA`, `RETRY_EXHAUSTED`, `UNCLASSIFIED_FAILURE` (an exception type `FailureClassifier` doesn't recognize -- non-retriable by default, but distinct from `RETRY_EXHAUSTED` since it was never actually retried; see Phase 2.4).
`errorMessage`	string	Human-readable error summary.
`failedAdapter`	string	Which consumer group/adapter produced this DLQ entry.
`attemptCount`	int	How many processing attempts were made before quarantine.
`quarantinedAt`	timestamp	When it was routed to the DLQ.
`replayed`	boolean	Set `true` by the DLQ Replay Tool (Release 8) once reprocessed — entry is kept for audit, not deleted.
`replayedAt`	timestamp, nullable	When replay occurred.

4. Target Storage Models

4.0 The generic landing table (Release 5 — the `postgres` adapter's default write)

The one genuinely new design question generalization raises: how does an arbitrary payload reach Postgres without a redeploy? Three answers were considered.

Option	Verdict	Why
Pure JSONB — one table, `payload jsonb`	rejected	Zero DDL ever, fully runtime — but loses typed columns everywhere, including where a guarantee depends on them.
DDL-per-contract — `CREATE TABLE` on registration	rejected	Real types, real SQL, but the service must hold DDL grants and own schema drift and `ALTER TABLE` migrations. Too much operational surface for a solo build.
Hybrid — typed exactly where a guarantee needs it, JSONB for the rest	chosen	Runtime-native like pure JSONB, but the two columns that must be typed and indexed — the idempotency key and the natural key — are.

sql

CREATE TABLE records (
  record_id    UUID PRIMARY KEY,          -- envelope; idempotency guard (universal)
  contract_id  TEXT NOT NULL,             -- envelope
  record_type  TEXT NOT NULL,             -- envelope
  natural_key  TEXT NOT NULL,             -- envelope; typed + indexed for per-contract upsert
  payload      JSONB NOT NULL,            -- per-contract, opaque to the adapter
  occurred_at  TIMESTAMPTZ NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (contract_id, natural_key)       -- conflict target for update-style events
);

-- Fields a contract marks "queryable" become expression indexes over payload, added per contract:
-- CREATE INDEX ON records ((payload->>'status')) WHERE contract_id = 'interns';

This is the same philosophy the rest of the platform already lives by: generic in transport, specific exactly where a guarantee needs it (Architecture principle 10). record_id is typed because ON CONFLICT (record_id) DO NOTHING is the idempotency guarantee; natural_key is typed and unique-per-contract because ON CONFLICT (contract_id, natural_key) DO UPDATE is the ordering-and-upsert guarantee. Everything else is payload, and the adapter has no opinion about it.

For some schemas JSONB is not a compromise but the better fit: a forms contract with 150 questions and variable-length option arrays flattens into columns badly, and stores as JSONB naturally.

Shaped-table mode. A contract that wants a dedicated, fully-typed table can declare one — the postgres adapter type supports it as per-attachment config, with the mapping in adapter_attachments.config. The records table is the zero-config default, not a mandate. §4.1 below is exactly that mode, as built for interns in Release 1.

4.1 PostgreSQL, shaped-table mode (`interns` — as built, Release 1)

record_id carries a UNIQUE constraint — this is what makes ON CONFLICT (record_id) DO NOTHING a correct idempotency guard for create-event redelivery. Identical in purpose to records.record_id above; the generic table did not weaken it.
intern_id also carries a UNIQUE constraint (one current-state row per intern) — the conflict target for intern.updated. This is the shaped-mode instance of the generic UNIQUE (contract_id, natural_key): with one contract per table, the contract_id half is implied.
INTERN_AUDIT_LOG (Release 7) is append-only by design — it's the audit trail, so it should not be upserted; every event (created/updated/deleted) adds a row here regardless of what happens to the INTERNS row.

4.2 CSV File (`csv` adapter)

Columns (Release 1, create-only, interns contract):

csv

record_id,intern_id,first_name,last_name,email,college,department,mentor,start_date,status,created_at

From Release 6 the column list is not compiled in: it comes from the adapter attachment's config, defaulting to the contract's declared field order with the envelope's record_id prefixed. A second contract attached to the csv type writes its own file with its own columns, from the same adapter instance.

Dedup index: a local key-value store mapping recordId -> written (bool), consulted before every append and updated after every successful write. This is what makes the naturally non-idempotent "append a line" operation idempotent in effect. As built it is a flat file — one recordId per line — because the adapter is single-writer (AD-6) and there is no concurrent-writer race an embedded database would be needed for.

Release 7 shift — snapshot rebuild instead of pure append: once update-style and tombstone record types exist, the file adapter maintains its keyed store as current state per naturalKey (not just a seen-set), and on every event rebuilds the contract's CSV from that current state:

This resolves the append-only-vs-mutable tension named in the original spec (§7) by treating the CSV as a projection of current state, not a raw event log — option 2 from the original design brief, adopted platform-wide rather than deferred.

5. Schema Evolution Rules

Evolution now happens at two levels, governed by the same principle at both: a change is compatible if already-deployed consumers keep working.

5.1 Envelope evolution (Schema Registry, platform-wide)

Compatibility mode: BACKWARD (default) — a new schema can read data written with the previous schema. This is the correct default because adapters are deployed independently (principle: independent deployability) and must not break when the Source Service ships a change first. Envelope changes are rare and affect every contract at once, so the bar is deliberately high.

5.2 Contract evolution (Contract Registry, per `contractId`)

The same BACKWARD rule is enforced per contract. A contract carries a schemaVersion; editing a contract is a compatibility-checked registry update, not an overwrite, and the version is stamped onto every envelope so a consumer can always tell which definition a payload was validated against.

The rules below apply identically at both levels — read "field" as an envelope field in §5.1 and a payload field in §5.2:

Change	Allowed under BACKWARD compatibility?	Notes
Add a new optional field with a default	Yes	Safe, most common evolution (e.g., adding `traceId`).
Add a new required field	No, unless a default is supplied	Would break consumers built against the old schema reading new data — must supply a default.
Remove a field consumers depend on	No	Breaking; requires a coordinated multi-step migration (deprecate, stop-writing, then remove).
Widen a type (e.g., `int` → `long`)	Generally yes (Avro promotion rules)	Verify against the specific serialization format's promotion table.
Rename a field	No (treated as remove + add)	Use aliasing support if the serialization format provides it, or a two-phase add-new/remove-old rollout.
Add a new enum value (e.g. to `status`)	Yes, if consumers treat unknown values defensively	Consumers should not `switch` exhaustively without a default case.
Change a contract's natural-key strategy	No	Repartitions the contract's entire stream and invalidates every existing `(contract_id, natural_key)` row. Treat as a new contract, not an edit.

What "allowed" requires of a consumer. Every "Yes" in the table above is conditional on deployed consumers tolerating what they do not recognise, and that is not automatic — it is a setting, and the usual default is the wrong one. Phase 4.11 found both IIP adapters rejecting an added optional field outright, because a hand-built Jackson ObjectMapper enables FAIL_ON_UNKNOWN_PROPERTIES by default. The registry, the compatibility gate, and the source service would all have approved the change; the first sign of trouble would have been one DLQ entry per record from a service nobody had touched. The rule to hold to is producer strict, consumer tolerant: the source service rejects an undeclared payload key, because there it is a typo or a client running ahead of the schema and accepting it would hide data loss behind a 202; an adapter ignores one, because there it means the contract has moved ahead of that adapter's mapping, which is the normal state of services that deploy independently. The same asymmetry is why the envelope's JSON Schema leaves additionalProperties open.

Governance process: an envelope change is proposed as a PR against the schema definition and validated by the Schema Registry's compatibility check in CI before merge. A contract change is validated the same way — the control-plane API runs the compatibility check before accepting the write, and CI runs it over every registered contract, so an incompatible contract edit fails at the API boundary rather than in production. An incompatible change fails the build, not production; that rule is now enforced for schemas that no longer live in the repository.

6. Canonical Model → Target Mapping

Each contract declares its own canonical→target mapping. There is no single platform-wide mapping table any more — there is a rule for the envelope (fixed) and a per-attachment mapping for the payload (data). The table below is the interns contract's mapping, kept as the worked example every adapter transform is unit-tested against (see Implementation Plan §5).

Envelope → target (fixed, every contract):

Envelope field	`records` column	Shaped-table equivalent	CSV column
`recordId`	`record_id` (PK)	`record_id` (UNIQUE)	`record_id`
`contractId`	`contract_id`	(implied by the table)	(implied by the file)
`recordType`	`record_type`	(drives which SQL runs)	(drives append vs. rebuild)
`naturalKey`	`natural_key` (UNIQUE w/ `contract_id`)	the table's own key column	the key column
`occurredAt`	`occurred_at`	`created_at` / `updated_at`	`created_at`
`schemaVersion`	(carried in `payload` metadata / logged)	(logged)	(logged)
`traceId`	(logged, not persisted in the row)	(logged)	(logged)

Payload → target, interns contract (per-attachment, data not code):

Payload field	`records` (default mode)	`interns` table (shaped mode)	CSV column
`internId`	`payload->>'internId'` (= `natural_key`)	`intern_id` (UNIQUE)	`intern_id`
`firstName`	`payload->>'firstName'`	`first_name`	`first_name`
`lastName`	`payload->>'lastName'`	`last_name`	`last_name`
`email`	`payload->>'email'`	`email`	`email`
`college`	`payload->>'college'`	`college`	`college`
`department`	`payload->>'department'`	`department`	`department`
`mentor`	`payload->>'mentor'`	`mentor` (nullable)	`mentor` (blank if absent)
`startDate`	`payload->>'startDate'`	`start_date`	`start_date`
`status`	`payload->>'status'` (queryable → expression index)	`status`	`status`

Read the two tables together and the division of labour is visible: the first is identical for every contract the platform will ever carry, and the second is a row in adapter_attachments.config.

Data Model & Contract Document ​

1. Canonical Model ​

1a. Envelope (fixed — the same for every contract) ​

1b. Payload (per contract — defined in the registry) ​

1c. Contract Definition ​

2. Record Types (Lifecycle) ​

3. DLQ Message Envelope ​

4. Target Storage Models ​

4.0 The generic landing table (Release 5 — the postgres adapter's default write) ​

4.1 PostgreSQL, shaped-table mode (interns — as built, Release 1) ​

4.2 CSV File (csv adapter) ​

5. Schema Evolution Rules ​

5.1 Envelope evolution (Schema Registry, platform-wide) ​

5.2 Contract evolution (Contract Registry, per contractId) ​

6. Canonical Model → Target Mapping ​