Generalization Strategy

System: Intern Integration Platform (IIP) → Integration Platform (same iip- prefix, wider goal) Status: adopted — supersedes the "schema is fixed" assumption in 01–05 Related docs: Architecture · Use Cases · Data Model · Implementation Plan · Phased RolloutCurrent build state: intern pod complete through end of Release 2 (baseline pipeline + failure isolation / DLQ hardening). Nothing below asks you to throw that away.

How to read this document. This is the rationale record for the generalization — the reasoning, the options rejected, and the doc-by-doc migration that was applied to 01–05. Docs 01–05 have already been updated to match it; where the two disagree, 01–05 are the current contract and this document is the argument that produced them. §5 is kept as the record of what changed and why, not as a to-do list.

---

0. What this document is

Two things: a strategy for turning the single-schema intern pipeline into a platform that can onboard any schema (forms being the first proof), and a doc-by-doc guide for updating 01–05 so the written contract matches the new goal instead of drifting from it.

The one-line thesis: we are adding a second, orthogonal axis of genericity. The existing docs already generalize the pipeline mechanics (add a target adapter, config-not-code — UC-9, UC-12). They deliberately hold the schema fixed. This work makes the schema an axis too — without weakening a single guarantee the current design earns.

---

1. The two paths, and which we're building

	Path A — instances as pods	Path B — instances as rows
Unit of a "schema"	A stamped source-service + adapter pods, one set per schema	A contract row in a registry, read by shared services
Genericity lives in	The control plane (an operator that spawns)	The data plane (services parameterized by contract)
Isolation	Hard (pod/namespace per schema)	Soft (logical, by contractId)
Ops substrate	Kubernetes operator + IIPInstance CRD	Existing Compose, plus a control-plane API
Solo-build cost	High — the operator is the biggest thing in the plan	Moderate — no build farm, no per-schema deploy
User-facing outcome	Identical	Identical

Decision: build Path B now; design so Path A is a later deployment topology, not a redesign.

The hinge that keeps both futures open is that the source-service becomes parameterized — it loads a contract from data at boot rather than compiling a schema in. In Path B that contract is a registry row. In Path A it's config mounted into a stamped pod. Same code, same contract shape, different deployment. Choosing parameterized over codegen now is exactly what makes the eventual jump to Path A a topology change instead of a rewrite.

We are not building: per-schema codegen, a UI that authors novel adapter logic (that's a low-code product — Retool/n8n — not an extension of IIP), or a Kubernetes operator yet.

---

2. Target architecture (Path B)

Five components. Two you already have; three are new.

Component	State	Role
Contract Registry	new	Stores contracts (schema + natural-key strategy + field defs) and adapter-attachments as data. The system of record for "what schemas exist and where they fan out."
Source Service (parameterized)	refactor of existing	One image. Loads a contract by contractId, validates incoming payloads against it, builds the canonical envelope, publishes. No compiled-in schema.
Adapters (config-driven)	refactor of existing	Consume by contractId/topic, read their target mapping from the registry, apply the generic write path. Catalog of types: postgres, csv, webhook.
Control-Plane API + UI	new	CRUD over contracts and adapter-attachments. "Define a schema," "attach an adapter" become registry writes — no redeploy.
Kafka / infra	have	Unchanged in principle; topics become derivable from contract + record type.

2.1 The envelope / payload split (the core idea)

Today's canonical record (03 §1) is flat — internId, firstName, … at top level. Generalization splits it into a fixed envelope (identical for every schema) and a per-contract payload (validated against the registry):

{
  "recordId":      "5c1f2e4a-…",        // per-event UUID, server-set — universal idempotency key
  "contractId":    "interns",           // which contract this conforms to
  "recordType":    "intern.created",    // event type within the contract
  "schemaVersion": 1,
  "naturalKey":    "INT001",            // derived per contract's key strategy — Kafka partition key
  "occurredAt":    "2026-07-21T14:10:00Z",
  "traceId":       "b6e2a9d0-…",
  "payload": {                          // ← the only part that varies by schema
    "internId": "INT001", "firstName": "John", "lastName": "Doe",
    "email": "john@example.com", "college": "MIT", "department": "Data Engineering",
    "mentor": "Alice", "startDate": "2026-08-01", "status": "ACTIVE"
  }
}

Interns becomes the first contract, not a special case. Forms becomes the second. The envelope is what crosses every service boundary; the payload is opaque to everything except the source-service (which validates it against the registry) and the adapter (which maps it per the registry).

2.2 What stays guaranteed vs what generalizes

This is the finding that makes the whole thing safe, and it's worth stating precisely because it's counterintuitive: the idempotency guarantee never depended on the schema being fixed.

Guarantee (from 03 / UC-5/6/7/8)	Envelope-level (universal) or per-contract?	Consequence
ON CONFLICT (record_id) DO NOTHING	Envelope — recordId is universal	Survives untouched. Exactly-once-in-effect holds for any schema, free.
Ordering by partition key	Envelope — naturalKey is the key	Survives; each contract declares how its key is derived.
Retry / DLQ / failure isolation	Envelope — operates on the message, not the payload	Survives; the generic adapter pattern (01 §6) was already payload-agnostic.
intern.updated → ON CONFLICT (intern_id) DO UPDATE	Per-contract — intern_id is a declared natural key	Generalizes to "upsert on the contract's declared natural key."
Typed columns / field-by-field mapping	Per-contract	Becomes the landing-table strategy below.

So the only thing that actually generalizes is the natural-key upsert and the field mapping — both of which the contract already declares. Everything that makes IIP trustworthy is envelope-level and comes along for free.

2.3 Landing-table strategy (the db-adapter's generic write)

The one genuinely new design question: how does an arbitrary payload reach Postgres without a redeploy? Three options; we take the hybrid.

• Pure JSONB — one table, payload jsonb. Zero DDL ever, fully runtime. Loses typed columns.
• DDL-per-contract — CREATE TABLE on contract registration. Real types, real SQL. Heavy ops: the service holds DDL grants, schema drift, ALTER TABLE migrations. Too much to run solo.
• Hybrid (chosen) — typed exactly where a guarantee needs it, JSONB for the rest:

CREATE TABLE records (
  record_id    UUID PRIMARY KEY,          -- envelope; idempotency guard (universal)
  contract_id  TEXT NOT NULL,             -- envelope
  record_type  TEXT NOT NULL,             -- envelope
  natural_key  TEXT NOT NULL,             -- envelope; typed + indexed for per-contract upsert
  payload      JSONB NOT NULL,            -- per-contract, opaque to the adapter
  occurred_at  TIMESTAMPTZ NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (contract_id, natural_key)       -- conflict target for update-style events
);
-- Fields a contract marks "queryable" become expression indexes over payload, added per contract:
-- CREATE INDEX ON records ((payload->>'status')) WHERE contract_id = 'interns';

Runtime-native like pure JSONB, but the two things that must be typed — the idempotency key and the natural key — are typed and indexed. This is the same philosophy the existing docs already live by: generic in transport, specific exactly where a guarantee needs it. For forms specifically (150 questions, variable-length options arrays) JSONB is a genuinely better fit than flattening into columns.

Contracts that want a dedicated, fully-typed table can still declare one — the postgres adapter type supports a "shaped table" mode as a per-contract config. The records table is the zero-config default, not a mandate.

---

3. Sequencing: generalize from where you are

You do not need to finish Releases 3–7 on interns first. The intern pod is not a dependency of the platform work — it is the first instance of it. R3–R7 add capability to one schema (lifecycle, dashboards, replay); this work adds capability across schemas. Orthogonal axes.

The trap to avoid: complete-then-generalize. Every intern-specific thing you hand-build in R3–R7 (compiled DTO, typed mapper, intern.updated handler) is code you'd re-derive generically later — i.e. more to build and then more to tear out. Worse, finishing R7 first bakes "schema is fixed" into five more releases' worth of code, making the eventual generalization more expensive, not less.

The rule: generalize at the current line. From end-of-Phase-2, the next move is the contract extraction (§4), with interns as the first contract — proving parameterization works by making your existing pod be a config-driven instance rather than a hardcoded one. That's the cheapest possible proof and it's available right now.

The one goals-question only you can answer: if the real objective is a polished portfolio piece, a deeply-featured single pipeline (R1–R7 on interns, dashboards, DLQ replay) may demonstrate more than a half-built platform-of-platforms. If "impressive and done" beats "ambitious and in-progress," invert this and finish the pod. Decide it deliberately — don't default into "finish everything first" because it feels orderly.

3.1 Proposed re-sequenced roadmap

Insert a generalization track immediately after current state; fold the already-planned registry work (Schema Registry was R3, Target Registry was R6) into it, since a contract registry is largely those two brought forward and given a UI.

New release	Theme	Notes
R1–R2	Baseline pipeline + failure isolation/DLQ	✅ done (your current state)
R3 (was Schema Registry)	Contract extraction + envelope split	Pull the intern schema out of code into a contract definition the source-service reads (first from a baked-in file, §4). Introduce the envelope. Interns = contract #1.
R4	Parameterized source-service + Contract Registry service	Source-service loads any contract by id. Registry becomes a real service with an API. Still one adapter type (db).
R5	Config-driven db-adapter + generic landing table	The records hybrid table; upsert on declared natural key. Prove forms lands end-to-end as contract #2.
R6 (was Target Registry / UC-9/12)	Adapter catalog + attachments + control-plane UI	csv and webhook adapter types; attach-adapter and define-contract as UI-driven registry writes. This is where the user-facing goal is met.
R7	Lifecycle events (created/updated/deleted), generalized	The intern.updated logic re-expressed as "update-style event on declared natural key" — now works for every contract.
R8	Dashboards + DLQ replay	Former R4/R7, unchanged in spirit; now multi-contract aware.
R9 (optional)	Path A: IIPInstance CRD + operator	Only if hard isolation becomes a real requirement. Reuses the same parameterized images — a topology change, not a rewrite.

---

4. The first commit: extract the contract

The single next step that serves both paths and commits to neither. Right now the intern schema lives implicitly in a compiled DTO + mapper. Pull it into an explicit contract the source-service reads — even if, at first, it reads one file baked into the image.

Moves out of code → into the contract:

• field names, types, required-ness
• the natural-key strategy (internId)
• the target mapping (which table, which conflict column)
• the enum domain for status

Stays in code (envelope-level, universal):

• recordId / createdAt generation
• envelope construction + publish
• retry / DLQ / idempotency-gate machinery
• offset-commit discipline

A first-cut contract file (later a registry row):

contractId: interns
title: Intern Records
schemaVersion: 1
naturalKey:
  strategy: field
  fields: [internId]            # composite keys join with '|', e.g. [formId, questionId]
recordTypes:
  - intern.created
  - intern.updated              # update-style: upsert on naturalKey
  - intern.deleted              # tombstone
fields:
  - { name: internId,   type: string, required: true,  queryable: true }
  - { name: firstName,  type: string, required: true }
  - { name: lastName,   type: string, required: true }
  - { name: email,      type: email,  required: true }
  - { name: college,    type: string, required: true }
  - { name: department, type: string, required: true }
  - { name: mentor,     type: string, required: false }
  - { name: startDate,  type: date,   required: true }
  - { name: status,     type: enum,   required: true, values: [ACTIVE, COMPLETED, WITHDRAWN], queryable: true }

When this file loads cleanly and the intern pod behaves exactly as before, you've proven the parameterization. Contract #2 (forms) is then a second file, no new code.

4.1 Contract Registry schema (R4)

CREATE TABLE contracts (
  contract_id     TEXT PRIMARY KEY,
  title           TEXT NOT NULL,
  schema_version  INT  NOT NULL,
  definition      JSONB NOT NULL,        -- the full contract (fields, key strategy, record types)
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE adapter_attachments (
  attachment_id   UUID PRIMARY KEY,
  contract_id     TEXT NOT NULL REFERENCES contracts(contract_id),
  adapter_type    TEXT NOT NULL,          -- 'postgres' | 'csv' | 'webhook'
  config          JSONB NOT NULL,         -- target-specific: table, endpoint, auth, mapping
  enabled         BOOLEAN NOT NULL DEFAULT true,
  UNIQUE (contract_id, adapter_type, config)
);

"Define a schema" = insert into contracts. "Attach an adapter" = insert into adapter_attachments. Both are runtime registry writes the UI performs. This is Path A's mounted-config and Path B's registry-row, unified.

---

5. Docs migration guide (01–05)

The rename first: the platform is no longer intern-specific. Keep the iip- prefix and the intern use case, but the framing shifts from "Intern Integration Platform" to "an integration platform whose first tenant is interns." Do this as a top note in each doc rather than a global find-replace that would erase the historical record in 00.

00 — Original Specification

Leave unchanged. It's preserved history by design. Add a single banner at top: "This is the original single-schema brief. The platform has since generalized to multi-schema contracts — see doc 06 and the ADs added to doc 01."

01 — Architecture

The heaviest changes live here. Add four ADs and update the container + deployment views.

• C4 L2 (containers): add Contract Registry and Control-Plane API/UI as new containers. Mark Source Service and adapters as "parameterized by contract."
• Deployment view: note the Path B topology (shared services + registry) and flag Path A (per-contract pods via operator) as a documented future option.
• §6 generic adapter pattern: clarify it now reads its mapping from the registry; the pattern itself is unchanged (this is the point — it was always payload-agnostic).

New ADs. (These were drafted as AD-7…AD-10; 01 already had eight decisions, so they landed as AD-9…AD-12 — the numbering below is the as-shipped numbering, not the draft's.)

AD-9 — Parameterized services over per-schema codegen.Context: onboarding a new schema must not require generating, building, and deploying a new code artifact per schema. Decision: services load their contract as data at boot; a new schema is a config/registry change, not a build. Rationale: one image to patch and version instead of N; instant spawn; validation failures surface as data, not compile errors; trivial rollback. Mirrors Kafka Connect / Debezium / Temporal worker patterns. Consequences: the source-service must have no compiled-in schema; all schema-specific behavior is contract-driven. Enables AD-12.

AD-10 — Adapter catalog + config over UI-authored adapters.Context: users need new fan-out targets without a developer for each. Decision: the UI instantiates and configures pre-built adapter types (postgres, csv, webhook); it does not author novel adapter logic. A generic webhook type absorbs the long tail of "some other HTTP API" as config. Rationale: UI-authored transforms are a low-code product (Retool/n8n) — an order of magnitude more surface than the platform itself, and a sandboxed-execution/versioning/testing burden unfit for a solo build. New types are added by a developer per UC-9. Consequences: "create an adapter via UI" means attach+configure, not author. Keeps faith with UC-9/UC-12.

AD-11 — Canonical model = fixed envelope + per-contract payload.Context: 03 previously declared one fixed canonical schema as the single source of truth for shape. Decision: redefine the canonical model as a registry of schemas sharing a common envelope. The envelope (recordId, contractId, recordType, schemaVersion, naturalKey, occurredAt, traceId) is fixed and crosses every boundary; the payload is per-contract and validated against the registry. Rationale: the idempotency and ordering guarantees are all envelope-level (recordId, naturalKey) and survive unchanged; only the natural-key upsert and field mapping generalize, and the contract already declares both. Consequences: 03 §1 is rewritten (see below). Interns becomes contract #1, not a special case.

AD-12 — Instances as registry rows now; pods later.Context: multi-schema could mean logical tenancy (shared services) or physical (pod per schema). Decision: ship logical tenancy by contractId (Path B). Introduce per-contract pods (Path A, IIPInstance CRD + operator) only if hard isolation becomes a real requirement. Rationale: identical user-facing outcome at a fraction of the solo ops cost; because services are parameterized (AD-9), the later jump is a deployment topology change, not a rewrite. Consequences: soft isolation initially (documented); Path A kept as a first-class, low-friction future.

02 — Use Cases

Add one actor and three use cases; re-frame three existing ones.

• New actor — Integration Designer (the person who defines a schema and wires its targets in the UI; a hat the solo dev wears, like the others).
• Re-frame UC-1 ("Submit Intern Record") as an instance of a generic "Submit Record to a Contract." Keep the intern walkthrough as the worked example.
• UC-9 / UC-12 already point this direction; update them to reference contracts and the registry explicitly rather than intern-only targets.

New UCs, in your table format:

UC-13 — Define a Contract via UI (new — R6)Actor: Integration Designer · Goal: register a new schema (fields, types, natural key, record types) so the platform can accept and route its records with no redeploy. Main flow: 1. Designer fills the contract form in the UI. 2. UI validates field/key definitions client-side. 3. UI POST /contracts. 4. Control-plane persists to the Contract Registry. 5. Parameterized source-service picks up the new contract (on refresh/next boot). 6. The contract is now live for submissions. Postcondition: a new contractId exists; records conforming to it are accepted and validated. NFRs: runtime extensibility, no redeploy.

UC-14 — Attach an Adapter via UI (new — R6)Actor: Integration Designer · Goal: fan a contract's records out to a target by instantiating a catalog adapter type as config. Main flow: 1. Designer picks an adapter type (postgres/csv/webhook) for a contract. 2. Enters target config (table / path / endpoint+auth). 3. UI POST /contracts/{id}/adapters. 4. Registry stores the attachment. 5. The relevant adapter, filtering by contractId, begins writing. Existing adapters/contracts unaffected (per UC-9's isolation principle). Postcondition: the target receives the contract's records going forward. NFRs: extensibility-by-addition, independent deployability.

UC-15 — Provision a Contract Instance (new — R9, Path A only)Actor: Platform Operator · Goal: run a schema in its own isolated pod set. Main flow: operator (or UI) writes an IIPInstance CR; the operator reconciles it into topics, a configured source-service pod, and attached adapter pods from the catalog images. Postcondition: hard-isolated instance. NFRs: blast-radius isolation, multi-tenancy. Gated on AD-12's "if isolation becomes a real requirement."

03 — Data Model

The structural rewrite. Concretely:

1. §1 Canonical Record Schema → split into §1a Envelope (the fixed fields table) and §1b Payload (per-contract, defined by the contract in the registry). Move the current intern field table into §1b as the interns contract's payload, not as "the" schema.
2. New §1c Contract Definition — the YAML/JSON shape from §4 above; the registry tables from §4.1.
3. §2 Event Types — generalize "intern.created/updated/deleted" to "created / update-style / tombstone" record types declared per contract; keep interns as the worked example.
4. §4 Target Storage Models — add the hybrid records table (§2.3) as the default; keep the typed interns table as the "shaped-table mode" example.
5. §5 Schema Evolution — now applies per contract; BACKWARD compatibility is enforced per contractId. Add: contracts are versioned (schemaVersion), and a contract edit is a compatibility-checked registry update.
6. §6 Mapping Summary — reframe as "each contract declares its own canonical→target mapping"; the intern table becomes one instance.

04 — Implementation Plan

• Replace the R1–R7 roadmap with the re-sequenced one (§3.1).
• Risk register — add: (a) generic landing table hides schema errors → mitigate with per-contract payload validation at the source before publish; (b) control-plane privilege creep → the UI writes registry rows only; no direct infra credentials in Path B; (c) contract/target drift → registry is system-of-record, adapters read from it, never hand-configured.
• DoD — add: every contract change is compatibility-checked in CI before it goes live, exactly as schema changes are today.

05 — Phased Rollout

Rewrite the checklist from the current line. First phases of the new R3:

• Introduce the envelope type; wrap the existing intern record in it (payload = current fields).
• Extract the intern schema to a contract file (§4); source-service reads it at boot.
• Delete the compiled intern DTO/mapper; drive validation + mapping from the loaded contract.
• Prove: intern pod behaves identically, all existing tests green. ← parameterization proven.
• Add forms as contract #2 (file only, no new code); prove a forms record validates + publishes.

---

6. The through-line

Everything above rhymes with one principle your docs already embody: generic in transport, specific exactly where a guarantee needs it. The envelope is generic; the payload and its natural-key upsert are specific. The services are parameterized; the adapter types are hand-built and trustworthy. The platform stamps instances; it does not author logic. Each choice keeps the guarantees the intern pipeline earned while opening the schema axis — and each one is reversible toward Path A precisely because parameterization, not codegen, is the hinge.

Start with §4. One commit, both futures open.