Data Model & Contract Document

System: Intern Integration Platform (IIP) Related docs: Architecture · Use Cases · Implementation Plan

This document defines the canonical event contract, its evolution rules, and how each target's storage model derives from it. The canonical model is the single source of truth for shape — no adapter or the source service should ever invent fields that aren't defined here first.

---

1. Canonical Record Schema

Field	Type	Required	Notes
recordId	UUID (string)	yes	Unique per event (not per intern). Generated server-side by the Source Service. Primary idempotency key for adapters.
internId	string	yes	Business identifier for the intern (e.g., INT001). Stable across the intern's lifecycle. Used as the Kafka partition key.
firstName	string	yes
lastName	string	yes
email	string (email format)	yes
college	string	yes
department	string	yes
mentor	string	no
startDate	date (YYYY-MM-DD)	yes
status	enum: ACTIVE, COMPLETED, WITHDRAWN	yes	Extendable via schema evolution (§5).
createdAt	timestamp (ISO-8601 UTC)	yes	Set server-side at publish time.
updatedAt	timestamp (ISO-8601 UTC)	conditional	Present on intern.updated events only (Release 5).
traceId	UUID (string)	yes (Release 5+)	Correlates all events belonging to one logical user action across the pipeline for end-to-end log reconstruction.

Example (intern.created):

{
  "recordId":   "5c1f2e4a-3b7d-4e9a-9c2f-8a6d7e1b0c3f",
  "internId":   "INT001",
  "firstName":  "John",
  "lastName":   "Doe",
  "email":      "john@example.com",
  "college":    "MIT",
  "department": "Data Engineering",
  "mentor":     "Alice",
  "startDate":  "2026-08-01",
  "status":     "ACTIVE",
  "createdAt":  "2026-07-21T14:10:00Z",
  "traceId":    "b6e2a9d0-1c4f-4a2e-8e7a-2f9d3c5b1a70"
}

No service other than the Source Service (producer) and the adapters (consumers) ever sees this record in a target-specific form — the canonical shape is what crosses every service boundary.

---

2. Event Types (Lifecycle)

Event	Topic	Introduced	Payload delta vs. intern.created
Created	intern.created	Release 1	Baseline schema above (minus updatedAt).
Updated	intern.updated	Release 5	Adds updatedAt; carries the full current state of the record (not a diff) — simplest for consumers to apply, avoids reconstructing state from partial deltas.
Deleted	intern.deleted	Release 5	Minimal tombstone payload: recordId, internId, deletedAt, traceId. No need to carry full record data for a deletion.

---

3. DLQ Message Envelope

Messages routed to intern.dlq wrap the original payload rather than replacing it, so nothing is lost and the failure is fully diagnosable.

Field	Type	Description
originalTopic	string	Topic the message originated from (e.g., intern.created).
originalPartition	int	Partition of the original message.
originalOffset	long	Offset of the original message, for correlation with broker logs.
originalKey	string	The original internId key.
originalPayload	bytes / JSON	The unmodified original message (preserved exactly).
errorType	string	Classification, e.g. SCHEMA_VIOLATION, UNMAPPABLE_DATA, RETRY_EXHAUSTED.
errorMessage	string	Human-readable error summary.
failedAdapter	string	Which consumer group/adapter produced this DLQ entry.
attemptCount	int	How many processing attempts were made before quarantine.
quarantinedAt	timestamp	When it was routed to the DLQ.
replayed	boolean	Set true by the DLQ Replay Tool (Release 7) once reprocessed — entry is kept for audit, not deleted.
replayedAt	timestamp, nullable	When replay occurred.

---

4. Target Storage Models

4.1 PostgreSQL (Database Adapter)

• record_id carries a UNIQUE constraint — this is what makes ON CONFLICT (record_id) DO NOTHING a correct idempotency guard for intern.created redelivery.
• intern_id also carries a UNIQUE constraint (one current-state row per intern) — this is the conflict target for intern.updated (ON CONFLICT (intern_id) DO UPDATE).
• INTERN_AUDIT_LOG (Release 5) is append-only by design — it's the audit trail, so it should not be upserted; every event (created/updated/deleted) adds a row here regardless of what happens to the INTERNS row.

4.2 CSV File (File Adapter)

Columns (Release 1, create-only):

record_id,intern_id,first_name,last_name,email,college,department,mentor,start_date,status,created_at

Dedup index: a local key-value store (RocksDB / SQLite / embedded H2) mapping recordId -> written (bool), consulted before every append and updated after every successful write. This is what makes the naturally non-idempotent "append a line" operation idempotent in effect.

Release 5 shift — snapshot rebuild instead of pure append: once intern.updated/intern.deleted exist, the file adapter maintains its keyed store as current state per internId (not just a seen-set), and on every event rebuilds interns.csv from that current state:

This resolves the append-only-vs-mutable tension named in the original spec (§7) by treating the CSV as a projection of current state, not a raw event log — option 2 from the original design brief, adopted platform-wide rather than deferred.

---

5. Schema Evolution Rules

Schema Registry compatibility mode: BACKWARD (default) — a new schema can read data written with the previous schema. This is the correct default for this platform because adapters are deployed independently (principle: independent deployability) and must not break when the Source Service ships a schema change first.

Change	Allowed under BACKWARD compatibility?	Notes
Add a new optional field with a default	Yes	Safe, most common evolution (e.g., adding traceId).
Add a new required field	No, unless a default is supplied	Would break consumers built against the old schema reading new data — must supply a default.
Remove a field consumers depend on	No	Breaking; requires a coordinated multi-step migration (deprecate, stop-writing, then remove).
Widen a type (e.g., int → long)	Generally yes (Avro promotion rules)	Verify against the specific serialization format's promotion table.
Rename a field	No (treated as remove + add)	Use aliasing support if the serialization format provides it, or a two-phase add-new/remove-old rollout.
Add a new enum value to status	Yes, if consumers treat unknown values defensively	Consumers should not switch exhaustively without a default case.

Governance process: every schema change is proposed as a PR against the schema definition, validated by the Schema Registry's compatibility check in CI before merge — an incompatible change fails the build, not production.

---

6. Canonical Model → Target Mapping Summary

Canonical field	PostgreSQL column	CSV column	Notes
recordId	record_id (UNIQUE)	record_id	Idempotency key everywhere
internId	intern_id (UNIQUE)	intern_id	Partition key / natural key
firstName	first_name	first_name
lastName	last_name	last_name
email	email	email
college	college	college
department	department	department
mentor	mentor (nullable)	mentor (blank if absent)
startDate	start_date	start_date
status	status	status
createdAt	created_at	created_at
updatedAt	updated_at (nullable)	(reflected via snapshot rebuild, no dedicated column)	Release 5
traceId	(logged, not persisted in the row)	(logged, not persisted)	Kept in structured logs / audit log table, not the projection tables

This table is the contract every adapter's transform is unit-tested against (see Implementation Plan §7 — Engineering Practices).