Appearance
Architecture Document
System: Integration Platform (IIP) Type: Service-Oriented, Event-Driven Integration Middleware Related docs: Original Specification · Use Cases · Data Model · Implementation Plan · Generalization Strategy
Naming note: the platform keeps its
iip-prefix and its intern use case, but it is no longer intern-specific — it is an integration platform whose first tenant is interns. Interns is contract #1, not the schema. The reasoning behind that shift, and the two deployment paths it keeps open, are in 06 — Generalization Strategy.
Scope note: This document describes the architecture of the full platform, not only the MVP slice. Sections are annotated with the Release (see Implementation Plan) in which each capability first appears, so the document stays accurate as the system grows instead of needing a rewrite per release. Release numbers follow the re-sequenced roadmap (Implementation Plan §2); Releases 1–2 are built, 3+ are planned.
1. Purpose
IIP decouples the systems that produce records from the systems that consume them (a relational database, a file feed, and any future target) using an event-driven middleware built around a canonical, schema-enforced data contract. This document describes the system's structure: its context, containers, components, data flows, deployment topology, and the key decisions behind each.
The platform is generic along two orthogonal axes:
| Axis | What varies | How it's absorbed |
|---|---|---|
| Targets | Where records fan out to (Postgres, CSV, webhook, …) | A new adapter subscribes to existing topics; nothing upstream changes (UC-9, AD-10) |
| Schemas | What shape a record has (interns, forms, …) | A contract in the Contract Registry; services are parameterized by it, so a new schema is a registry write, not a build (UC-13, AD-9) |
The first axis was in the design from day one. The second is what the generalization work adds — and it adds it without weakening a single guarantee the intern pipeline already earns, because every reliability guarantee is envelope-level (see Data Model §1a).
2. Architectural Principles
| # | Principle | Enforcement mechanism in this system |
|---|---|---|
| 1 | Loose coupling | Source service has zero target-specific code; it only knows the canonical envelope. |
| 2 | Canonical data model as contract | The envelope is fixed and Schema Registry-enforced for every producer/consumer; the payload is validated against its contract in the Contract Registry. Nothing publishes unvalidated. |
| 3 | At-least-once delivery | Kafka's default durability + manual offset commit only after successful processing. |
| 4 | Idempotency everywhere | DB adapter: SQL upsert. File adapter: processed-ID dedup store. Any new adapter must implement the same contract. |
| 5 | Failure isolation | Independent consumer groups per adapter; retry/DLQ per-message, not per-partition or per-topic. |
| 6 | Independent deployability | Each service is its own container/deployment unit with its own lifecycle. |
| 7 | Extensibility by addition | New targets subscribe to existing topics; zero changes to UI, source service, or envelope schema. |
| 8 | Observability | Actuator health/metrics + consumer lag from day one; tracing and dashboards added as the system matures. |
| 9 | Runtime extensibility (config, not code) | A new schema is a Contract Registry row and a new target is an adapter attachment — neither requires a build, a codegen step, or a redeploy of the core services (AD-9, AD-10). |
| 10 | Generic in transport, specific exactly where a guarantee needs it | The envelope is universal and opaque-to-payload; typed storage and natural-key upserts exist only where a stated guarantee depends on them (AD-11). |
These ten principles are the acceptance criteria for every architectural decision below — if a proposed change violates one of them, it needs an explicit, documented justification, not a quiet exception.
3. System Context (C4 Level 1)
Actors:
| Actor | Type | Interacts via |
|---|---|---|
| HR Staff | Human | Web UI (submits records against the interns contract) |
| Integration Designer | Human | Control-Plane UI (defines contracts, attaches adapter types) — Release 6 |
| Platform Operator | Human | Actuator endpoints, Kafka UI, admin dashboard, DLQ tooling |
| Downstream target systems | System | Kafka consumer contract (canonical envelope) |
4. Container View (C4 Level 2)
Container responsibilities:
| Container | Responsibility | Introduced |
|---|---|---|
| Web UI | Form-based intake, record listing, submission-status display | Release 1 |
| Source Service | Single entry point: loads a contract, validates the payload against it, builds the canonical envelope, publishes. No compiled-in schema (AD-9). | Release 1 (intern-specific) → Release 4 (parameterized) |
| Contract Registry | System of record for what schemas exist and where they fan out: contract definitions + adapter attachments | Release 3 (baked-in file) → Release 4 (service + API) |
| Schema Registry | Enforces the fixed envelope contract and governs its evolution; per-contract payload compatibility is checked against the Contract Registry | Release 4 |
| Control-Plane API + UI | CRUD over contracts and adapter attachments — "define a schema" and "attach an adapter" become registry writes | Release 6 |
Kafka ({contract}.created) | Durable, ordered (per naturalKey) primary event stream per contract | Release 1 |
Kafka (iip.dlq) | Quarantine for poison / retry-exhausted messages, across all contracts | Release 1 |
Kafka ({contract}.updated, .deleted) | Update-style and tombstone record types | Release 7 |
postgres adapter | Envelope → SQL; idempotent upsert on record_id, natural-key upsert per contract; generic records landing table + optional shaped tables | Release 1 (typed, intern-only) → Release 5 (config-driven) |
csv adapter | Envelope → CSV; idempotent dedup-append / snapshot rebuild | Release 1 → Release 6 (config-driven) |
webhook adapter | Generic HTTP fan-out type — absorbs the long tail of "some other API" as config (AD-10) | Release 6 |
| Actuator / Kafka UI | Baseline health + lag visibility | Release 1 |
| Prometheus / Grafana | Aggregated dashboards: lag, DLQ depth, throughput, adapter health — multi-contract aware | Release 8 |
Where the Target Registry went. The Release-6 "Target Registry" of the original design is not dropped — it is subsumed by the Contract Registry's
adapter_attachmentstable (Data Model §1c). One registry answers both "what schemas exist" and "where does each fan out," because the second question is meaningless without the first.
5. Component View — Source Service (C4 Level 3)
The source service is intentionally the thinnest possible layer over "load contract, validate, canonicalize, publish." It contains no SQL, CSV, or target-specific logic — that boundary is what keeps loose coupling real rather than aspirational — and, from Release 4, no compiled-in schema either: every field name, type, enum domain, and key strategy it enforces comes from the contract it loaded at boot (AD-9).
What is not contract-driven, deliberately — these stay in code because they are universal and are exactly what makes the platform trustworthy:
| Stays in code (envelope-level) | Moves to the contract (per-schema) |
|---|---|
recordId / occurredAt / traceId generation | Field names, types, required-ness, enum domains |
| Envelope construction and publish | The natural-key strategy |
| Retry / DLQ / idempotency-gate machinery | The target mapping (which table, which conflict column) |
| Offset-commit discipline | The declared record types |
As built (Releases 1–2) this component is intern-specific: InternController, CanonicalMapper, InternRecordStore, and a compiled CanonicalInternRecord. Release 3 wraps that record in the envelope; Release 4 replaces the compiled DTO with the loaded contract. The diagram above is the Release-4 shape.
6. Component View — Generic Adapter Pattern
Every consumer (database, file, or any future target) implements the same internal shape, so the reliability guarantees apply uniformly as new adapters are added.
Why this shape is mandatory, not optional: the platform's headline guarantee ("no data loss, no duplicate side effects, no poison-message stall") is only true if every adapter honors this contract. A new adapter that skips the idempotency gate or the failure classifier silently breaks the platform's core promise, so this diagram is the acceptance checklist for any new adapter (see Use Case UC-9).
What generalization changed here: almost nothing — and that's the point. Two boxes are new (ContractFilter, MappingResolver) and both are pure lookups; the reliability spine (IdempotencyGate → TargetWriter → FailureClassifier → RetryPolicy → DlqPublisher) is unchanged, because it operates on the envelope, never on the payload. It was already payload-agnostic before there was more than one payload shape. The only substantive shift is that TargetTransformer reads its field mapping from the registry instead of having it compiled in — the pattern itself is untouched.
The two new boxes are also what make an adapter a catalog type rather than a bespoke service: one postgres adapter image serves every contract attached to it, filtering by contractId and resolving its mapping per attachment (AD-10).
7. Data Flow — Happy Path (Sequence)
Note what the fan-out half of this diagram does not contain: any knowledge of what an intern is. Both adapters branch on contractId and recordId — envelope fields — and hand the payload to a mapping they looked up. Swap interns for forms and the diagram is unchanged.
8. Data Flow — Failure Handling (Sequence)
Key property demonstrated: a transient outage on one adapter's target never affects the other adapter (separate consumer groups), and a single poison message never stalls the partition for messages behind it (bounded retry → DLQ → commit). Both properties hold per-message, and therefore per-contract for free: a contract whose payloads are systematically bad fills the DLQ with its own records without slowing any other contract sharing the same adapter.
9. Data Flow — DLQ Replay (Release 8)
10. Topic & Partitioning Design
Topics are derived from the contract, not hand-declared per schema: a contract's record types map onto {contractId}.{recordClass} topics that the control plane creates when the contract is registered. The DLQ is platform-wide, not per-contract, since it is an operator surface rather than a fan-out stream.
| Topic | Key | Partitions (baseline) | Replication | Retention | Introduced |
|---|---|---|---|---|---|
{contractId}.created (e.g. interns.created) | naturalKey | 3 | 3 | 7 days (tune per volume) | Release 1 |
iip.dlq | naturalKey | 1 | 3 | 30 days (operator review window) | Release 1 |
{contractId}.updated | naturalKey | 3 | 3 | 7 days | Release 7 |
{contractId}.deleted | naturalKey | 3 | 3 | 7 days | Release 7 |
As built vs. as designed. Releases 1–2 shipped with the literal names
intern.createdandintern.dlqand a partition key ofinternId. Under the generalized scheme these becomeinterns.createdandiip.dlq, keyed bynaturalKey— which for theinternscontract isinternId, so the key's runtime value is unchanged. The rename is a Release-3 step (Phased Rollout 3.6), and it is a rename only: no envelope field, guarantee, or partitioning behavior changes with it.
Why key by naturalKey: Kafka guarantees ordering within a partition only. Keying by the contract's declared natural key pins every event for a given business entity to the same partition, so created → updated → deleted for that entity are always processed in order by any single-instance consumer of that partition. Without this, a delete could theoretically be processed before its create. Each contract declares how its key is derived (single field, composite join, or a derived expression — Data Model §1c), which is precisely why this guarantee generalizes without weakening: the mechanism is universal, only the derivation is per-contract.
Why recordId is not the key: it is unique per event, so it would scatter one entity's lifecycle across partitions and destroy the ordering guarantee. It is the idempotency key, not the routing key — a distinction the envelope makes explicit by carrying both.
Why a single partition for iip.dlq: the DLQ is read by operators, not scaled consumers; strict ordering of "what went wrong, in what order" is more valuable there than parallelism. Keeping it single and platform-wide also means a new contract adds no operator surface — DLQ triage stays one queue, filterable by contractId.
11. Deployment View
11.1 Path B — instances as registry rows (the topology being built)
One set of shared, parameterized services; every schema is a row in the Contract Registry and a contractId on the wire. Adding a schema adds no containers.
Each application service is an independently buildable/deployable container. docker-compose.yml is the local/demo orchestration; the same images are portable to Kubernetes when the platform needs to scale beyond a single host.
Isolation characteristics, stated honestly: isolation between contracts in this topology is logical, not physical — contracts share adapter processes, the broker, and (by default) the records landing table, separated by contractId. Kafka's per-message failure handling means one contract's bad data cannot stall another's (§8), but a contract can consume a shared adapter's throughput. That trade is deliberate and is what AD-12 buys back if it ever stops being acceptable.
11.2 Path A — instances as pods (documented future, Release 9, gated)
If hard isolation ever becomes a real requirement, the same images are stamped per contract by a Kubernetes operator reconciling an IIPInstance custom resource — the contract that was a registry row becomes config mounted into a dedicated pod set.
Why this is a topology change and not a rewrite: the services are parameterized (AD-9), so the only difference between the two paths is where the contract comes from — a registry row or a mounted file. Same image, same contract shape, same envelope on the wire. Choosing parameterization over per-schema codegen now is precisely what keeps this future cheap (AD-12).
12. Technology Stack
| Layer | Technology | Rationale |
|---|---|---|
| UI | React + TypeScript + Tailwind | Fast to build, deliberately not the focus of engineering effort |
| Source Service | Spring Boot (Java) | Mature validation, Actuator, Kafka client, and Testcontainers ecosystem |
| Messaging | Apache Kafka | Durable log, consumer groups give free fan-out + independent failure domains |
| Envelope contract | Confluent Schema Registry, holding the envelope as JSON Schema (subject iip.envelope-value, BACKWARD) | Turns the fixed envelope into an enforced, versioned contract every service is checked against. Built in Release 4 as validation only: services fetch the schema and check every message against it on the way out and on the way in, while the wire stays plain canonical JSON — no Confluent framing, so an adapter still needs nothing but the bytes (AD-13) |
| Schema contracts | Contract Registry — PostgreSQL contracts / adapter_attachments tables, JSONB definitions | Contracts must be writable at runtime by the control plane; a relational store with a JSONB column gives that plus compatibility-checkable versioning without DDL per schema |
| Database | PostgreSQL | Unique constraint on record_id enables trivial idempotent upsert; JSONB gives a zero-DDL landing table for arbitrary payloads (AD-11) |
| File output | CSV + flat-file dedup index (processed recordIds, one per line) | Simplest representative "legacy file feed" target — file adapter is single-writer (AD-6), so there's no concurrent-writer race an embedded database (RocksDB/SQLite) would be needed for |
| Dead-letter | Kafka topic iip.dlq | Reuses existing broker; no new infrastructure; one queue across all contracts, filterable by contractId |
| Admin/API for control plane | Spring Boot | Consistency with rest of the stack |
| Container orchestration (Path A, Release 9, optional) | Kubernetes operator + IIPInstance CRD | Only if hard isolation becomes a requirement — reuses the same images (AD-12) |
| Build | Maven | Standard for the Spring Boot ecosystem |
| Containerization | Docker Compose (→ Kubernetes-ready) | Independent deployability principle |
| Observability | Spring Boot Actuator, Kafka UI, Prometheus, Grafana | Health/metrics near-free from the framework; lag is the critical EDA signal |
| Testing | JUnit, Testcontainers (Kafka + Postgres) | Real infrastructure in tests, not mocks — proves guarantees rather than asserting them |
13. Key Architectural Decisions
| # | Decision | Alternatives considered | Why this choice |
|---|---|---|---|
| AD-1 | Event-driven fan-out via Kafka, not point-to-point calls from the source service | Direct HTTP calls to each target; RabbitMQ/other broker | Kafka's durable log + consumer groups gives replay, ordering-per-key, and free multi-consumer fan-out without the source service knowing how many consumers exist |
| AD-2 | Canonical schema enforced via Schema Registry, not "documented convention" | Shared DTO library; JSON with no enforcement | Convention-only contracts silently drift; a registry makes an invalid publish/consume a compile/runtime failure instead of a production incident |
| AD-3 | At-least-once delivery + idempotent consumers, not exactly-once semantics | Kafka transactional exactly-once | Exactly-once adds significant operational complexity across heterogeneous targets (a CSV file can't participate in a distributed transaction); idempotent-consumer at-least-once achieves the same observable guarantee more simply |
| AD-4 | Bounded retry + DLQ per adapter, not infinite retry | Infinite retry with uncommitted offset | Infinite retry on a poison message stalls the entire partition; a DLQ trades "guaranteed eventual automatic success" for "guaranteed pipeline liveness + no silent data loss" |
| AD-5 | Each adapter is its own consumer group | One shared consumer group for all adapters | A shared group means only one adapter instance gets each message — breaks fan-out entirely; independent groups are required for the "every target gets every message" guarantee |
| AD-6 | File adapter is single-writer | Multiple instances writing one shared CSV | Concurrent appenders interleave writes and corrupt the file; documented as a real limitation of file targets rather than papered over |
| AD-7 | naturalKey as the partition key (not recordId) | Random/round-robin partitioning; recordId as key | Ordering must be guaranteed per business entity (create before update before delete); a random key or per-event UUID key cannot provide that. Originally stated as internId; generalized to "the contract's declared natural key," of which internId is the interns contract's instance |
| AD-8 | Target Registry (config-driven adapters) deferred, not built first | Build registry before any adapter exists | You cannot design a good abstraction for "pluggable adapters" before at least two concrete adapters exist to abstract over — premature abstraction risk. Now realized as the Contract Registry's adapter_attachments (Release 6) rather than a standalone registry |
13.1 Generalization decisions (added with 06 — Generalization Strategy)
These four are recorded in full rather than as table rows, because each one closes off a genuinely attractive alternative and the reasoning is the point.
AD-9 — Parameterized services over per-schema codegen.Context: onboarding a new schema must not require generating, building, and deploying a new code artifact per schema. Decision: services load their contract as data at boot; a new schema is a config/registry change, not a build. Rationale: one image to patch and version instead of N; instant spawn; validation failures surface as data, not compile errors; trivial rollback. Mirrors Kafka Connect / Debezium / Temporal worker patterns. Consequences: the source-service must have no compiled-in schema; all schema-specific behavior is contract-driven. Enables AD-12.
AD-10 — Adapter catalog + config over UI-authored adapters.Context: users need new fan-out targets without a developer for each. Decision: the UI instantiates and configures pre-built adapter types (
postgres,csv,webhook); it does not author novel adapter logic. A genericwebhooktype absorbs the long tail of "some other HTTP API" as config. Rationale: UI-authored transforms are a low-code product (Retool/n8n) — an order of magnitude more surface than the platform itself, and a sandboxed-execution/versioning/testing burden unfit for a solo build. New types are added by a developer per UC-9. Consequences: "create an adapter via UI" means attach+configure, not author. Keeps faith with UC-9 / UC-12.
AD-11 — Canonical model = fixed envelope + per-contract payload.Context: 03 previously declared one fixed canonical schema as the single source of truth for shape. Decision: redefine the canonical model as a registry of schemas sharing a common envelope. The envelope (
recordId,contractId,recordType,schemaVersion,naturalKey,occurredAt,traceId) is fixed and crosses every boundary; thepayloadis per-contract and validated against the registry. Rationale: the idempotency and ordering guarantees are all envelope-level (recordId,naturalKey) and survive unchanged; only the natural-key upsert and field mapping generalize, and the contract already declares both. Consequences: 03 §1 is rewritten as an envelope/payload split. Interns becomes contract #1, not a special case.
AD-12 — Instances as registry rows now; pods later.Context: multi-schema could mean logical tenancy (shared services) or physical (pod per schema). Decision: ship logical tenancy by
contractId(Path B). Introduce per-contract pods (Path A,IIPInstanceCRD + operator) only if hard isolation becomes a real requirement. Rationale: identical user-facing outcome at a fraction of the solo ops cost; because services are parameterized (AD-9), the later jump is a deployment topology change, not a rewrite. Consequences: soft isolation initially (documented in §11.1); Path A kept as a first-class, low-friction future.
AD-13 — The Schema Registry owns the envelope schema; it does not own the wire format.Context: Release 4 phase 4.8 called for a validating serializer and validating deserializers. The canonical way to get them is Confluent's own serializers, which prefix every message with a magic byte and a schema id. Decision: register the envelope as JSON Schema (
iip.envelope-value, BACKWARD), have all three services fetch it at startup and refuse to start without it, and validate every message against it on the way out and on the way in — but keep plain canonical JSON on the wire, with no framing. Rationale: framing would put every adapter's ability to read bytes behind a Confluent client library and a reachable registry, undoing the property held since Phase 1.12 — an adapter reads canonical JSON off a topic and needs no shared artifact to understand it (AD-1, AD-2). The registry's real value here is being the single versioned, compatibility-checked home of the envelope, and that is preserved in full. Every guarantee 4.8 asked for holds: a non-conforming envelope fails insideKafkaProducer.sendbefore batching, and a non-conforming message is quarantined by an adapter before it touches a database or a file. Consequences: each service carries its own small validation code and a test-only copy ofenvelope.json; the CI gate (infra/scripts/compatibility-gate.sh) fails if a copy drifts, since a drifted fixture makes a green suite meaningless. A topic can still be read withkafka-console-consumerandjq, which is an operability property worth having.
14. Non-Functional Requirements
| Category | Requirement | How the architecture satisfies it |
|---|---|---|
| Reliability | No record is silently lost | At-least-once delivery + DLQ quarantine for unprocessable records |
| Reliability | No duplicate side effects on redelivery | Idempotent consumers (upsert / dedup-append) on every adapter |
| Availability | A single target outage doesn't halt other targets | Independent consumer groups per adapter |
| Availability | A single bad record doesn't halt the pipeline | Bounded retry + DLQ, offset committed after quarantine |
| Extensibility | New target added without touching existing services | Adapters are independent consumers of existing topics; adapter attachments (Release 6) make activation config-driven |
| Extensibility | New schema onboarded without a build or redeploy | Services are parameterized by contract (AD-9); a contract is a registry row written by the control plane (UC-13) |
| Extensibility | New target attached to a schema without a developer | Pre-built adapter types are instantiated as config, not authored (AD-10, UC-14) |
| Consistency | All services agree on record shape | Schema Registry-enforced envelope; per-contract payloads validated against the Contract Registry at the source, before publish |
| Isolation | One contract's failures don't affect another's | Per-message retry/DLQ (not per-partition); DLQ entries carry contractId. Logical isolation only in Path B — see §11.1 |
| Ordering | Events for one business entity are processed in order | Partition key = naturalKey, derived per the contract's declared key strategy |
| Observability | Operators can see system health and backlog | Actuator /health + /metrics, Kafka UI consumer lag, Prometheus/Grafana dashboards |
| Traceability | Any record's journey is reconstructable end-to-end | recordId/traceId propagated through every hop and logged at each stage |
| Testability | Guarantees are proven, not just claimed | Testcontainers integration tests against real Kafka + Postgres, including idempotency and DLQ tests |
15. Cross-Cutting Concerns
- Traceability: every envelope carries
recordId,contractId, and atraceId; all services log all three at every processing stage (received, validated, published, consumed, written, dead-lettered), enabling end-to-end reconstruction of a single record's journey from logs alone — and, becausecontractIdis on every line, per-schema triage without a separate log stream per schema. - Configuration: the split is deliberate and worth stating. Operational parameters (retry limits, backoff, connection settings, replica counts) are externalized per service via Spring
application.yml/ environment variables. Contract parameters (schema fields, key strategy, target mapping, adapter attachments) live in the Contract Registry and are written at runtime by the control plane — never inapplication.yml, because they must change without a restart. Confusing the two is how a "config-driven" platform quietly becomes a redeploy-driven one. - Security (noted for completeness, not a Release-1 focus): the UI/source-service boundary is the natural place for authN/authZ if this were to leave a trusted internal network; out of scope for the current release set but the REST boundary is where it would be added without touching the eventing core. The control plane raises the stakes slightly — it can define contracts and attach targets — but deliberately holds no infrastructure credentials: it writes registry rows and nothing else, so a compromised control plane can misroute future records but cannot reach a target system directly (see Implementation Plan §7 risk register).
- Schema evolution governance: Schema Registry compatibility mode is backward-compatible by default for the envelope (new schema can read old data), which is what allows new optional envelope fields without breaking deployed consumers. The same rule is applied per
contractIdto payload definitions: a contract edit is a compatibility-checked registry update, versioned byschemaVersion— see Data Model §5.