Appearance
Architecture Document
System: Intern Integration Platform (IIP) Type: Service-Oriented, Event-Driven Integration Middleware Related docs: Original Specification · Use Cases · Data Model · Implementation Plan
Scope note: This document describes the architecture of the full platform, not only the MVP slice. Sections are annotated with the Release (see Implementation Plan) in which each capability first appears, so the document stays accurate as the system grows instead of needing a rewrite per release.
1. Purpose
IIP decouples the systems that produce intern data (HR) from the systems that consume it (a relational database, a payroll file feed, and any future target) using an event-driven middleware built around a canonical, schema-enforced data contract. This document describes the system's structure: its context, containers, components, data flows, deployment topology, and the key decisions behind each.
2. Architectural Principles
| # | Principle | Enforcement mechanism in this system |
|---|---|---|
| 1 | Loose coupling | Source service has zero target-specific code; it only knows the canonical schema. |
| 2 | Canonical data model as contract | Schema Registry rejects any producer/consumer that doesn't conform. |
| 3 | At-least-once delivery | Kafka's default durability + manual offset commit only after successful processing. |
| 4 | Idempotency everywhere | DB adapter: SQL upsert. File adapter: processed-ID dedup store. Any new adapter must implement the same contract. |
| 5 | Failure isolation | Independent consumer groups per adapter; retry/DLQ per-message, not per-partition or per-topic. |
| 6 | Independent deployability | Each service is its own container/deployment unit with its own lifecycle. |
| 7 | Extensibility by addition | New targets subscribe to existing topics; zero changes to UI, source service, or topic schema. |
| 8 | Observability | Actuator health/metrics + consumer lag from day one; tracing and dashboards added as the system matures. |
These eight principles are the acceptance criteria for every architectural decision below — if a proposed change violates one of them, it needs an explicit, documented justification, not a quiet exception.
3. System Context (C4 Level 1)
Actors:
| Actor | Type | Interacts via |
|---|---|---|
| HR Staff | Human | Web UI |
| Platform Operator | Human | Actuator endpoints, Kafka UI, admin dashboard, DLQ tooling |
| Downstream target systems | System | Kafka consumer contract (canonical schema) |
4. Container View (C4 Level 2)
Container responsibilities:
| Container | Responsibility | Introduced |
|---|---|---|
| Web UI | Form-based intake, record listing, submission-status display | Release 1 |
| Source Service | Single entry point: validation, canonical mapping, schema-validated publish | Release 1 |
| Schema Registry | Enforces the canonical contract; governs schema evolution | Release 3 |
Kafka (intern.created) | Durable, ordered (per internId) primary event stream | Release 1 |
Kafka (intern.dlq) | Quarantine for poison / retry-exhausted messages | Release 1 |
Kafka (intern.updated, intern.deleted) | Lifecycle events beyond create | Release 5 |
| Database Adapter | Canonical → SQL, idempotent upsert | Release 1 |
| File Adapter | Canonical → CSV, idempotent dedup-append | Release 1 |
| Target Registry | Config-driven enablement of adapters (no hardcoding) | Release 6 |
| Additional adapters | Prove extensibility-by-addition (e.g. webhook, document store) | Release 6 |
| Actuator / Kafka UI | Baseline health + lag visibility | Release 1 |
| Prometheus / Grafana | Aggregated dashboards: lag, DLQ depth, throughput, adapter health | Release 4 |
5. Component View — Source Service (C4 Level 3)
The source service is intentionally the thinnest possible layer over "validate, canonicalize, publish." It contains no SQL, CSV, or target-specific logic — that boundary is what keeps loose coupling real rather than aspirational.
6. Component View — Generic Adapter Pattern
Every consumer (database, file, or any future target) implements the same internal shape, so the reliability guarantees apply uniformly as new adapters are added.
Why this shape is mandatory, not optional: the platform's headline guarantee ("no data loss, no duplicate side effects, no poison-message stall") is only true if every adapter honors this contract. A new adapter that skips the idempotency gate or the failure classifier silently breaks the platform's core promise, so this diagram is the acceptance checklist for any new adapter (see Use Case UC-9).
7. Data Flow — Happy Path (Sequence)
8. Data Flow — Failure Handling (Sequence)
Key property demonstrated: a transient outage on one adapter's target never affects the other adapter (separate consumer groups), and a single poison message never stalls the partition for messages behind it (bounded retry → DLQ → commit).
9. Data Flow — DLQ Replay (Release 7)
10. Topic & Partitioning Design
| Topic | Key | Partitions (baseline) | Replication | Retention | Introduced |
|---|---|---|---|---|---|
intern.created | internId | 3 | 3 | 7 days (tune per volume) | Release 1 |
intern.dlq | internId | 1 | 3 | 30 days (operator review window) | Release 1 |
intern.updated | internId | 3 | 3 | 7 days | Release 5 |
intern.deleted | internId | 3 | 3 | 7 days | Release 5 |
Why key by internId: Kafka guarantees ordering within a partition only. Keying by internId pins every event for a given intern to the same partition, so created → updated → deleted for that intern are always processed in order by any single-instance consumer of that partition. Without this, a delete could theoretically be processed before its create.
Why a single partition for intern.dlq: the DLQ is read by operators, not scaled consumers; strict ordering of "what went wrong, in what order" is more valuable there than parallelism.
11. Deployment View
Each application service is an independently buildable/deployable container. docker-compose.yml is the local/demo orchestration; the same images are portable to Kubernetes when the platform needs to scale beyond a single host (out of scope for the initial releases, noted here as the natural next step).
12. Technology Stack
| Layer | Technology | Rationale |
|---|---|---|
| UI | React + TypeScript + Tailwind | Fast to build, deliberately not the focus of engineering effort |
| Source Service | Spring Boot (Java) | Mature validation, Actuator, Kafka client, and Testcontainers ecosystem |
| Messaging | Apache Kafka | Durable log, consumer groups give free fan-out + independent failure domains |
| Schema contract | Confluent Schema Registry (Avro or JSON Schema) | Turns the canonical model into an enforced, versioned contract |
| Database | PostgreSQL | Unique constraint on record_id enables trivial idempotent upsert |
| File output | CSV + embedded dedup index (RocksDB/SQLite) | Simplest representative "legacy file feed" target |
| Dead-letter | Kafka topic intern.dlq | Reuses existing broker; no new infrastructure |
| Admin/API for control plane | Spring Boot | Consistency with rest of the stack |
| Build | Maven | Standard for the Spring Boot ecosystem |
| Containerization | Docker Compose (→ Kubernetes-ready) | Independent deployability principle |
| Observability | Spring Boot Actuator, Kafka UI, Prometheus, Grafana | Health/metrics near-free from the framework; lag is the critical EDA signal |
| Testing | JUnit, Testcontainers (Kafka + Postgres) | Real infrastructure in tests, not mocks — proves guarantees rather than asserting them |
13. Key Architectural Decisions
| # | Decision | Alternatives considered | Why this choice |
|---|---|---|---|
| AD-1 | Event-driven fan-out via Kafka, not point-to-point calls from the source service | Direct HTTP calls to each target; RabbitMQ/other broker | Kafka's durable log + consumer groups gives replay, ordering-per-key, and free multi-consumer fan-out without the source service knowing how many consumers exist |
| AD-2 | Canonical schema enforced via Schema Registry, not "documented convention" | Shared DTO library; JSON with no enforcement | Convention-only contracts silently drift; a registry makes an invalid publish/consume a compile/runtime failure instead of a production incident |
| AD-3 | At-least-once delivery + idempotent consumers, not exactly-once semantics | Kafka transactional exactly-once | Exactly-once adds significant operational complexity across heterogeneous targets (a CSV file can't participate in a distributed transaction); idempotent-consumer at-least-once achieves the same observable guarantee more simply |
| AD-4 | Bounded retry + DLQ per adapter, not infinite retry | Infinite retry with uncommitted offset | Infinite retry on a poison message stalls the entire partition; a DLQ trades "guaranteed eventual automatic success" for "guaranteed pipeline liveness + no silent data loss" |
| AD-5 | Each adapter is its own consumer group | One shared consumer group for all adapters | A shared group means only one adapter instance gets each message — breaks fan-out entirely; independent groups are required for the "every target gets every message" guarantee |
| AD-6 | File adapter is single-writer | Multiple instances writing one shared CSV | Concurrent appenders interleave writes and corrupt the file; documented as a real limitation of file targets rather than papered over |
| AD-7 | internId as the partition key (not recordId) | Random/round-robin partitioning; recordId as key | Ordering must be guaranteed per intern (create before update before delete); a random key or per-event UUID key cannot provide that |
| AD-8 | Target Registry (config-driven adapters) deferred to Release 6, not built first | Build registry before any adapter exists | You cannot design a good abstraction for "pluggable adapters" before at least two concrete adapters exist to abstract over — premature abstraction risk |
14. Non-Functional Requirements
| Category | Requirement | How the architecture satisfies it |
|---|---|---|
| Reliability | No record is silently lost | At-least-once delivery + DLQ quarantine for unprocessable records |
| Reliability | No duplicate side effects on redelivery | Idempotent consumers (upsert / dedup-append) on every adapter |
| Availability | A single target outage doesn't halt other targets | Independent consumer groups per adapter |
| Availability | A single bad record doesn't halt the pipeline | Bounded retry + DLQ, offset committed after quarantine |
| Extensibility | New target added without touching existing services | Adapters are independent consumers of existing topics; Target Registry (Release 6) makes activation config-driven |
| Consistency | All services agree on record shape | Schema Registry-enforced canonical contract |
| Ordering | Events for one intern are processed in order | Partition key = internId |
| Observability | Operators can see system health and backlog | Actuator /health + /metrics, Kafka UI consumer lag, Prometheus/Grafana dashboards |
| Traceability | Any record's journey is reconstructable end-to-end | recordId/traceId propagated through every hop and logged at each stage |
| Testability | Guarantees are proven, not just claimed | Testcontainers integration tests against real Kafka + Postgres, including idempotency and DLQ tests |
15. Cross-Cutting Concerns
- Traceability: every canonical record carries
recordIdand atraceId; all services log both at every processing stage (received, validated, published, consumed, written, dead-lettered), enabling end-to-end reconstruction of a single intern's journey from logs alone. - Configuration: target activation (Release 6 Target Registry), retry limits, and backoff parameters are externalized (Spring
application.yml/ environment variables), not hardcoded, consistent with the independent-deployability principle. - Security (noted for completeness, not a Release-1 focus): the UI/source-service boundary is the natural place for authN/authZ if this were to leave a trusted internal network; out of scope for the current release set but the REST boundary is where it would be added without touching the eventing core.
- Schema evolution governance: Schema Registry compatibility mode is set to backward-compatible by default (new schema can read old data), which is what allows new optional fields to be added without breaking already-deployed consumers — see Data Model §5.