Intern Integration Platform

Appearance

Intern Integration Platform (IIP) ​

A Service-Oriented, Event-Driven Integration Middleware

Design intent: This is a small enterprise integration platform, not "a project with Kafka." Kafka is one component — the transport backbone — inside a larger architecture built around a canonical data model, pluggable adapters, and explicit reliability guarantees. Every design choice below is made to be defensible in an architecture review, not just to work in a demo.

1. Business Scenario ​

An HR department enters details of new interns. Multiple downstream systems need that information (an HR database, a file-based payroll feed, and — over time — others). Rather than have the HR application write to every destination directly (N×M point-to-point coupling), the application submits each record once to an integration middleware. The middleware distributes it to every registered target. If a target is temporarily unavailable, the record is delivered automatically once it recovers, with no data loss and no duplicate side effects.

The core problem this solves is coupling. Point-to-point integration means every new consumer forces a change to the producer. This platform inverts that: the producer publishes an event and is done; consumers subscribe independently.

2. Architectural Principles ​

These are the non-negotiables the whole design is measured against.

PrincipleWhat it means here
Loose couplingThe source service knows nothing about SQL, CSV, XML, or any target. It publishes one canonical event.
Canonical data model as a contractEvery service speaks one schema, and that schema is enforced (schema registry), not merely agreed by convention.
At-least-once deliveryNo record is ever silently dropped. The tradeoff — possible duplicates — is handled explicitly by idempotent consumers.
Idempotency everywhereBecause at-least-once guarantees occasional redelivery, every consumer must produce the same end state whether it sees a message once or five times.
Failure isolationA broken target or a single poison message must never stall the pipeline or take down other targets.
Independent deployabilityEach service builds, deploys, scales, and fails on its own.
Extensibility by additionNew targets are added by deploying a new consumer — with zero changes to the UI, source service, or Kafka.
ObservabilityHealth, consumer lag, and per-record traceability are first-class, not bolted on.

3. High-Level Architecture ​

text
                        +----------------------+
                        |      Mock UI         |
                        |  Intern Form (React) |
                        +----------+-----------+
                                   | POST /interns
                                   v
                    +-----------------------------+
                    | Source Service (Spring Boot)|
                    | - REST API                  |
                    | - Validation                |
                    | - Canonical record creation |
                    | - Schema validation         |
                    | - Kafka producer            |
                    +--------------+--------------+
                                   | key = internId
                                   v
                 ============================================
                 ||  Kafka Topic: intern.created           ||
                 ||  (partitioned by internId, replicated) ||
                 ============================================
                       |                          |
       consumer group: |          consumer group: |
       db-adapter       |          file-adapter    |
                       v                          v
      +-----------------------+       +-----------------------+
      | Database Adapter      |       | File Adapter          |
      | Kafka Consumer        |       | Kafka Consumer        |
      | Canonical -> SQL      |       | Canonical -> CSV      |
      | UPSERT (idempotent)   |       | dedup + append        |
      +----------+------------+       +-----------+-----------+
                 | on non-retriable failure ...   |
                 v                                v
        +---------------------+        +----------------------+
        | PostgreSQL          |        | interns.csv          |
        | (unique on          |        | (single-writer)      |
        |  record_id)         |        +----------------------+
        +---------------------+

           both adapters, on poison/exhausted-retry:
                          |
                          v
              ==================================
              ||  Kafka Topic: intern.dlq      ||
              ==================================

4. Services ​

4.1 UI (React + TypeScript + Tailwind) ​

Deliberately minimal — a plain, validated form. This is not where engineering time should go.

Responsibilities: Add Intern, (optional) Edit Intern, View Submitted Records, Submit Record, client-side validation, display of submission status returned by the source service.

4.2 Source Service (Spring Boot / Java) ​

The single entry point. Spring Boot is chosen because its ecosystem (validation, Actuator, Kafka integration, Testcontainers) is well suited to enterprise integration work.

Responsibilities:

  • Expose POST /interns (and GET /interns for the view screen).
  • Validate the incoming request (Bean Validation / @Valid).
  • Generate a recordId (UUID) and createdAt timestamp server-side.
  • Build the canonical record and validate it against the registered schema.
  • Publish to intern.created, keyed by internId.
  • Return a synchronous acknowledgement (accepted / rejected) to the UI.

The source service must never contain any reference to SQL, CSV, XML, or any target-specific API. If it does, the loose-coupling principle is broken.

4.3 Kafka (Message Broker) ​

Acts as broker, persistent queue, and failure-recovery buffer.

Topics:

TopicPurposeStatus
intern.createdPrimary event streamMVP
intern.dlqDead-letter queue for poison / retry-exhausted messagesMVP
intern.updatedFuture — intern record changedPhase 2
intern.deletedFuture — intern record removedPhase 2

Partitioning: messages are keyed by internId so that all events for the same intern land on the same partition and are processed in order. This is invisible today but becomes essential the moment intern.updated / intern.deleted exist — you never want a delete processed before the create it depends on.

Consumer groups: each adapter is its own consumer group. That is what produces fan-out — every adapter receives every message independently. Within an adapter you can add more instances to scale, and Kafka will distribute partitions across them (with the file-adapter caveat in §6).

4.4 Database Adapter ​

Consumes intern.created, maps the canonical record to a SQL row, and writes to PostgreSQL.

Idempotent write: because delivery is at-least-once, this uses an upsert, not a plain insert:

sql
-- interns.record_id has a UNIQUE constraint
INSERT INTO interns (record_id, intern_id, first_name, last_name, email,
                     college, department, mentor, start_date, status, created_at)
VALUES (:recordId, :internId, ...)
ON CONFLICT (record_id) DO NOTHING;   -- or DO UPDATE for update semantics

A redelivered message therefore produces no duplicate row.

4.5 File Adapter ​

Consumes intern.created, maps the canonical record to a CSV line, and appends to interns.csv.

Idempotent append: appending is not naturally idempotent — a redelivered message would produce a duplicate line. The adapter therefore keeps a small store of already-processed recordIds (an embedded key-value store such as RocksDB / a local SQLite file, or a sidecar .processed index) and skips any recordId it has already written before appending.

Single-writer constraint: multiple instances appending to one shared interns.csv will interleave and corrupt it. This adapter is intentionally single-instance (one partition-consuming writer). If it must scale, each instance writes its own file (interns-{instance}.csv) and the files are merged downstream. This is a real limitation of file targets and is stated on purpose.

5. Canonical Data Model — as an Enforced Contract ​

Every service speaks the same language:

json
{
  "recordId":   "UUID",
  "internId":   "INT001",
  "firstName":  "John",
  "lastName":   "Doe",
  "email":      "john@example.com",
  "college":    "MIT",
  "department": "Data Engineering",
  "mentor":     "Alice",
  "startDate":  "2026-08-01",
  "status":     "ACTIVE",
  "createdAt":  "2026-07-21T14:10:00Z"
}

No service sees SQL. No service sees CSV. Only adapters perform those transformations.

The upgrade that makes this industry-grade: the canonical model is registered in a Schema Registry (Confluent Schema Registry with Avro or JSON Schema). This turns the model from a convention everyone promises to honor into an enforced, versioned contract:

  • The producer serializes against the schema; a malformed record can't even be published.
  • Consumers deserialize against the same schema and reject anything that doesn't conform.
  • Schema evolution is governed (backward/forward compatibility rules), which is exactly what you need before adding intern.updated / intern.deleted — new fields can be added without breaking existing consumers.

This is a small addition that does the most to justify the phrase "enterprise integration platform."

6. Reliability & Failure Handling ​

At-least-once delivery only holds if failures are handled deliberately. The key distinction the platform makes is retriable vs. non-retriable failure.

6.1 Transient failure (retriable) — e.g. PostgreSQL is down ​

text
UI -> Source Service -> Kafka -> Database Consumer  ❌ (DB unreachable)
                                        |
                                 offset NOT committed
                                        |
                              message stays in Kafka

The File consumer, being a separate consumer group, still succeeds — failure is isolated. When PostgreSQL recovers, the Database consumer re-reads the uncommitted messages, upserts them (idempotent, so no harm even if some partially succeeded before), and commits the offset. Nothing is lost, nothing is duplicated.

6.2 Poison message (non-retriable) — e.g. permanently malformed / unprocessable record ​

This is the failure mode the original design missed. If a single bad message is retried forever on the "leave the offset uncommitted" strategy, it blocks every message behind it on that partition — a poison-message stall. The reliability guarantee would be only half-true.

The platform handles it with retry-with-limit → dead-letter:

text
consume message
   |
   v
process ---- success ----> commit offset
   |
 failure
   |
 retriable?  --- yes ---> retry (bounded: e.g. 3 attempts w/ backoff)
   |                          |
   no                    still failing after N?
   |                          |
   +--------------------------+
   |
   v
publish original message + error metadata to intern.dlq
   |
   v
commit offset   <-- pipeline keeps moving; bad record is quarantined, not lost
  • Retriable (broker/DB unavailable, timeout): offset stays uncommitted, retry.
  • Non-retriable / retry-exhausted (schema violation, unprocessable data): route to intern.dlq, then commit the offset so the partition is unblocked.

A minimal DLQ therefore belongs in the MVP, not Phase 2 — without it, the headline reliability property doesn't fully hold. The DLQ can later be drained by an operator, replayed after a fix, or surfaced on the admin dashboard.

6.3 Traceability ​

Every record carries its recordId (and optionally a traceId) through the entire pipeline, so a single intern's journey — submitted → published → consumed by each adapter → written or dead-lettered — can be reconstructed from logs end to end.

7. Update / Delete and the Append-Only Tension ​

This is a design tension worth naming explicitly, because seeing it matters more than fully solving it in an MVP.

An event-log / append-only mindset (great for Kafka and for the CSV-as-append target) is fundamentally at odds with mutable records (Edit / Delete). You cannot "edit a line" in an append-only CSV by appending.

Options, from simplest to most complete:

  1. Scope Edit/Delete out of the MVP. Ship create-only, and state clearly why: it keeps the file target honestly append-only and avoids premature complexity. (Recommended for the timebox.)
  2. CSV as a rebuilt snapshot. Maintain a keyed store (the same dedup store from §4.5) and, on each change, rebuild interns.csv from the current state rather than appending. The file becomes a projection of state, not an event log.
  3. Full event-sourcing. intern.created / intern.updated / intern.deleted are all events; each target's current state is a materialized view folded from the event stream. Most "correct," most work.

The database adapter handles updates gracefully already (ON CONFLICT ... DO UPDATE). The file adapter is where the tension bites. Pick option 1 for the MVP and cite option 2/3 as the extension path.

8. Observability ​

Two of the non-functional requirements come almost for free with the chosen stack:

  • Spring Boot Actuator exposes /health, /metrics, and Kafka consumer metrics with near-zero effort. Wire it into each Spring service.
  • Kafka UI (or kafka-consumer-groups) shows consumer lag per group — the single most useful health signal in an event-driven system, because rising lag means a consumer is falling behind or stuck.

Phase 2 observability: Prometheus + Grafana dashboards for lag, throughput, adapter health, DLQ depth, and per-adapter success/failure counts.

9. Testing Strategy (a differentiator) ​

  • Unit tests for validation, canonical mapping, and each adapter's transform (canonical → SQL, canonical → CSV).
  • Idempotency tests: deliver the same message twice; assert exactly one row / one CSV line.
  • Integration tests with Testcontainers: spin up real Kafka + PostgreSQL in the test, publish to intern.created, and assert a row appears in Postgres and a line in the CSV. An end-to-end test like this — publish an event, watch it land in every target — is genuinely impressive coming from an intern and proves the guarantees rather than asserting them.
  • DLQ test: publish a deliberately poison message; assert it lands in intern.dlq and that a following good message still processes (pipeline not blocked).

10. Extensibility ​

New targets are added by deploying a new consumer subscribed to intern.created — with no changes to the UI, source service, or Kafka:

text
Azure SQL Adapter   MongoDB Adapter   REST Adapter   S3 Adapter
XML Adapter         FTP Adapter       Email Adapter  Webhook Adapter

Each is its own consumer group, applies the same idempotency and DLQ patterns, and transforms the canonical record into its target format.

11. Suggested Tech Stack ​

LayerTechnology
UIReact + TypeScript + Tailwind
Source ServiceSpring Boot (Java)
MessagingApache Kafka
Schema contractConfluent Schema Registry (Avro / JSON Schema)
DatabasePostgreSQL (unique constraint on record_id)
File outputCSV (with dedup index)
Dead-letterKafka topic intern.dlq
BuildMaven
ContainerizationDocker Compose
ObservabilitySpring Boot Actuator + Kafka UI (Phase 2: Prometheus + Grafana)
TestingJUnit + Testcontainers (Kafka + Postgres)

12. Scope: MVP vs. Later ​

Realistically, React + Spring Boot + Kafka + Postgres + multiple adapters + Docker Compose is a lot for an internship timebox. Prioritize accordingly.

Must ship (MVP):

  • Source service with validation, canonical model, schema-validated publish, keyed by internId.
  • One topic (intern.created) + a minimal DLQ (intern.dlq).
  • Both adapters, each with correct idempotency (DB upsert, file dedup).
  • Retriable-vs-non-retriable failure handling.
  • A minimal UI (plain validated form — no polish).
  • At least one Testcontainers end-to-end test.

Phase 2 (great "where I'd take it next" talking points — don't build unless there's time):

  • Target Registry so enabled adapters are configured, not hardcoded.
  • Prometheus + Grafana admin dashboard (lag, adapter health, DLQ depth, message counts).
  • Pluggable formatter strategies (CSV / JSON / XML) per adapter.
  • intern.updated / intern.deleted with full audit logging and trace IDs.
  • DLQ replay tooling.

The result is a clean, defensible MVP with obvious, well-reasoned extension points — which is exactly what demonstrates good SOA/EDA judgement in a review.

Note: This document is the original design brief as authored. Scope has since been expanded — see 04-implementation-plan.md — to build the full platform, including everything listed above as "Phase 2," not just the MVP. The MVP/Phase-2 split above is preserved here as historical context for why the architecture is layered the way it is: the MVP is Release 1 of the full plan, not the finish line.