Phased Rollout Plan

System: Intern Integration Platform (IIP) Related docs: Architecture · Use Cases · Data Model · Implementation Plan

This document takes each Release's backlog (Implementation Plan §4) and breaks it into much smaller, sequential, individually-completable phases — small enough that each one is roughly a single sitting's work and a single commit. It's an execution checklist, not an architecture artifact: check items off as they land, and expect the phase breakdown for later releases to be revised once earlier ones are actually built (that's normal XP feedback, not drift).
Only Release 1's phases are meant to be taken literally right now. Releases 2–7 are broken down to the same granularity so the whole rollout is visible up front, but per the project's own "simple design" / YAGNI practice (Implementation Plan §1), don't start building Release 3's schema registry plumbing while Release 1 is still in progress — the phase numbers are a map, not a schedule.

How to read this

Phases are numbered R.P (Release.Phase) and listed in build order within a release.
Each phase names the repo(s) it touches, what to build, and a tiny, concrete done-when check — usually one test or one observable behavior, matching the project's test-first practice.
A phase that says "test-first" means: write the failing test in the same phase, before or alongside the implementation — not as a follow-up phase.
[ ] checkboxes are for tracking progress in this file as you go.

Spring Boot 4 / Jackson 3 gotchas hit so far

Discovered while building source-service (Phases 1.1–1.8), and worth watching for in db-adapter/file-adapter too (Spring Data JPA has had similar module-boundary changes in the same Spring Boot generation):

Gotcha	What actually happened	Fix
Web starter renamed	`spring-boot-starter-web` is now `spring-boot-starter-webmvc`	Use the new artifact id (Spring Initializr will get this right if you regenerate)
Test starters split per-feature	The old monolithic `spring-boot-starter-test` is gone; each feature has its own `-test` starter (`spring-boot-starter-webmvc-test`, `-kafka-test`, `-actuator-test`, `-validation-test`, `-restclient-test`, …)	Add the specific `-test` starter for whatever you're slice-testing; missing one shows up as a compile error or a missing bean, not a clear message
Test annotations moved packages	`@WebMvcTest` → `org.springframework.boot.webmvc.test.autoconfigure`; `TestRestTemplate` → `org.springframework.boot.resttestclient`	Search the actual jar (`unzip -l ... \| grep ClassName.class`) rather than guessing the package — guessed imports fail with "package does not exist," not a helpful redirect
`TestRestTemplate` no longer auto-registered	`@SpringBootTest(webEnvironment = RANDOM_PORT)` alone doesn't give you a `TestRestTemplate` bean anymore	Add `@AutoConfigureTestRestTemplate` and the `spring-boot-starter-restclient-test` dependency (it needs `RestTemplateBuilder`, which nothing else pulls in)
Spring Boot's own default Jackson is now Jackson 3	`tools.jackson.core`, not `com.fasterxml.jackson.core` — but Spring Kafka's `JsonSerializer` still uses legacy Jackson 2.x internally	Add `com.fasterxml.jackson.datatype:jackson-datatype-jsr310` (the 2.x one) explicitly, or `java.time` fields fail to serialize with an `InvalidDefinitionException` about missing modules
Jackson still defaults to `WRITE_DATES_AS_TIMESTAMPS`	Without disabling it, `java.time` fields serialize as numeric arrays/epoch numbers, not the ISO-8601 strings this project's canonical examples show	Build the Kafka value-serializer's `ObjectMapper` with `JavaTimeModule` registered and `WRITE_DATES_AS_TIMESTAMPS` disabled — via `DefaultKafkaProducerFactoryCustomizer` (`org.springframework.boot.kafka.autoconfigure`), not a hand-built `ProducerFactory`, so Spring Boot's own connection-detail handling stays intact
`@ServiceConnection` bypasses `spring.kafka.bootstrap-servers`	Spring Boot wires the Testcontainers address directly into the autoconfiguration; the property itself still just reflects `application.yml`'s fallback	Any raw test Kafka client needs its address from the `KafkaContainer` bean directly (`kafkaContainer.getBootstrapServers()`), never from `@Value` on that property
Initializr's `bootVersion` id has a `.RELEASE` suffix that isn't a real artifact version	`4.1.0.RELEASE` doesn't exist on Maven Central; the actual version is `4.1.0`	Strip the suffix from the generated `pom.xml`'s parent version before the first build
`JsonDeserializer` trusts the producer's `__TypeId__` header by default	A cross-service consumer (`db-adapter`) can't resolve a header naming the producer's (`source-service`'s) Java class, which isn't even on its classpath	`.ignoreTypeHeaders()` and always deserialize to the consumer's own local DTO — also the architecturally correct choice, since the JSON contract crossing the boundary shouldn't imply a shared Java type
`@Modifying @Query` methods aren't auto-transactional	Unlike `save()` (transactional via `SimpleJpaRepository`), a custom native update/insert query throws `TransactionRequiredException` without one	Add `@Transactional` on the repository method itself

The common thread: every one of these was invisible until something actually ran — compiling cleanly, and even a mocked unit test passing, told us nothing. Real Testcontainers integration tests and live docker compose checks caught all of them.

Release 1 — Walking Skeleton

Goal: docker compose up, submit one intern through the real UI, watch it land in Postgres and the CSV, with basic reliability (idempotency, isolated failures, a working DLQ) already true — not bolted on later.

Source Service

1.1 — Canonical record type. source-service: add a CanonicalInternRecord (Java record) with the fields from Data Model §1 — no logic, just the shape. Done when: it compiles and a trivial construction test passes.
1.2 — Intake DTO + validation. source-service: CreateInternRequest DTO with Bean Validation annotations (@NotBlank, @Email, @NotNull on startDate, etc.). Done when: a validator unit test shows a request missing email fails validation and a fully-populated one passes.
1.3 — Canonical mapper. source-service: CanonicalMapper mapping CreateInternRequest → CanonicalInternRecord, generating recordId (UUID) and createdAt server-side. Done when: a unit test asserts two calls with identical input produce different recordIds and both createdAts are non-null.
1.4 — Kafka topics as code. source-service: NewTopic beans for intern.created (3 partitions) and intern.dlq (1 partition), matching Architecture §10 — makes topic creation part of the app, not just infra's manual init job. Done when: app starts against a real broker and the topics exist with the right partition counts (verify via kafka-topics.sh --describe, same check already used against infra).
1.5 — Kafka publisher. source-service: KafkaEventPublisher wrapping KafkaTemplate<String, CanonicalInternRecord> (JSON), publishing keyed by internId. Done when: a unit test with a mocked KafkaTemplate verifies the key and topic passed to send(...).
1.6 — POST /interns. source-service: InternController wiring @Valid request → mapper → publisher, returning 202 Accepted with recordId; 400 with field errors on invalid input. Done when: a @WebMvcTest/slice test covers both the happy path and a validation-failure path.
1.7 — Publish integration test. source-service: Testcontainers-Kafka test hitting POST /interns (real HTTP call, real embedded broker) and asserting exactly one message on intern.created with the correct key. Done when: that test passes and would fail if the key were wrong (mutate it locally to confirm the test actually catches it, then revert).
1.8 — GET /interns read path. source-service: simplest correct read model owned by the Source Service itself (e.g. an in-memory or JPA-backed store populated at publish time) — not a query against Postgres/the CSV, preserving the loose-coupling boundary (Architecture §4.2). Done when: submitting via POST and then calling GET /interns shows the new record.

Reviewed 2026-07-22 (Phases 1.1–1.8): all eight done-when checks re-verified — 23/23 tests green, the actual Docker image (not just mvn spring-boot:run) rebuilt and exercised against a fresh Kafka via docker compose, POST/GET round-tripped correctly, and the message read back off the real topic. One genuine Definition of Done gap found: item 5 (recordId/traceId in logs at every new step) isn't met yet — there's no application-level logging at all in source-service so far. Not blocking (full traceId propagation is explicitly Release 4 / Phase 4.7 work), but plain recordId logging could reasonably start now rather than waiting; left as an open gap rather than silently fixed, since it wasn't part of any phase's stated done-when criteria. Two doc inaccuracies fixed as part of this review: Architecture §5's Source Service diagram called the read model InternQueryRepository (built as InternRecordStore) and didn't show ApiExceptionHandler; UC-1 step 5 stated Schema Registry validation as already true (it's Release 3+, not yet built).

UI

1.9 — Intern form. ui: replace the walking-skeleton placeholder with a real form (fields matching the canonical model) with client-side validation, calling POST /interns via VITE_API_BASE_URL. Done when: submitting a valid form in the browser gets a 202 and shows a confirmation; an invalid one shows inline errors without calling the API.
1.10 — Records view. ui: a list view calling GET /interns, rendering submitted records. Done when: a freshly submitted record appears in the list after a refresh.

Verified 2026-07-22 (Phases 1.9–1.10): both done-when checks confirmed against the real running stack (Kafka via infra docker-compose, source-service on localhost:8080, ui dev server) with a Playwright browser session, not just a build check. An empty-form submit produced 7 inline client-side errors and no API call; a valid submit returned 202 with a confirmation banner and the record appeared in the list without a manual refresh; light/dark themes (Catppuccin Latte/Mocha) and layout responsiveness at desktop (1400px), tablet (820px), and mobile (390px) widths were all checked, including the dark-mode preference persisting across a reload. One real bug found and fixed along the way: Spring Boot doesn't enable CORS by default, so the browser silently discarded the API's response even though the server processed it (confirmed via curl — 202 with no Access-Control-Allow-Origin header); fixed with a WebMvcConfigurer (WebConfig) in source-service scoped to /interns/**, with allowed origins externalized via iip.cors.allowed-origins. All 23 source-service tests re-run green after the fix.

Database Adapter

1.11 — JPA entity. db-adapter: Intern entity mapped to the existing interns table (schema in the infra repo's postgres/init.sql) + a Spring Data repository. Done when: the app starts against Postgres with ddl-auto: validate passing (no mismatch).
1.12 — Kafka consumer (naive). db-adapter: @KafkaListener on intern.created, JSON-deserialize, map to entity, save(). Done when: publishing one message via the CLI or a test results in one row.
1.13 — Make it idempotent. db-adapter: replace save() with a native INSERT ... ON CONFLICT (record_id) DO NOTHING (@Modifying @Query). Test-first: write the "deliver the same message twice, assert exactly one row" Testcontainers test before making it pass. Done when: that test passes.

Verified 2026-07-22 (Phases 1.11–1.13): all three done-when checks confirmed via real Testcontainers Postgres + Kafka, not mocks. 1.11: init.sql (mirrored from infra since this repo can't reference a sibling repo's file) applied to a real Postgres, ddl-auto: validate passed, entity save/find round-tripped correctly including a null mentor. 1.12: a raw JSON message published directly to intern.created (not via any shared Java type -- db-adapter must never depend on source-service's classes, only the canonical JSON contract) was consumed, deserialized, and landed as one row. 1.13, genuinely test-first: first confirmed what the naive save() from 1.12 actually did under duplicate delivery -- it already avoided a second row, but only incidentally (a non-null @Id makes Spring Data treat the entity as an update, not an insert, which isn't the same as a guaranteed atomic constraint) -- then replaced it with the explicit native ON CONFLICT (record_id) DO NOTHING upsert the phase calls for, confirmed via a real double-publish test. One gotcha hit and fixed along the way: @Modifying @Query methods aren't automatically transactional the way save() is via SimpleJpaRepository -- needed an explicit @Transactional on the repository method, or every call failed with TransactionRequiredException.

File Adapter

1.14 — Dedup store + CSV writer. file-adapter: a local dedup store (recording processed recordIds — an embedded file/SQLite is fine for R1) and a CsvInternWriter appending lines matching Data Model §4.2. Done when: a unit test writes two different records and asserts two distinct CSV lines.
1.15 — Kafka consumer + idempotent append. file-adapter: @KafkaListener on intern.created: check the dedup store, skip-or-append-then-record. Test-first: "deliver the same message twice, assert exactly one CSV line." Done when: that test passes.

Reviewed 2026-07-22 (Phases 1.1–1.15, full re-check): every phase's done-when re-verified, not just re-read. Full suites re-run clean: source-service 23/23, db-adapter 5/5, file-adapter 10/10; ui build + lint clean. Then a genuine end-to-end pass against the real Docker images (docker compose up --build, not mvn spring-boot:run) — submitted one intern both via raw HTTP and via a real browser session against the UI, and confirmed the identical recordId landed correctly in all three places: GET /interns (Source Service's own read path), a Postgres row, and a CSV line, with matching data in each. Two real doc inaccuracies found and fixed during this pass: Architecture's tech stack table claimed the file adapter's dedup index was "RocksDB/SQLite," but what's actually implemented (deliberately) is a flat file — file adapter is single-writer (AD-6), so there's no concurrent-writer race an embedded database would be needed for; and UC-5 claimed Schema Registry validation in its main flow, the same Release-1-vs-Release-3+ gap already caught and fixed for UC-1 in the earlier 1.1–1.8 review, just missed here since db-adapter didn't exist yet at that time.

Verified 2026-07-22 (Phases 1.14–1.15): all done-when checks confirmed. 1.14: plain unit tests (no Testcontainers needed) covered the literal done-when plus two edge cases worth locking in — a college name containing a comma is RFC4180-quoted rather than corrupting the column count, and reopening an existing CSV file doesn't duplicate the header. DedupStore is a flat file (one recordId per line), not an embedded database — file-adapter runs as exactly one instance (Architecture AD-6), so there's no concurrent-writer race a heavier store would be needed for. 1.15: a real Testcontainers Kafka integration test published one message (one CSV line) and then the same message twice (still one line). One real bug caught during this phase, before it reached a commit: the bare context-load test (FileAdapterApplicationTests) had no path override, so CsvInternWriter/DedupStore beans fell back to application.yml's default ./data/... paths and wrote a real file into the repo's working directory on every test run — caught when data/interns.csv showed up as an untracked file about to be committed. Fixed by isolating that test with a @TempDir (matching the pattern already used in the Kafka consumer test) and adding data/ to .gitignore.

Reliability (still Release 1 — see Implementation Plan §4, not deferred)

1.16 — Failure isolation test. db-adapter + file-adapter: Testcontainers test that stops Postgres mid-stream and asserts the File Adapter still succeeds independently while the DB Adapter's offset stays uncommitted (message not lost), then recovers once Postgres comes back. Done when: that test passes.
1.17 — Shared FailureClassifier. db-adapter + file-adapter: a small classifier distinguishing retriable vs. non-retriable exceptions, used identically in both adapters (Architecture §6). Done when: a table-driven unit test covers at least "connection refused → retriable" and "deserialization error → non-retriable."
1.18 — Bounded retry. db-adapter + file-adapter: wrap the write in the classifier + a bounded retry with backoff (config via application.yml, not a hardcoded constant). Done when: a test asserts the retry count and that it gives up after the configured max.
1.19 — DLQ on exhaustion/non-retriable. db-adapter + file-adapter: on non-retriable failure or retry exhaustion, publish the original message + error metadata (Data Model §3) to intern.dlq, then commit the offset. Test-first: "publish a malformed message, assert it lands in intern.dlq and a subsequent good message still processes." Done when: that test passes for both adapters.

Verified 2026-07-22 (Phases 1.16–1.19): implemented in dependency order (1.17 → 1.18 → 1.19 → 1.16) since each builds on the last, then committed in the phase numbering order shown above. db-adapter: 18/18 tests green (was 10). file-adapter: 19/19 tests green (was 10). Both adapters now deserialize the Kafka message by hand (ObjectMapper.readValue inside the listener) instead of via a Kafka JsonDeserializer — this was a deliberate design change from 1.12/1.15, not just an addition: it's what lets a malformed payload be classified and routed to the DLQ by the same code path as a write failure, and it removed KafkaConsumerConfig's wildcard-capture workaround entirely in both adapters. FailureClassifier is genuinely two different implementations sharing one shape, not one shared library — db-adapter's retriable set is ConnectException/TransientDataAccessException, file-adapter's is ConnectException/generic IOException (the realistic transient-failure analogue for a local-disk target), each adapter-appropriate. One real bug caught by the classifier's own test: file-adapter's first draft checked the retriable branch before the non-retriable branch, and since Jackson's JsonProcessingException is itself an IOException subtype, the broad IOException retriable check silently shadowed the deserialization-error case — the deserialization-error test failed, not the connection-refused one, which is what caught it. 1.16 needed two attempts: stopping and restarting the same Testcontainers Postgres instance left connections failing for 60+ seconds after start() returned (container restart isn't a reliable simulate-recovery technique in practice); switched to Docker pause/unpause, which freezes the container without tearing down its network binding, and that worked cleanly. file-adapter doesn't have its own 1.16 test — "the other target is unaffected" is a structural guarantee of Kafka consumer groups (separate groups on the same topic never block each other), already evidenced by file-adapter's suite never touching Postgres at all; inventing a test that touches Postgres from file-adapter's repo would have been testing Kafka's architecture, not this codebase.

Proof & demo

1.20 — Full end-to-end test. One Testcontainers test: publish once (via the real HTTP path), assert a row appears in Postgres and a line appears in the CSV — the proof test called out in Original Specification §9.
1.21 — Manual demo pass. infra: docker compose up --build, submit a real intern through the browser UI, confirm it in Postgres (psql) and the CSV file. Update infra/README.md with this as a documented smoke-test recipe.

Release 2 — Reliability Hardening

2.1 — Formalize the classifier. Extract FailureClassifier (from 1.17) into a shared, explicitly-documented exception → classification table; expand the unit test table with more exception types as they're discovered in practice.
2.2 — Chaos test: Postgres. Formalize 1.16 into a repeatable chaos suite: kill/restart Postgres via Testcontainers mid-consumption, assert eventual consistency with no loss/duplication.
2.3 — Chaos test: Kafka reachability. Same for a Kafka broker disruption affecting the File Adapter.
2.4 — DLQ envelope completeness check. Audit the DLQ payload against the full schema in Data Model §3 (attemptCount, failedAdapter, timestamps, etc.) and fill any gaps left from 1.19.
2.5 — Retry/backoff externalized per-adapter. Confirm max attempts and backoff are independently tunable per adapter via config, not shared hardcoded values.

Release 3 — Contract Enforcement

3.1 — Schema Registry in infra. infra: add Confluent/compatible Schema Registry to docker-compose.yml.
3.2 — Canonical schema as a versioned artifact. Define the canonical schema (Avro or JSON Schema) matching Data Model §1 as a file checked into each service that needs it (or a small shared schema package).
3.3 — Schema-validating producer. source-service: switch to a schema-validating serializer. Test: a deliberately non-conforming record fails to publish (fails fast, before Kafka).
3.4 — Schema-validating consumers. db-adapter + file-adapter: switch to a schema-validating deserializer. Test: a schema-invalid message is rejected at deserialization, not deep in business logic.
3.5 — CI compatibility gate. A script/check that runs the registry's compatibility check; test case using a deliberately-breaking schema change to prove the gate actually blocks it.
3.6 — Backward-compatible field addition, proven. Add one new optional field, redeploy only source-service, confirm the un-redeployed db-adapter/file-adapter still process new messages correctly.

Release 4 — Observability

4.1 — Metrics endpoint. All three Spring services: add micrometer-registry-prometheus, expose /actuator/prometheus.
4.2 — Prometheus in infra. infra: add Prometheus scraping all three services.
4.3 — Grafana in infra. infra: add Grafana with a provisioned Prometheus datasource.
4.4 — Lag panel. Grafana dashboard panel: consumer lag per adapter over time.
4.5 — DLQ depth panel. Grafana dashboard panel: DLQ depth over time.
4.6 — Throughput/error panel. Grafana dashboard panel: per-adapter success/failure counts and throughput.
4.7 — traceId propagation. Generate a traceId at Source Service intake, carry it in the canonical record and every log line at every hop (structured logging config in all three services). Done when: a documented "search all logs for one traceId" walkthrough actually reconstructs one submission's full journey.

Release 5 — Mutable Lifecycle

Release 6 — Extensibility

6.1 — Target Registry config shape. Define how adapter enablement is configured (start simple — env-driven flags are fine — before reaching for anything heavier).
6.2 — Extract the generic adapter pattern. Refactor db-adapter and file-adapter to visibly share the same shape (idempotency gate, classifier, retry/DLQ) — this is the refactor-after-duplication moment named in Implementation Plan §1, done once, not designed upfront.
6.3 — Pluggable formatter strategy. file-adapter: extract a CSV/JSON/XML formatter interface, selected via config.
6.4 — A genuinely new adapter. Build a second real target (e.g. a webhook adapter) as its own repo, reusing the pattern from 6.2, with zero changes to ui/source-service/existing adapters — the concrete proof of UC-9.
6.5 — Shared adapter test harness. A base test class/suite ("every adapter must pass idempotency + DLQ tests") applied to all adapters, including the new one.

Release 7 — Operability

7.1 — DLQ read API. A non-destructive read of intern.dlq, grouped by error type/adapter.
7.2 — Admin dashboard: DLQ viewer. UI page rendering 7.1's data.
7.3 — DLQ replay tool. Re-publish a selected message to its source topic; mark the DLQ entry replayed (audit trail, not deleted).
7.4 — Admin dashboard: operational overview. Aggregate health/lag/DLQ depth (Release 4's data) into one view.

Keeping this document honest

When a phase's actual implementation reveals the next one needs re-splitting or reordering, edit this file — it's a working checklist, not a frozen spec.
Every phase should map to one commit (or a small handful) in the relevant repo. If a phase is producing multi-day, multi-file sprawl, it wasn't tiny enough — split it before continuing, per the project's small-releases practice.
Re-check Implementation Plan §6 Definition of Done at each phase, not just at the end of a release.

Phased Rollout Plan ​

How to read this ​

Spring Boot 4 / Jackson 3 gotchas hit so far ​

Release 1 — Walking Skeleton ​

Source Service ​

UI ​

Database Adapter ​

File Adapter ​

Reliability (still Release 1 — see Implementation Plan §4, not deferred) ​

Proof & demo ​

Release 2 — Reliability Hardening ​

Release 3 — Contract Enforcement ​

Release 4 — Observability ​

Release 5 — Mutable Lifecycle ​

Release 6 — Extensibility ​

Release 7 — Operability ​

Keeping this document honest ​