Appearance
Implementation Plan
System: Integration Platform (IIP) Methodology: Extreme Programming (XP) Related docs: Architecture · Use Cases · Data Model · Generalization Strategy
Scope statement: This plan builds the full platform described in the architecture document — not just the MVP. What the original design brief labeled "Phase 2" is not an optional stretch goal here; it has a home in the releases below. There are deliberately no dates or hour estimates in this document — XP replaces fixed-date planning with small, working, continuously-integrated releases ordered by value and by what each subsequent release depends on.
Roadmap re-sequenced (see 06 — Generalization Strategy). Releases 3 onward were re-ordered to insert the generalization track immediately after the current line, rather than after the intern pipeline is feature-complete. Nothing is dropped: the former Schema Registry release (R3) and Target Registry release (R6) are folded into the contract-registry work, and the former observability (R4) and operability (R7) releases move to R8. Releases 1–2 are built and unchanged.
1. Why XP Fits This Project
This is a solo project — one developer, no team. XP's team-oriented rituals (pairing, collective ownership, planning-game negotiation between separate customer/developer roles) don't disappear here; they're adapted to a single-developer context rather than dropped. The practices that don't depend on headcount (small releases, test-first, continuous integration, refactoring, simple design, sustainable pace) apply unchanged.
| XP practice | How it applies here |
|---|---|
| Small releases | Each release below is a working, deployable increment of the real system — not a demo throwaway. Release 1 is a genuine walking skeleton: UI → Source Service → Kafka → both adapters → both targets, fully working end-to-end, before anything else is added. |
| Planning game | Solo-adapted: you play both roles. The backlog (§4) is self-prioritized by pulling the next-highest-value story off the top, not by a pre-committed schedule — there's no separate customer to negotiate scope with, so the architecture's guarantees (§14 NFR table in Architecture) act as the acceptance authority. |
| Test-first development | Every story's Definition of Done (§6) requires tests written alongside (or before) the implementation — this is especially non-negotiable for idempotency and DLQ behavior, since those guarantees are invisible unless tested. Solo-adapted: with no pairing partner, the test is the second pair of eyes. |
| Continuous integration | Every merge to main triggers the full build + test suite (including Testcontainers integration tests) — the "walking skeleton" must never be allowed to stop walking. Matters more solo, not less: there's no teammate to notice a broken build by accident. |
| Refactoring | The generic adapter pattern (Architecture §6) is expected to be discovered by refactoring the second adapter to look like the first, not designed upfront in the abstract. |
| Simple design | Build the two concrete adapters (DB, File) before the adapter-attachment abstraction (Release 6) — the abstraction is extracted once real duplication exists, not speculated in advance (see Architecture AD-8). The same rule is why the contract is extracted to a file in Release 3 before it becomes a registry service in Release 4: prove parameterization with the cheapest possible mechanism first. |
| Structural consistency (solo-adapted "collective ownership") | With one developer, ownership isn't the issue — context-switching is. The generic adapter pattern and canonical schema mean any adapter looks like any other, so picking a story back up after weeks away costs re-reading one shape, not one shape per adapter. |
| Sustainable pace / no dates | Releases are ordered by dependency and value, not scheduled against a calendar — this document intentionally contains no dates or hour estimates. Solo, this is a guardrail against burnout as much as a planning technique. |
| Metaphor | "Every intern record is a single event flowing down a shared pipe; each target adapter is a tap on that pipe that can be added, removed, or temporarily closed without affecting the others." |
2. Release Roadmap (Dependency Order, Not a Calendar)
Releases 1–2 (green) are built. Release 6 is where the user-facing goal is met — define a schema and attach a target from a UI, no redeploy. Releases are shown linear for dependency clarity; in practice R7 and R8 can interleave with each other once R6 lands, since neither strictly depends on the other — worth reconsidering at each planning-game pass rather than treating the chain as rigid.
2.1 Why the generalization track goes here, not after R7
The intern pipeline is not a dependency of the platform work — it is the first instance of it. The former R3–R7 added capability to one schema (lifecycle, dashboards, replay); the generalization adds capability across schemas. Orthogonal axes, so ordering is a choice rather than a constraint.
The trap the re-sequencing avoids is complete-then-generalize. Every intern-specific thing hand-built first — a compiled DTO, a typed mapper, an intern.updated handler — is code that would be re-derived generically later: more to build, and then more to tear out. Finishing the single-schema roadmap first would also bake "schema is fixed" into five more releases' worth of code, making the eventual generalization more expensive, not less.
The rule: generalize at the current line. From end-of-R2, the cheapest possible proof of parameterization is available immediately — make the existing intern pod a config-driven instance rather than a hardcoded one (R3). That is why R3 is contract extraction and not, say, the control-plane UI: it's the smallest step that commits to neither Path A nor Path B while making both cheap.
The counter-argument, recorded rather than buried. If the objective were a polished portfolio piece, a deeply-featured single pipeline (lifecycle, dashboards, DLQ replay on interns) might demonstrate more than a half-built platform-of-platforms. That trade is real. This plan takes the platform path deliberately — but if "impressive and done" ever beats "ambitious and in-progress," the correct move is to invert this ordering and finish the pod, not to split the difference.
3. Release Overview Table
| Release | Theme | Key capabilities | Primary use cases | Depends on |
|---|---|---|---|---|
| 1 ✅ | Walking Skeleton (MVP) | UI, Source Service, intern.created, minimal DLQ, DB adapter (upsert), File adapter (dedup-append), basic Actuator health | UC-1, UC-2, UC-5, UC-6, UC-7, UC-8, UC-10 (basic) | — |
| 2 ✅ | Reliability Hardening | Formalized retry policy w/ backoff, failure classification taxonomy, chaos/failure-injection tests, idempotency test suite | UC-7, UC-8 | Release 1 |
| 3 | Contract Extraction + Envelope Split | The envelope wraps the existing record; the intern schema moves out of compiled code into a contract file the source-service reads; interns becomes contract #1; topics renamed to the derived scheme | UC-1 (re-framed) | Release 1–2 |
| 4 | Parameterized Source Service + Contract Registry | Source-service loads any contract by id; Contract Registry becomes a real service with an API; envelope registered and enforced in the Schema Registry; compatibility CI gate. (Absorbs the former "Contract Enforcement" release.) | UC-1, UC-13 (API-level) | Release 3 |
| 5 | Config-Driven DB Adapter + Generic Landing Table | The hybrid records table; upsert on the declared natural key; adapter filters by contractId and resolves mapping from the registry. Exit criterion: forms lands end-to-end as contract #2. | UC-5 | Release 4 |
| 6 | Adapter Catalog + Attachments + Control-Plane UI | csv and webhook as catalog types; attach-adapter and define-contract as UI-driven registry writes. (Absorbs the former "Target Registry" release.) This is where the user-facing goal is met. | UC-9, UC-12, UC-13, UC-14 | Release 5 (pattern proven twice before abstracting) |
| 7 | Generalized Lifecycle | Update-style and tombstone record types declared per contract; Edit/Delete UI + endpoints; natural-key upsert; CSV snapshot-rebuild; audit log | UC-3, UC-4 | Release 4 (contract evolution needed) |
| 8 | Observability + Operability | Prometheus scraping, Grafana dashboards (lag, DLQ depth, throughput) broken down by contractId, traceId propagation, DLQ inspection + replay tooling, admin dashboard | UC-10, UC-11 | Release 2 (DLQ maturity), Release 6 |
| 9 (optional) | Path A: IIPInstance CRD + operator | Per-contract pod isolation via a Kubernetes operator, reusing the same parameterized images | UC-15 | Release 6 — and a real hard-isolation requirement (AD-12) |
4. Backlog by Release
Each story follows the standard XP story-card shape: a plain statement of value, plus acceptance criteria. With no separate customer role, the acceptance criteria are checked against the architecture's own guarantees (Architecture §14) rather than a stakeholder sign-off. Sizing is intentionally not included — stories are pulled by value and risk, not by estimated effort, and no dates are attached per the project's XP approach.
Release 1 — Walking Skeleton (MVP)
| Story | Acceptance criteria |
|---|---|
| As HR Staff, I can submit a new intern record via a form | Invalid input is rejected with field-level errors; valid input returns a confirmation with a recordId |
| As HR Staff, I can view previously submitted records | List reflects submissions via the Source Service's own read path, not by querying adapter targets |
As the platform, every valid submission becomes exactly one canonical event on intern.created, keyed by internId | Verified with a Testcontainers test: publish once, assert the message exists with the correct key |
| As the Database Adapter, I persist every event to PostgreSQL without creating duplicates on redelivery | Idempotency test: deliver the same message twice, assert exactly one row |
As the File Adapter, I append every event to interns.csv without creating duplicate lines on redelivery | Idempotency test: deliver the same message twice, assert exactly one CSV line |
| As the platform, a target being down does not affect the other target | Integration test: stop Postgres, assert File adapter still succeeds and DB adapter's offset stays uncommitted |
| As the platform, a poison message is quarantined instead of blocking the pipeline | DLQ test: publish a malformed message, assert it lands in intern.dlq and a subsequent good message still processes |
| As a Platform Operator, I can check basic health of every service | /health and /metrics respond correctly on each service via Spring Boot Actuator |
| As the developer, I can run the whole flow locally with one command | docker-compose up brings up UI, Source Service, Kafka, both adapters, Postgres, and produces a working end-to-end demo |
Definition of Done for Release 1: see §6 — this release additionally requires at least one full Testcontainers end-to-end test (publish → both targets updated) as an explicit exit criterion, since it's the proof the architecture's core promise holds.
Release 2 — Reliability Hardening
| Story | Acceptance criteria |
|---|---|
| As the platform, transient vs. permanent failures are classified consistently across adapters | A shared FailureClassifier component/interface is used by both adapters; unit-tested against a table of exception types |
| As the platform, retries are bounded with backoff, not immediate infinite loops | Configurable max attempts + backoff; test asserts retry count and timing behavior |
| As the platform, I have automated chaos/failure-injection tests | Tests that kill/restart the target mid-stream (Testcontainers) and assert eventual consistency with no loss/duplication |
| As a developer, the DLQ envelope carries enough metadata to diagnose the failure without re-deriving it | DLQ message includes original topic/partition/offset/key, error type, error message, failed adapter, attempt count, timestamp |
Release 3 — Contract Extraction + Envelope Split
| Story | Acceptance criteria |
|---|---|
| As the platform, every record on the wire carries a fixed envelope around a per-contract payload | The existing intern fields move under payload; recordId/contractId/recordType/schemaVersion/naturalKey/occurredAt are present on every message (Data Model §1a); all Release 1–2 tests still pass after the shape change |
| As the platform, the intern schema is data the source-service reads, not code it compiles | A contract file (Data Model §1c) declares fields, types, required-ness, enum domain, and key strategy; the compiled CanonicalInternRecord/CanonicalMapper pair is deleted, not merely bypassed |
| As the platform, validation and natural-key derivation are driven by the loaded contract | Removing a required field from the contract file makes a previously-valid submission fail with a 400 — without recompiling |
| As the platform, the intern pod behaves exactly as it did before | The full Release 1–2 suite (source-service, both adapters, e2e) is green with no test weakened to accommodate the change. ← this is what "parameterization proven" means |
| As the platform, a second contract needs no new code | A forms contract file validates and publishes a forms record end-to-end, with zero Java changes beyond adding the file |
Release 4 — Parameterized Source Service + Contract Registry
| Story | Acceptance criteria |
|---|---|
| As the platform, the source-service loads any contract by id at runtime | One running instance accepts submissions for interns and forms concurrently, validating each against its own definition |
| As the platform, contracts live in a registry service with an API | POST /contracts / GET /contracts/{id} back the definitions from Data Model §1c; the source-service reads from it instead of a baked-in file |
| As the platform, the canonical envelope is registered and versioned in the Schema Registry | Source Service publish fails fast (before hitting Kafka) if the envelope doesn't conform; adapters reject a non-conforming envelope at deserialization, not deep in business logic |
| As the developer, incompatible contract changes are caught before they go live | The control-plane API runs a per-contractId BACKWARD compatibility check before accepting a write; CI runs the same check over every registered contract; a deliberately-breaking change is used as a test case |
| As a developer, I can add a new optional field without breaking existing consumers | Add a field to a contract, restart nothing, assert existing adapters still process new messages correctly |
Release 5 — Config-Driven DB Adapter + Generic Landing Table
| Story | Acceptance criteria |
|---|---|
As the postgres adapter, I write any contract's payload without a redeploy | The hybrid records table (Data Model §4.0) accepts a forms record and an intern record, with record_id and natural_key typed and indexed |
As the postgres adapter, I only write contracts attached to me | An envelope whose contractId has no attachment is skipped and its offset committed; a test asserts no row is written and the partition keeps moving |
| As the platform, idempotency is unchanged by generalization | "Deliver twice, assert one row" passes for a contract the adapter has never seen before — no per-contract test scaffolding |
| As the platform, forms lands end-to-end as contract #2 | Release exit criterion: submit a forms record through the real HTTP path, assert it lands in Postgres, with no code written for forms specifically |
| As a contract owner, I can still have a fully-typed table when I want one | Shaped-table mode is selectable per attachment; the existing interns table keeps working under it, proving the default is a default and not a mandate |
Release 6 — Adapter Catalog + Attachments + Control-Plane UI
| Story | Acceptance criteria |
|---|---|
| As an Integration Designer, I can define a contract from a UI | UC-13 end-to-end: fill the form, submit, and a record conforming to the new contract is accepted — with no redeploy of any service |
| As an Integration Designer, I can attach an adapter to a contract from a UI | UC-14 end-to-end: pick a type, enter target config, and the target starts receiving that contract's records; existing contracts/adapters are provably unaffected |
| As a Platform Operator, I can enable/disable a specific fan-out | An attachment's enabled flag takes effect without modifying core service code; disabling one contract's CSV feed leaves another contract's untouched |
| As a developer, I can add a new adapter type to the catalog | A genuinely new type (webhook) is built and deployed with zero changes to UI/Source Service/existing adapters — the concrete proof of UC-9, not just a diagram — and is immediately attachable to existing contracts |
As the csv adapter, my file path and columns come from the attachment | Two contracts attached to one csv adapter instance write two different files with two different column sets |
| As a developer, each target can use a different serialization format | Formatter strategy (CSV/JSON/XML) is pluggable per attachment, selected via config |
Release 7 — Generalized Lifecycle
| Story | Acceptance criteria |
|---|---|
| As a Record Submitter, I can edit an existing record | PUT /contracts/{id}/records/{naturalKey} publishes an update-style record type; the postgres adapter upserts on the contract's declared natural key; the file target reflects the change |
| As a Record Submitter, I can delete a record | Publishes a tombstone; row removed/soft-deleted; the file target's snapshot omits the record |
| As the platform, lifecycle handling is declared, not hardcoded | A contract that declares no update-style record type rejects an edit with 409; a contract that declares one works with no code specific to it |
| As the platform, events for one entity are always applied in the order issued | Ordering test relying on the naturalKey partition key: interleave create/update/delete rapidly, assert final state is correct — run against two contracts, including one with a composite key |
| As the platform, the CSV reflects current state, not a raw append log | The file adapter rebuilds from its keyed state store on every event; test asserts an edited/deleted record is not duplicated/stale |
| As a Platform Operator, every lifecycle event is captured in an audit trail | An audit row per event, independent of the current-state table |
Release 8 — Observability + Operability
| Story | Acceptance criteria |
|---|---|
| As a Platform Operator, I can see consumer lag per adapter over time | Grafana panel sourced from Prometheus scraping Kafka/Actuator metrics |
| As a Platform Operator, I can see DLQ depth over time | Grafana panel; alerts (or at least a visual threshold) on sustained non-zero depth |
| As a Platform Operator, I can tell which contract is causing a backlog | Lag, throughput, and DLQ-depth panels break down by contractId — one dashboard for the platform, not one per schema |
| As a Platform Operator, I can see per-adapter success/failure counts and throughput | Grafana panel; counters exposed via Actuator custom metrics |
| As a developer, every log line for a given submission can be correlated end-to-end | traceId is generated at submission and propagated/logged at every hop alongside recordId and contractId; a log-search-by-traceId walkthrough is documented |
| As a Platform Operator, I can view DLQ contents grouped by contract/error type in a UI | Admin dashboard reads the DLQ non-destructively and renders it |
| As a Platform Operator, I can replay a fixed DLQ message back into the pipeline | Replay tool re-publishes to the topic derived from the entry's own envelope; the original DLQ entry is marked replayed (not deleted) for audit |
| As a Platform Operator, I can see an operational overview in one place | Dashboard aggregates health/lag/DLQ depth into one view |
Release 9 — Path A: Per-Contract Instances (optional, gated)
Not planned work. Recorded so the option stays cheap, and explicitly gated on hard isolation becoming a stated requirement rather than a hypothetical one (AD-12).
| Story | Acceptance criteria |
|---|---|
| As a Platform Operator, I can run a contract in its own isolated pod set | An IIPInstance CR is reconciled into topics, a configured source-service pod, and attached adapter pods — using the same images Path B runs, proving the parameterization bet paid off |
5. Cross-Release Engineering Practices
These apply from Release 1 onward, not as later additions:
| Practice | Application |
|---|---|
| Test-Driven Development | Adapter transform logic (canonical → SQL / canonical → CSV) and the FailureClassifier are natural TDD candidates — pure functions with clear input/output tables. |
| Testcontainers over mocks for integration tests | Kafka and Postgres are spun up for real in integration tests from Release 1 onward — this is what makes the reliability guarantees provable rather than asserted (see Original Specification §9). |
| Deliberate self-review on cross-cutting components | The generic adapter pattern, FailureClassifier, and canonical schema changes affect every consumer. Solo-adapted pairing: step away and re-read the diff cold (or read it out loud) before merging anything that touches these — the goal a pair would serve, catching contract breaks early, still has to happen, just via a second pass instead of a second person. |
| Continuous Integration | Every push runs unit tests, integration tests (Testcontainers), and — from Release 4 — the envelope schema compatibility check and a per-contractId compatibility check over every registered contract. A red build blocks merge. |
| Refactor before extending | Before Release 6 extracts the adapter catalog / attachment abstraction, refactor the DB and File adapters (built independently in Release 1) to visibly share the same shape — the abstraction should describe two real implementations, not precede them. |
| YAGNI on Phase-2-flavored complexity | Even though the full system is the goal, still resist building Release 6/8 machinery (control-plane UI, replay tooling) early — sequence still matters even without dates attached to it. In particular: do not build a Kubernetes operator (Release 9) to make multi-schema work. That's the whole content of AD-12. |
| Interns is a contract, not a special case | From Release 3 onward, any code path that names an intern is a bug in waiting. If a change would be hard to make work for a second contract, that's the signal to stop and generalize it now rather than to add the second branch. |
6. Definition of Done (applies to every story, every release)
A story is not done until:
- Unit tests cover the new/changed logic (validation, mapping, classification).
- Where the story touches Kafka and/or a target system, an integration test using Testcontainers proves the behavior against real infrastructure.
- If the story affects idempotency-sensitive code, an explicit "deliver twice, assert once" test exists.
- If the story affects the canonical envelope, a Schema Registry compatibility check passes in CI (Release 4+).
- If the story changes a contract, the same compatibility check passes for that
contractIdbefore it goes live — enforced at the control-plane API and re-run in CI over every registered contract. Contracts left the repository; the gate did not follow them out (Release 4+). - Logging includes
recordId/contractId/traceIdat every new processing step the story introduces. - The service builds and runs via
docker-compose upalongside all other services with no manual steps. - Actuator
/healthreflects the new component's status if it can fail independently (Release 1+). - Documentation impact is checked: does this story change the architecture, a use case, or the data model enough that the relevant doc needs a diff? (Living docs, not a one-time artifact.)
7. Risk Register
| Risk | Impact | Mitigation |
|---|---|---|
File adapter scaled to multiple instances, corrupting interns.csv | Data corruption on the file target | Single-writer constraint enforced architecturally (documented in Architecture AD-6); CI/deploy config pins File Adapter to one replica |
| Schema changes break already-deployed adapters | Consumer crashes or silently mis-processes data | Schema Registry compatibility gate in CI from Release 4 onward; BACKWARD compatibility mode enforced for the envelope, and per-contractId for payloads |
| Generic landing table hides schema errors | A malformed payload is happily stored as JSONB, and the mistake surfaces months later as a query returning nothing | Validate the payload against its contract at the source, before publish (UC-1 alternate flow 3a). The JSONB column is permissive by design, so the strictness has to live upstream of it — an invalid record must never reach Kafka in the first place. Contract validation is therefore a hard gate, not a warning |
| Control-plane privilege creep | A UI that can define contracts drifts into a UI that holds database credentials and DDL grants — the blast radius of a compromise becomes every target system | The control plane writes registry rows only; it holds no infrastructure credentials in Path B. Adapters read config from the registry and apply it with their own credentials. Explicitly rejected: DDL-per-contract, which would have required the service to hold CREATE TABLE grants (Data Model §4.0) |
| Contract/target drift | An adapter is hand-configured to a target the registry doesn't know about; the registry stops being the system of record and nobody notices until a fan-out silently stops | The registry is the only source of adapter configuration — adapters read attachments from it and are never hand-configured per contract. Any adapter config that can't be expressed as an attachment is a design smell, not a shortcut |
| Parameterization stalls half-done | Some paths read the contract, others keep a compiled fallback; the codebase carries both mechanisms indefinitely and neither is trustworthy | Release 3's exit criterion is deleting the compiled DTO/mapper, not bypassing it. A fallback that still compiles is a fallback that will be used |
| Retry storms overwhelm a recovering target | Target flaps between down/overloaded | Bounded retry with exponential backoff (Release 2); consumer-level backpressure via bounded poll size |
| DLQ grows unbounded and is never triaged | Silent data loss in practice (technically preserved but never actioned) | Release 8's dashboard surfaces DLQ depth prominently, broken down by contractId so a single bad contract is visible rather than averaged away; the same release gives operators a direct replay path so triage has a low-friction next step |
| New adapter developer breaks the idempotency/retry contract | The platform's core reliability promise silently stops holding for that target | Generic adapter pattern documented as the acceptance checklist (Architecture §6); shared test harness/base test class for "every adapter must pass these idempotency + DLQ tests" is a Release 6 deliverable alongside the new adapter type itself |
| Natural key reused/collides through a data-entry error | Ordering guarantee (create-before-update-before-delete) violated for that key | Source Service validation treats natural-key uniqueness on create as a hard constraint per contract; duplicate submission is rejected, not silently repartitioned |
| A contract's natural-key strategy is edited after records exist | Repartitions the stream and invalidates every existing (contract_id, natural_key) row — silent, and very hard to unwind | Key-strategy changes are not a compatible contract edit (Data Model §5); the API rejects them. Changing the key means a new contract |
8. Traceability Back to Source Documents
| This plan's release | Original spec section it fulfills |
|---|---|
| Release 1 | §12 "Must ship (MVP)" |
| Release 2 | §6 (reliability/failure handling, hardened) |
| Release 3–4 | §5 "The upgrade that makes this industry-grade" — schema-as-enforced-contract, now a registry of contracts |
| Release 5–6 | §12 "Phase 2": Target Registry and pluggable formatters, realized as the adapter catalog + attachments |
| Release 7 | §7 "Update / Delete and the Append-Only Tension," option 2 |
| Release 8 | §8 Observability, Phase 2 half; §12 "Phase 2": DLQ replay tooling, admin dashboard |
| Release 9 | (beyond the original brief — kept optional and gated) |
Every "Phase 2" line item from the original design brief has a home in this plan — nothing from the original scope is dropped, only sequenced. The re-sequencing changed when each lands, not whether it does.