Appearance
Implementation Plan
System: Intern Integration Platform (IIP) Methodology: Extreme Programming (XP) Related docs: Architecture · Use Cases · Data Model
Scope statement: This plan builds the full platform described in the architecture document — not just the MVP. What the original design brief labeled "Phase 2" is not an optional stretch goal here; it is Releases 4–7 below. There are deliberately no dates or hour estimates in this document — XP replaces fixed-date planning with small, working, continuously-integrated releases ordered by value and by what each subsequent release depends on.
1. Why XP Fits This Project
This is a solo project — one developer, no team. XP's team-oriented rituals (pairing, collective ownership, planning-game negotiation between separate customer/developer roles) don't disappear here; they're adapted to a single-developer context rather than dropped. The practices that don't depend on headcount (small releases, test-first, continuous integration, refactoring, simple design, sustainable pace) apply unchanged.
| XP practice | How it applies here |
|---|---|
| Small releases | Each release below is a working, deployable increment of the real system — not a demo throwaway. Release 1 is a genuine walking skeleton: UI → Source Service → Kafka → both adapters → both targets, fully working end-to-end, before anything else is added. |
| Planning game | Solo-adapted: you play both roles. The backlog (§4) is self-prioritized by pulling the next-highest-value story off the top, not by a pre-committed schedule — there's no separate customer to negotiate scope with, so the architecture's guarantees (§14 NFR table in Architecture) act as the acceptance authority. |
| Test-first development | Every story's Definition of Done (§6) requires tests written alongside (or before) the implementation — this is especially non-negotiable for idempotency and DLQ behavior, since those guarantees are invisible unless tested. Solo-adapted: with no pairing partner, the test is the second pair of eyes. |
| Continuous integration | Every merge to main triggers the full build + test suite (including Testcontainers integration tests) — the "walking skeleton" must never be allowed to stop walking. Matters more solo, not less: there's no teammate to notice a broken build by accident. |
| Refactoring | The generic adapter pattern (Architecture §6) is expected to be discovered by refactoring the second adapter to look like the first, not designed upfront in the abstract. |
| Simple design | Build the two concrete adapters (DB, File) before the Target Registry abstraction (Release 6) — the abstraction is extracted once real duplication exists, not speculated in advance (see Architecture AD-8). |
| Structural consistency (solo-adapted "collective ownership") | With one developer, ownership isn't the issue — context-switching is. The generic adapter pattern and canonical schema mean any adapter looks like any other, so picking a story back up after weeks away costs re-reading one shape, not one shape per adapter. |
| Sustainable pace / no dates | Releases are ordered by dependency and value, not scheduled against a calendar — this document intentionally contains no dates or hour estimates. Solo, this is a guardrail against burnout as much as a planning technique. |
| Metaphor | "Every intern record is a single event flowing down a shared pipe; each target adapter is a tap on that pipe that can be added, removed, or temporarily closed without affecting the others." |
2. Release Roadmap (Dependency Order, Not a Calendar)
Releases are shown linear for dependency clarity; in practice R3/R4 and R6/R7 can interleave once R1–R2 are stable, since they don't strictly depend on each other — worth reconsidering at each planning-game pass rather than treating the chain as rigid.
3. Release Overview Table
| Release | Theme | Key capabilities | Primary use cases | Depends on |
|---|---|---|---|---|
| 1 | Walking Skeleton (MVP) | UI, Source Service, intern.created, minimal intern.dlq, DB adapter (upsert), File adapter (dedup-append), basic Actuator health | UC-1, UC-2, UC-5, UC-6, UC-7, UC-8, UC-10 (basic) | — |
| 2 | Reliability Hardening | Formalized retry policy w/ backoff, failure classification taxonomy, chaos/failure-injection tests, idempotency test suite | UC-7, UC-8 | Release 1 |
| 3 | Contract Enforcement | Schema Registry integration, compatibility CI gate, versioned canonical schema | UC-1, UC-5, UC-6 (validation step) | Release 1 |
| 4 | Observability | Prometheus scraping, Grafana dashboards (lag, DLQ depth, throughput, adapter health), structured logging w/ traceId | UC-10 | Release 1–3 |
| 5 | Mutable Lifecycle | intern.updated/intern.deleted topics, Edit/Delete UI + endpoints, DB update/delete handling, CSV snapshot-rebuild model, audit log | UC-3, UC-4 | Release 3 (schema evolution needed) |
| 6 | Extensibility | Target Registry (config-driven adapter enablement), pluggable formatter strategies (CSV/JSON/XML), one additional real adapter proving the pattern | UC-9, UC-12 | Release 1–2 (pattern must be proven twice before abstracting) |
| 7 | Operability | DLQ inspection + replay tooling, admin dashboard UI | UC-11 | Release 4 (dashboard infra), Release 2 (DLQ maturity) |
4. Backlog by Release
Each story follows the standard XP story-card shape: a plain statement of value, plus acceptance criteria. With no separate customer role, the acceptance criteria are checked against the architecture's own guarantees (Architecture §14) rather than a stakeholder sign-off. Sizing is intentionally not included — stories are pulled by value and risk, not by estimated effort, and no dates are attached per the project's XP approach.
Release 1 — Walking Skeleton (MVP)
| Story | Acceptance criteria |
|---|---|
| As HR Staff, I can submit a new intern record via a form | Invalid input is rejected with field-level errors; valid input returns a confirmation with a recordId |
| As HR Staff, I can view previously submitted records | List reflects submissions via the Source Service's own read path, not by querying adapter targets |
As the platform, every valid submission becomes exactly one canonical event on intern.created, keyed by internId | Verified with a Testcontainers test: publish once, assert the message exists with the correct key |
| As the Database Adapter, I persist every event to PostgreSQL without creating duplicates on redelivery | Idempotency test: deliver the same message twice, assert exactly one row |
As the File Adapter, I append every event to interns.csv without creating duplicate lines on redelivery | Idempotency test: deliver the same message twice, assert exactly one CSV line |
| As the platform, a target being down does not affect the other target | Integration test: stop Postgres, assert File adapter still succeeds and DB adapter's offset stays uncommitted |
| As the platform, a poison message is quarantined instead of blocking the pipeline | DLQ test: publish a malformed message, assert it lands in intern.dlq and a subsequent good message still processes |
| As a Platform Operator, I can check basic health of every service | /health and /metrics respond correctly on each service via Spring Boot Actuator |
| As the developer, I can run the whole flow locally with one command | docker-compose up brings up UI, Source Service, Kafka, both adapters, Postgres, and produces a working end-to-end demo |
Definition of Done for Release 1: see §6 — this release additionally requires at least one full Testcontainers end-to-end test (publish → both targets updated) as an explicit exit criterion, since it's the proof the architecture's core promise holds.
Release 2 — Reliability Hardening
| Story | Acceptance criteria |
|---|---|
| As the platform, transient vs. permanent failures are classified consistently across adapters | A shared FailureClassifier component/interface is used by both adapters; unit-tested against a table of exception types |
| As the platform, retries are bounded with backoff, not immediate infinite loops | Configurable max attempts + backoff; test asserts retry count and timing behavior |
| As the platform, I have automated chaos/failure-injection tests | Tests that kill/restart the target mid-stream (Testcontainers) and assert eventual consistency with no loss/duplication |
| As a developer, the DLQ envelope carries enough metadata to diagnose the failure without re-deriving it | DLQ message includes original topic/partition/offset/key, error type, error message, failed adapter, attempt count, timestamp |
Release 3 — Contract Enforcement
| Story | Acceptance criteria |
|---|---|
| As the platform, the canonical schema is registered and versioned in a Schema Registry | Source Service publish fails fast (before hitting Kafka) if the record doesn't conform |
| As the platform, adapters validate incoming messages against the same registry | A schema-invalid message is provably rejected at deserialization, not deep in business logic |
| As the developer, incompatible schema changes are caught in CI, not production | CI pipeline runs a compatibility check against the registry before merge; a deliberately-breaking PR is used as a test case |
| As a developer, I can add a new optional field without breaking existing consumers | Add a field, redeploy only the Source Service, assert existing (un-redeployed) adapter still processes new messages correctly |
Release 4 — Observability
| Story | Acceptance criteria |
|---|---|
| As a Platform Operator, I can see consumer lag per adapter over time | Grafana panel sourced from Prometheus scraping Kafka/Actuator metrics |
| As a Platform Operator, I can see DLQ depth over time | Grafana panel; alerts (or at least visual threshold) on sustained non-zero depth |
| As a Platform Operator, I can see per-adapter success/failure counts and throughput | Grafana panel; counters exposed via Actuator custom metrics |
| As a developer, every log line for a given submission can be correlated end-to-end | traceId is generated at submission and propagated/logged at every hop; a log-search-by-traceId walkthrough is documented |
Release 5 — Mutable Lifecycle
| Story | Acceptance criteria |
|---|---|
| As HR Staff, I can edit an existing intern record | PUT /interns/{internId} publishes to intern.updated; DB adapter applies ON CONFLICT (intern_id) DO UPDATE; File adapter reflects the change |
| As HR Staff, I can delete an intern record | DELETE /interns/{internId} publishes to intern.deleted; DB row removed/soft-deleted; File adapter's snapshot omits the intern |
| As the platform, update/delete for one intern are always applied in the order they were issued | Ordering test relying on internId partition key: interleave create/update/delete rapidly, assert final state is correct |
| As the platform, the CSV file reflects current state, not a raw append log | File adapter rebuilds interns.csv from its keyed state store on every event; test asserts an edited/deleted intern is not duplicated/stale in the file |
| As a Platform Operator, every lifecycle event is captured in an audit trail | INTERN_AUDIT_LOG table gets a row per event, independent of the current-state table |
Release 6 — Extensibility
| Story | Acceptance criteria |
|---|---|
| As a Platform Operator, I can enable/disable which adapters are active via configuration | Target Registry config change takes effect without modifying core service code |
| As a developer, I can add a new target by writing only a new adapter service | A second, genuinely new adapter (e.g., a webhook or document-store target) is built and deployed with zero changes to UI/Source Service/existing adapters — this is the concrete proof of UC-9, not just a diagram |
| As a developer, each target can use a different serialization format for its output | Formatter strategy (CSV/JSON/XML) is pluggable per adapter, selected via config |
Release 7 — Operability
| Story | Acceptance criteria |
|---|---|
| As a Platform Operator, I can view DLQ contents grouped by error type/adapter in a UI | Admin dashboard reads intern.dlq non-destructively and renders it |
| As a Platform Operator, I can replay a fixed DLQ message back into the pipeline | Replay tool re-publishes to the correct source topic; original DLQ entry is marked replayed (not deleted) for audit |
| As a Platform Operator, I can see an operational overview (health, lag, DLQ depth) in one place | Dashboard aggregates Release 4's Grafana data or embeds equivalent views |
5. Cross-Release Engineering Practices
These apply from Release 1 onward, not as later additions:
| Practice | Application |
|---|---|
| Test-Driven Development | Adapter transform logic (canonical → SQL / canonical → CSV) and the FailureClassifier are natural TDD candidates — pure functions with clear input/output tables. |
| Testcontainers over mocks for integration tests | Kafka and Postgres are spun up for real in integration tests from Release 1 onward — this is what makes the reliability guarantees provable rather than asserted (see Original Specification §9). |
| Deliberate self-review on cross-cutting components | The generic adapter pattern, FailureClassifier, and canonical schema changes affect every consumer. Solo-adapted pairing: step away and re-read the diff cold (or read it out loud) before merging anything that touches these — the goal a pair would serve, catching contract breaks early, still has to happen, just via a second pass instead of a second person. |
| Continuous Integration | Every push runs unit tests, integration tests (Testcontainers), and — from Release 3 — the schema compatibility check. A red build blocks merge. |
| Refactor before extending | Before Release 6 extracts the Target Registry / generic adapter interface, refactor the DB and File adapters (built independently in Release 1) to visibly share the same shape — the abstraction should describe two real implementations, not precede them. |
| YAGNI on Phase-2-flavored complexity | Even though the full system is the goal, still resist building Release 6/7 machinery (registry, replay tooling) inside Release 1 — sequence still matters even without dates attached to it. |
6. Definition of Done (applies to every story, every release)
A story is not done until:
- Unit tests cover the new/changed logic (validation, mapping, classification).
- Where the story touches Kafka and/or a target system, an integration test using Testcontainers proves the behavior against real infrastructure.
- If the story affects idempotency-sensitive code, an explicit "deliver twice, assert once" test exists.
- If the story affects the canonical schema, a Schema Registry compatibility check passes in CI (Release 3+).
- Logging includes
recordId/traceIdat every new processing step the story introduces. - The service builds and runs via
docker-compose upalongside all other services with no manual steps. - Actuator
/healthreflects the new component's status if it can fail independently (Release 1+). - Documentation impact is checked: does this story change the architecture, a use case, or the data model enough that the relevant doc needs a diff? (Living docs, not a one-time artifact.)
7. Risk Register
| Risk | Impact | Mitigation |
|---|---|---|
File adapter scaled to multiple instances, corrupting interns.csv | Data corruption on the file target | Single-writer constraint enforced architecturally (documented in Architecture AD-6); CI/deploy config pins File Adapter to one replica |
| Schema changes break already-deployed adapters | Consumer crashes or silently mis-processes data | Schema Registry compatibility gate in CI from Release 3 onward; BACKWARD compatibility mode enforced |
| Retry storms overwhelm a recovering target | Target flaps between down/overloaded | Bounded retry with exponential backoff (Release 2); consumer-level backpressure via bounded poll size |
| DLQ grows unbounded and is never triaged | Silent data loss in practice (technically preserved but never actioned) | Release 4 dashboard surfaces DLQ depth prominently; Release 7 gives operators a direct replay path so triage has a low-friction next step |
| New adapter developer breaks the idempotency/retry contract | The platform's core reliability promise silently stops holding for that target | Generic adapter pattern documented as the acceptance checklist (Architecture §6); shared test harness/base test class for "every adapter must pass these idempotency + DLQ tests" is a Release 6 deliverable alongside the new adapter itself |
internId reused/collides across HR data entry error | Ordering guarantee (create-before-update-before-delete) violated for that key | Source Service validation treats internId uniqueness on create as a hard constraint; duplicate submission is rejected, not silently repartitioned |
8. Traceability Back to Source Documents
| This plan's release | Original spec section it fulfills |
|---|---|
| Release 1 | §12 "Must ship (MVP)" |
| Release 2 | §6 (reliability/failure handling, hardened) |
| Release 3 | §5 "The upgrade that makes this industry-grade" |
| Release 4 | §8 Observability, Phase 2 half |
| Release 5 | §7 "Update / Delete and the Append-Only Tension," option 2 |
| Release 6 | §12 "Phase 2": Target Registry, pluggable formatters |
| Release 7 | §12 "Phase 2": DLQ replay tooling, admin dashboard |
Every "Phase 2" line item from the original design brief has a home in this plan — nothing from the original scope is dropped, only sequenced.