Skip to main content

Migration Plan

:::info Source Sourced from services/analytics-service/MIGRATION_PLAN.md in the documentation repo. :::

1. Rules​

  • Analytics envelope (F27) frozen M0 end.
  • Metric registry (F29) frozen M3 start.
  • Events pipeline is append-only; never alter raw events.
  • Cold archive immutable after close-of-month.

2. ClickHouse Schema Evolution​

  • Add columns: ALTER TABLE … ADD COLUMN with default.
  • Remove columns: deprecate; move to _deprecated suffix; retire after 1 milestone.
  • Materialized views versioned; new versions run parallel.

3. Kafka Topic Evolution​

  • New event types auto-ingested.
  • Schema registry tracks per-topic schemas.
  • Breaking change: new topic with v2 suffix.

4. Postgres Metadata Evolution​

  • Standard additive migrations.

5. Metric Definition Evolution​

  • Versioned; old metrics remain callable.
  • Query template changes don't invalidate historical dashboards pinned to old version.

6. Event Schema Consumer Drift​

  • Firehose tolerates unknown fields.
  • Alerts on schema registry drift for review.

7. AI Insight Migration​

  • Prompt versioning via ai-gateway.
  • At-risk predictor retraining requires compliance sign-off (EU AI Act).

8. Data Residency​

  • ClickHouse cluster per region.
  • Event routing by tenant homeRegion.
  • Migration: copy partitions to target; verify sums; swap.

9. Cold Archive Migration​

  • Parquet format versioned; new columns additive.
  • ClickHouse reads multiple versions transparently.

10. GDPR Erasure​

  • Hot tier: DELETE with propagation to materialized views.
  • Cold tier: Parquet rewrite (async; 30-day SLA).
  • Audit log of erasure retained.

11. Changelog​

Per-release: new event types ingested, new materialized views, metric schema changes, AI prompt versions.