file-storage-service — LOCAL_DEV_SETUP

Sister docs: SERVICE_OVERVIEW · API_CONTRACTS · DATA_MODEL · TESTING_STRATEGY · DEPLOYMENT_TOPOLOGY

This guide gets a developer from a fresh checkout to running the service, exercising the signed-URL workflow against a fake GCS, scanning a file, optimizing an image, and replaying events — entirely on a laptop, with no cloud credentials required.

1. Prerequisites

Tool	Version	Notes
Node.js	20.x LTS	use `volta` or `nvm`
pnpm	9.x	monorepo package manager
Docker	≥ 24.x	for `docker-compose.dev.yml`
Docker Compose	v2 plugin	`docker compose ...`
GNU Make	any	thin wrappers in `Makefile`
`gcloud` CLI	optional	only for emulators / `pubsub-emulator` if not using docker
`mkcert`	optional	locally-trusted certs for HTTPS dev
`httpie` or `curl`	any	manual API exercises

Disk: ≥ 5 GB free for fake-gcs blobs and Postgres data.

Memory: 8 GB minimum (Postgres + Redis + ClamAV + LocalStack-equivalent + service container).

2. Repository layout (local)

services/file-storage-service/
├── src/
│   ├── domain/                 # pure aggregates, value objects
│   ├── application/            # use cases, ports, sagas
│   ├── infrastructure/
│   │   ├── http/               # NestJS controllers (Fastify)
│   │   ├── persistence/        # Kysely repositories, migrations
│   │   ├── gcs/                # GCS adapter (real) + fake-gcs adapter
│   │   ├── pubsub/             # publisher + subscriber + outbox relay
│   │   ├── scanner/            # ClamAV / Cloud DLP adapters
│   │   ├── optimizer/          # sharp / ffmpeg workers
│   │   ├── ai/                 # ai-orchestrator client
│   │   └── observability/      # otel + pino + prom
│   └── main.ts
├── test/
│   ├── unit/
│   ├── integration/
│   ├── pact/
│   └── e2e/
├── docker-compose.dev.yml
├── Makefile
├── .env.example
└── package.json

3. Bootstrap

git clone git@github.com:ghasi-tech/ghasi-melmastoon.git
cd ghasi-melmastoon
pnpm install                    # workspace install
cp services/file-storage-service/.env.example services/file-storage-service/.env
make -C services/file-storage-service dev-up

dev-up brings up:

postgres (16-alpine, port 5433)
redis (7-alpine, port 6380, used as Memorystore mock)
pubsub-emulator (gcloud, port 8085)
fake-gcs-server (port 4443, persisted under ./.dev/data/fake-gcs/)
clamav (port 3310, with TCP listener)
mailhog (1025/8025, used by other services in the same compose; harmless)
otel-collector (4318, exports to console)

The service itself is not containerized in dev — you run it with pnpm dev.

4. Environment variables

services/file-storage-service/.env.example:

# --- runtime ---
NODE_ENV=development
PORT=3007
LOG_LEVEL=debug

# --- database ---
DATABASE_URL=postgres://app:app@localhost:5433/file_storage
DATABASE_RLS_ROLE=file_storage_app
DATABASE_MIGRATION_USER=postgres
DATABASE_MIGRATION_PASSWORD=postgres

# --- redis ---
REDIS_URL=redis://localhost:6380/0

# --- pub/sub emulator ---
PUBSUB_EMULATOR_HOST=localhost:8085
GOOGLE_CLOUD_PROJECT=melmastoon-dev

# --- gcs ---
GCS_ENDPOINT=http://localhost:4443
GCS_PROJECT=melmastoon-dev
GCS_USE_FAKE=true
GCS_PRIVATE_BUCKET=mel-private-dev
GCS_MEDIA_BUCKET=mel-media-dev
GCS_ARCHIVE_BUCKET=mel-archive-dev
GCS_QUARANTINE_BUCKET=mel-quarantine-dev

# --- signed urls ---
SIGNED_URL_TTL_UPLOAD_SECONDS=600
SIGNED_URL_TTL_DOWNLOAD_DEFAULT_SECONDS=300
SIGNED_URL_TTL_DOWNLOAD_MAX_SECONDS=3600
SIGNED_URL_HMAC_KEY_ID=hmac-dev-1
SIGNED_URL_HMAC_KEY_VALUE=local-only-not-secret

# --- scanner ---
SCANNER_PRIMARY=clamav
CLAMAV_HOST=localhost
CLAMAV_PORT=3310
DLP_ENABLED=false

# --- optimizer ---
OPTIMIZER_MAX_RSS_MB=512
OPTIMIZER_MAX_PIXELS=80000000
SHARP_MAX_INPUT=83886080  # 80 MB

# --- ai-orchestrator ---
AI_ORCHESTRATOR_BASE_URL=http://localhost:3099
AI_ORCHESTRATOR_API_KEY=dev-key
AI_ORCHESTRATOR_TIMEOUT_MS=8000
AI_DAILY_BUDGET_USD=5
AI_HITL_THRESHOLD=0.85

# --- iam ---
IAM_JWKS_URL=http://localhost:3001/.well-known/jwks.json
IAM_ISSUER=https://iam.local.melmastoon.dev
IAM_AUDIENCE=file-storage

# --- multi-tenancy ---
TENANT_ID_HEADER=x-tenant-id
TENANT_FALLBACK=tnt_dev_main

For convenience the Makefile provides make seed-tenant TENANT=tnt_dev_main which inserts a row in buckets and a stub quotas row.

5. Database migrations

pnpm --filter file-storage-service migrate:latest
# rollback last:
pnpm --filter file-storage-service migrate:down

Migrations live under src/infrastructure/persistence/migrations/. Authoring:

pnpm --filter file-storage-service migrate:make add_xyz

The dev container runs migrations as postgres; the service runs as file_storage_app (no DDL, RLS-enforced). To verify RLS is working:

pnpm --filter file-storage-service test:integration -t "tenant-isolation"

Migration policy: see MIGRATION_PLAN.

6. Run the service

pnpm --filter file-storage-service dev
# or
make -C services/file-storage-service dev

Endpoints:

GET /healthz → 200 {"status":"ok"}
GET /readyz → checks DB, Redis, PubSub, GCS
GET /metrics → Prometheus
API base: http://localhost:3007/api/v1
OpenAPI UI: http://localhost:3007/docs (dev only)

7. End-to-end smoke (manual)

Step 1 — initiate upload:

curl -sS http://localhost:3007/api/v1/files/upload-sessions \
  -H 'Authorization: Bearer <dev-jwt>' \
  -H 'X-Tenant-Id: tnt_dev_main' \
  -H 'Idempotency-Key: 8f1c... 36 chars' \
  -H 'Content-Type: application/json' \
  -d '{
        "scope":"property_photo",
        "filename":"front-desk.jpg",
        "contentType":"image/jpeg",
        "sizeBytes":248123,
        "sha256":"<hex>",
        "ownership":{"propertyId":"prp_pilot_01"}
      }' | jq

Step 2 — PUT the bytes against the returned uploadUrl (resumable session URI on fake-gcs):

curl -X PUT --data-binary @./front-desk.jpg "$UPLOAD_URL" \
  -H 'Content-Length: 248123' -H 'Content-Type: image/jpeg'

Step 3 — confirm:

curl -sS -X POST http://localhost:3007/api/v1/files/upload-sessions/$SID/confirm \
  -H 'Authorization: Bearer <dev-jwt>' \
  -H 'X-Tenant-Id: tnt_dev_main' \
  -H 'Idempotency-Key: 8f1c... 36 chars'

The optimizer worker (run with pnpm --filter file-storage-service worker:optimizer) will pick up the optimization.requested.v1 event from the local Pub/Sub emulator, compute three variants (thumb / hero / full WebP), upload them to fake-gcs, and emit optimization.completed.v1.

The scanner worker (pnpm worker:scanner) sends the bytes to ClamAV (eicar test file is helpful) and emits scan.passed.v1 / scan.failed.v1.

Step 4 — request a download URL:

curl -sS -X POST http://localhost:3007/api/v1/files/$FID/downloads \
  -H 'Authorization: Bearer <dev-jwt>' \
  -H 'X-Tenant-Id: tnt_dev_main' \
  -d '{"variant":"hero","ttlSeconds":300}' | jq

8. Auth in dev

The IAM service issues dev JWTs via iam-service dev-token endpoint. Quick alias:

export DEV_JWT=$(curl -s http://localhost:3001/dev/token \
  -d '{"sub":"usr_dev","tenantId":"tnt_dev_main","roles":["TENANT_ADMIN","STAFF"]}' \
  -H 'Content-Type: application/json' | jq -r .accessToken)

A second dev tenant tnt_dev_secondary exists to run tenant-isolation manual checks.

9. Workers

The service has the following workers (run separately so you can debug each in isolation):

pnpm --filter file-storage-service worker:scanner      # ClamAV adapter
pnpm --filter file-storage-service worker:optimizer    # sharp / ffmpeg
pnpm --filter file-storage-service worker:relay        # outbox → pub/sub
pnpm --filter file-storage-service worker:retention    # nightly sweeper
pnpm --filter file-storage-service worker:erasure      # GDPR erasure
pnpm --filter file-storage-service worker:cdn-purge    # CDN invalidation

In CI / production, all workers run as Cloud Run jobs or separate Cloud Run services; locally they're concurrent processes.

10. Pub/Sub topology in dev

Topics + subscriptions are created on service startup if missing (idempotent). Names match production minus the dev- prefix added by the emulator project. To inspect:

gcloud --project=melmastoon-dev pubsub topics list \
  --pubsub-emulator-endpoint=localhost:8085

11. Common dev gotchas

Symptom	Cause	Fix
`RLS denied` on insert	service running as `postgres` not `file_storage_app`	check `DATABASE_URL`; restart service
Upload PUT returns 410	fake-gcs session expired (10 min)	re-initiate session
Optimizer never runs	optimizer worker not started	`pnpm worker:optimizer`
Scan always passes (eicar passes)	ClamAV definitions stale	`docker exec ghasi-clamav freshclam`
Cross-tenant API calls succeed	RLS disabled (running as `postgres`)	switch to `file_storage_app`
OpenAPI not loading	`NODE_ENV=production`	`NODE_ENV=development`
Variants 404 from CDN	local CDN not implemented	call file-storage signed URL endpoint instead

12. Test commands

pnpm --filter file-storage-service test                # all
pnpm --filter file-storage-service test:unit
pnpm --filter file-storage-service test:integration
pnpm --filter file-storage-service test:contract       # pact
pnpm --filter file-storage-service test:e2e
pnpm --filter file-storage-service lint
pnpm --filter file-storage-service typecheck
pnpm --filter file-storage-service test:cov            # coverage

Coverage gate is 80% lines / 80% branches; CI fails below threshold.

13. Linting and formatting

pnpm lint        # eslint --max-warnings=0
pnpm format      # prettier --write
pnpm check:format

Conventions: no any, strict TS, eslint-plugin-import enforces clean architecture boundaries (domain → no infra imports).

14. Resetting state

make dev-down            # stop containers
make dev-clean           # stop + remove volumes (drop fake-gcs blobs + DB data)
make dev-reset           # dev-clean + dev-up + migrate + seed

15. Editor configuration

.editorconfig, .vscode/settings.json, and .vscode/launch.json are committed. Recommended VS Code extensions:

ms-azuretools.vscode-docker
dbaeumer.vscode-eslint
esbenp.prettier-vscode
prisma.prisma (Kysely typegen viewing)
ms-vscode-remote.remote-containers
cweijan.vscode-redis-client

16. Useful seed scripts

Script	Purpose
`pnpm seed:tenant`	seeds dev tenants + buckets
`pnpm seed:files`	uploads sample images for property_photo scope
`pnpm seed:eicar`	uploads EICAR test file to verify scanner
`pnpm seed:large`	generates a 500 MB file to exercise resumable upload
`pnpm seed:erasure`	seeds a guest with a couple of files for erasure replay

17. Connecting Electron desktop in dev

Run apps/melmastoon-desktop with MELMASTOON_API_BASE=http://localhost:8080 (the BFF) and the BFF will proxy to file-storage. Direct upload from Electron to fake-gcs does work because fake-gcs accepts arbitrary headers; in production, GCS resumable uploads must be CORS-permitted and origin-pinned.

18. Where to go next

API quickstart: API_CONTRACTS
How events flow: EVENT_SCHEMAS, APPLICATION_LOGIC §4
Failure modes you'll hit while developing: FAILURE_MODES
Production posture: DEPLOYMENT_TOPOLOGY, SECURITY_MODEL

1. Prerequisites​

2. Repository layout (local)​

3. Bootstrap​

4. Environment variables​

5. Database migrations​

6. Run the service​

7. End-to-end smoke (manual)​

8. Auth in dev​

9. Workers​

10. Pub/Sub topology in dev​

11. Common dev gotchas​

12. Test commands​

13. Linting and formatting​

14. Resetting state​

15. Editor configuration​

16. Useful seed scripts​

17. Connecting Electron desktop in dev​

18. Where to go next​