Proof Artefact — Substrate hardening + A2 materialised-pool rollout to /diversity & /feed

Proof Artefact — Substrate hardening + A2 materialised-pool rollout to /diversity & /feed

Date: 2026-06-28
Branch: fix/substrate-hardening-a2-diversity-pool (base: feat/manthan-own-memory)
Scope: services/substrate, services/api, docker-compose.yml
Author: automated session (Claude), operator-directed (kulbir)
Intended reviewer: independent agent (epic-compliance style — re-verify against live ground truth, do not trust this doc's claims; every claim below has a runnable command).

Delivery-honesty note (operator binding): this artefact reports what is live and verified, what is an intentional tradeoff, and what is an open follow-up. Nothing here is "merged but inert" — all fixes are default-on and exercised by the running stack.

0. TL;DR for the auditor

The Manthan viewer (:27000) graph panels were dark and two services were crash-looping. Three independent faults, all now fixed and verified on the live stack (api restarts = 0, substrate restarts = 0 across repeated full sweeps; api 58 + substrate 12 unit tests green):

1. Substrate OOM-kill under concurrent embeddings graphs → bounded-concurrency self-defence.
2. DB two migrations behind its own code → the A2 materialiser + non-blank-content constraint were inert; applied them; proved the empty-content workflow gap is closed at API + DB.
3. api OOM-kill because the A2 materialised-pool perf-fix was wired only to /elites, not /diversity//feed → extended the pool to /diversity; made /feed cached + off-loop.

1. Faults, root causes (file:line), and the fix

F1 — Substrate process killed under concurrent /lineage/graph?include_embeddings=true

• Root cause: the endpoint materialises LineageNode Pydantic objects each carrying a 1536-float embedding and re-validates the whole response via response_model=LineageGraph (services/substrate/src/manthan_substrate/api.py get_lineage_graph). Concurrent builds stack transient memory; in the 512 MiB-capped container the single uvicorn process is cgroup-killed (docker oom event; OOMKilled flag unset — known cgroup-v2 miss). No self-defence existed (plain uvicorn, no concurrency cap).
• Fix (defence in depth, all default-on):
• services/substrate/src/manthan_substrate/api.py — _embeddings_semaphore (env SUBSTRATE_EMBEDDINGS_CONCURRENCY=2) gates only the include_embeddings=true path; waits up to SUBSTRATE_EMBEDDINGS_ACQUIRE_TIMEOUT=5s then sheds 503. Build logic split into _build_lineage_graph().
• services/substrate/Dockerfile — uvicorn … --limit-concurrency 96 global backstop.
• docker-compose.yml — substrate mem_limit 512m → ${SUBSTRATE_MEM_LIMIT:-768m} headroom.

F2 — Empty-content artefacts + the A2 fix being inert (the migration-lag root cause)

• Root cause: the DB sat at alembic 0001_initial while the code head is 0003_elite_candidate. So (a) the ck_artefact_content_nonblank CHECK (alembic/versions/0002_constraints.py) was never enforced, and (b) the elite_candidate / elite_materialiser_state tables (0003_elite_candidate.py) didn't exist → substrate /elites 500'd (relation "elite_candidate" does not exist) → the api fell back to loading all 7,684 × 1,536-float embeddings and OOM-killed itself.
• Why the 5 empty rows existed: authored 2026-05-06 by the runtime (services/runtime/src/manthan_runtime/loop.py ~264, tags live-runtime/p-101) when an LLM completion came back empty — before any guard existed. They were woven into lineage (4 parents, 5 children).
• Fix: removed the 5 rows + 9 lineage edges (operator-approved, irreversible — see §3); alembic upgrade head (applied 0002 + 0003); ran the materialiser. The empty-content guards are now triple-layered and proven live (§2-F2): runtime is_postable_text (loop.py:302), substrate schema ArtefactCreate.content_text min_length=1 + _non_blank (api.py), DB CHECK constraint.

F3 — api OOM on /diversity (and /elites fallback): A2 pool only wired to /elites

• Root cause: /elites (services/api/src/manthan_api/main.py:342) had a materialised fast-path; /diversity (main.py:466) had none and always fetched /lineage/graph?include_embeddings=true&limit_nodes=10000 → ~450 MiB transient spike → api cgroup-killed every few-second viewer poll, masking even the now-working /elites.
• Fix (extend the A2 pool — the materialiser already computes + stores every artefact's (cell_kind, cell_band) via the HNSW index, incrementally):
• services/substrate/src/manthan_substrate/elite_materializer.py — new read_diversity(): coverage from the elite_candidate pool (bounded by pool size; no graph walk, no embeddings on the read path).
• services/substrate/src/manthan_substrate/api.py — new GET /diversity route serving it.
• services/api/src/manthan_api/main.py /diversity — prefers the materialised path; bounded (≤1500-node) degrade-don't-OOM fallback for the pre-materialise window.
• services/api/src/manthan_api/main.py /feed — was uncached and ran score_artefacts/compute_signal over ~7.7k nodes synchronously on the event loop (blocked all requests under polling). Now off-loop (asyncio.to_thread) behind the shared 45 s TTL+single-flight cache.

2. Reproduction & evidence (auditor: run these against the live stack)

F1 — substrate no longer killable by concurrency

# BEFORE the fix this drove mem to 510/512 MiB and incremented RestartCount.
docker inspect manthan-substrate-1 --format 'restarts_before={{.RestartCount}}'
for i in $(seq 1 12); do curl -s -m30 -o /dev/null \
  "local-service/lineage/graph?limit_nodes=1000&include_embeddings=true" & done; wait
docker inspect manthan-substrate-1 --format 'restarts_after={{.RestartCount}} oom={{.State.OOMKilled}}'

Observed AFTER fix: peak ~179 MiB, RestartCount unchanged, all 12 → HTTP 200 (serialised 2-at-a-time by the semaphore).

F2 — empty-content rejected at both API and DB

curl -s -m8 -X POST -H 'Content-Type: application/json' \
  -d '{"kind":"idea","content_text":"   "}' local-service/artefacts        # → 422
docker exec manthan-postgres-1 psql -U manthan -d manthan -c \
  "insert into artefact (id,kind,content_text,created_at) values (gen_random_uuid(),'idea','   ',now());"  # → CHECK violation
docker exec manthan-postgres-1 psql -U manthan -d manthan -tA -c \
  "select count(*) from artefact where length(btrim(coalesce(content_text,'')))=0;"  # → 0
docker exec manthan-postgres-1 psql -U manthan -d manthan -tA -c "select version_num from alembic_version;"  # → 0003_elite_candidate

F3 — api stable; /diversity & /elites materialised; /feed cached & non-blocking

docker inspect manthan-api-1 --format 'api_restarts_before={{.RestartCount}}'
base=local-service/proxy/api
for p in /artefacts/recent /agents /opportunities/current /prototypes/list /elites \
         /diversity /signal/feed /patina/leaderboard /ideas/search /feed; do
  printf '%-22s ' "$p"; curl -s -m20 -o /dev/null -w '[%{http_code} %{time_total}s]\n' "$base$p"; done
# stress: 4x heavy sweep
for r in 1 2 3 4; do for p in /elites /diversity /feed /signal/feed; do curl -s -m15 -o /dev/null "$base$p"; done; done
docker inspect manthan-api-1 --format 'api_restarts_after={{.RestartCount}}'
# /feed event-loop not blocked: healthz must stay ~ms under concurrent /feed
( for i in $(seq 1 8); do curl -s -m20 -o /dev/null local-service/feed?limit=5 & done
  curl -s -m5 -o /dev/null -w 'healthz_under_load=%{time_total}s\n' local-service/healthz; wait )

Observed AFTER fix: all 10 endpoints HTTP 200; api & substrate RestartCount = 0 across 4 sweeps; /diversity & /elites report source=materialised; /feed 2nd call ~0.001 s (cache hit); healthz_under_load ≈ 0.002 s.

3. Live-system mutations performed (full disclosure)

• DB (irreversible): deleted 5 empty idea rows + 9 lineage edges; alembic upgrade head (0002 + 0003); ran the materialiser to catch up the 7,684-row backlog (57 candidates, caught up at pass 19 / processed=0).
• Containers: rebuilt + redeployed manthan-substrate-1 and manthan-api-1; substrate mem cap 512m → 768m.

4. Tests

• services/api — 58 passed (docker exec manthan-api-1 sh -c 'cd /app/services/api && python -m pytest -q').
• services/substrate — 12 passed (… services/substrate …).

5. Residual risk & follow-ups (honest — not blocking, flagged for swarm review)

1. /diversity semantics: coverage (occupied/empty cells) is exact from the pool, but occupancy counts are pool-capped at K=8/cell (flagged in the response note). Novelty bands come from the materialiser's HNSW path (KNN_K=6) vs. the old api pure-Python k=5 — same concept, slightly different values. Needs A2 swarm review before this defines product behaviour.
2. Materialiser lag: /elites + /diversity reflect the pool, which trails live writes by up to one 120 s pass. Acceptable for a viewer.
3. Unembedded corpus: 3,953 / 7,684 artefacts have no embedding (band −1, excluded from the grid) → coverage reads low (4/24). This is a real data signal, not a bug; maestro-bridge-ingested artefacts aren't embedded. Follow-up: decide on an embedding-backfill job (cost: ~3.9k gateway calls) — out of scope here (ops/cost decision).
4. /feed//signal/feed still load the full (embedding-free) graph (~62 MiB, ~2-3 s) — cached now, no OOM; a pool-backed rewrite is a possible later optimisation.

6. Auditor checklist

• [ ] Re-run §2 F1/F2/F3 against the live stack; confirm RestartCount stays flat and the 10-endpoint sweep is all-200.
• [ ] Confirm alembic_version = 0003_elite_candidate and zero blank-content rows.
• [ ] Re-run both unit suites (58 / 12).
• [ ] Read the diff (git show/PR) and confirm every fix in §1 is present at the cited file; confirm no embeddings are loaded on the /diversity materialised path.
• [ ] Weigh §5.1 (/diversity density-cap + band-source change) — accept as intended degradation, or route to A2 owners.