Quality Fabric — Productization Gap Analysis

Quality Fabric — Productization Gap Analysis

Date: 2026-05-08
Benchmark: .claude/skills/service-productization
Scope: conduir-clean-room/quality-fabric/ (full repo)
Mode: Read-only audit — no service-code edits made in this pass. This note
captures the productization call and follow-on planning updates.

TL;DR

The repo presents itself as "production ready" in 23 root-level markdown files,
but against the productization skill's gates it fails *8 of 11 mandatory
checks* and has already burned a 9th in history (deletion-without-callsite-scan
regression, services/ai/__init__.py:21-43).

After multi-agent review (see Appendix A), the picture is *worse than the
initial pass found: there are three P0 issues — a JWT issuer/verifier secret
split that silently rejects valid tokens, a K8s deployment that will
CrashLoopBackoff on first apply (probe-path drift + wrong env-var name), and a
TEST_MODE auth-bypass kill switch with no production guard. The original
"~2 days of surgical fixes" estimate is fantasy — realistic scope is 4–6
working days* for one engineer just to clear the P0/P1 list. The call for this
phase is good now, great later: clear production-blocking runtime, auth,
deployability, and evidence gates first; defer broad cleanup and polish unless
they block those gates.

Phase 0 — Folder classification

Net: one real backend service (FastAPI), one real frontend service (dashboard),
two ghost services (template_tracker collector + maturity) advertised by
Dockerfiles whose entrypoints don't exist.

Mandatory gates — pass/fail

Six-dimension scoring (the FastAPI service)

Provisional overall maturity: ~1.8 / 5 (source-inspection score, revised down from 2.3 after multi-agent review surfaced the auth/k8s P0 issues; final score requires V1-V5 evidence).

Productization call — good now vs. great follow-on

Good now: minimum credible productization

These are the smallest changes that move the backend service from audit finding
to deployable service evidence. This is the recommended scope for this phase.

1. Collapse JWT handling to one issuer/verifier and one canonical secret.
Keep a short compatibility alias path, but fail closed outside dev/test and
remove the start.sh dev-secret footgun.
2. Fix K8s runtime correctness. Align probe paths with real endpoints, add
or remove /health/startup, wire the canonical JWT secret name, and make the
default image entrypoint intentional rather than accidentally legacy.
3. Resolve port and runtime-contract drift. Pick one backend port for local,
compose, config, frontend, and K8s; add /version; mount Prometheus at the
documented /metrics path or remove the unused 9090 promise.
4. Make cold-start local and reproducible. Provide a service-local compose
path with embedded Postgres + Redis and no dependency on maestro-network,
maestro-postgres, or maestro-redis.
5. Decide the legacy/modular API boundary for release. Either point all
runtime entrypoints to the modular API, or explicitly bless the legacy API
and document the risk. Any deletion must include callsite-clearance evidence.
6. Remove or disable ghost service entrypoints. Delete or quarantine the
template_tracker Dockerfiles unless collector/maturity modules are actually
implemented for this release.
7. Externalize Maestro dependencies cleanly. The six wheels are already
declared in pyproject.toml; the issue is portability. Use versioned package
sources/registry artifacts, or replace the direct logging dependency where
productization does not need Maestro coupling.
8. Produce V1-V5 evidence. Re-run cold start, import graph, API contract
probes, target tests, and doc-reference checks with artifact IDs.

Great follow-on: useful, but not this phase's blocker

• Productize frontend/dashboard/ as a separate service.
• Reconcile and archive the 23 root markdown status/completion files.
• Add full governance polish (LICENSE, SECURITY.md, CONTRIBUTING.md) if the

repo is being published or handed to a broader team.

• Backfill full security/chaos suites and broaden ≥80% coverage beyond the

backend productization scope.

• Remove the quality_fabric/ re-export shim after confirming downstream imports.
• Implement template_tracker as a real service if product direction still needs it.

Per-component verdicts (services/*)

• api/ — primary service, but two coexisting entrypoints; cleanup overdue.
• adapters/ — coherent library; package with versioned API surface.
• ai/ — large module (15+ files); carries an explicit deletion-regression

stub at services/ai/__init__.py:21-43; needs callsite-clearance audit.

• cicd/ — small library; fine; not a service.
• core/ — heavy "kitchen sink" library: config, logging, telemetry, SLOs,

scoring, registry, orchestrator, aggregator. Split candidate, but not by
productizing each piece as its own service.

• database/ — clean SQLAlchemy + Redis; only qf_logging's maestro dep leaks.
• orchestration/ — k8s execution engine; library, not a service.
• template_tracker/ — broken: ghost Dockerfiles ↔ missing modules.
• webhooks/ — single file webhook_manager.py; router lives at

services/api/routers/webhooks.py. Naming overloaded; consolidate.

Folder-by-folder due diligence (clean-room only)

This expands the service inventory above. It intentionally excludes platform
gateway services outside conduir-clean-room; no separate gateway/ service
folder exists in this repo.

services/api/ - backend API service host

Call: not good yet. It is the real backend service, but it needs a narrow
Phase I hardening pass before it should be called productized.

Evidence and risks:

• run_server.py and the root Dockerfiles mostly boot

services.api.testing_service_api:app, while services/api/main.py is the
newer modular API. This means local, Docker, demo, and production paths can
exercise different applications.

• The legacy auth dependency in services/api/testing_state.py returns a user

object even without credentials. Since legacy is still the default Docker and
console entrypoint, this is not just test-only dead code.

• services/api/routers/health.py can return HTTP 200 with

{"status": "not_ready"}, and services/core/connection_pools.py marks
pools initialized even when database or Redis initialization failed.

• services/api/main.py logs failed database readiness and continues startup.

That is useful for local development, but it makes Kubernetes readiness and
release validation unreliable.

• Auth is route-local rather than globally enforced. The artifact router exposes

tenant-scoped list/download/delete paths without obvious auth or tenant
binding at the router boundary.

• Health exists, but a product service still needs a stable /version,

readiness that fails hard on missing dependencies, and a metrics endpoint or
explicit decision to export metrics elsewhere.

Good now: choose one entrypoint, preferably services.api.main:app; make
legacy opt-in only; make readiness return non-2xx when DB/Redis are unavailable;
fail startup in production when required dependencies or secrets are missing;
add an auth/tenant dependency audit for every router; add /version and
document the API surface.

Great follow-on: split long-running execution from synchronous API traffic,
publish OpenAPI contract snapshots, and add consumer contract tests for the
dashboard and external callers.

services/adapters/ - execution adapter library

Call: good as a library after hardening; do not productize as an independent
service in Phase I.

Evidence and risks:

• The folder models pluggable adapters for pytest, Playwright, Jest, k6,

security, and related test execution flows. It has no independent runtime
manifest, port, readiness endpoint, or deployment boundary.

• It executes untrusted or semi-trusted commands. The existing review already

flags shell execution in the Node/Playwright adapter. That is a hardening
issue inside the API/worker process, not proof of a service boundary.

• Adapter result schemas need to stay stable because API routes, CI/CD logic,

and quality gates consume them.

Good now: keep it as a versioned internal package, remove shell execution
where avoidable, pin adapter command contracts, and add focused tests for result
normalization and failure reporting.

Great follow-on: move risky execution into a sandboxed worker service only
after the API productization baseline is stable.

services/ai/ - AI/ML analysis library

Call: not a service. It is a large domain library with import-regression
risk.

Evidence and risks:

• API routers import AI modules directly. There is no standalone process,

Dockerfile, port, queue consumer, or service-local health surface.

• The package carries an explicit compatibility/deletion-regression stub for

federated_learning_engine, which is evidence that callsites and imports are
not fully settled.

• Because these modules sit on user-facing prediction and analysis paths, import

failures can become API startup or route-time failures.

Good now: preserve it as library code, add import graph tests for every API
router using it, remove compatibility stubs only after callsites are corrected,
and define stable request/response models for each AI-backed API route.

Great follow-on: extract a separate AI service only if model lifecycle,
GPU/CPU isolation, cost controls, or asynchronous workload management require a
real runtime boundary.

services/cicd/ - CI/CD quality logic library

Call: good enough as a library; not a service.

Evidence and risks:

• The folder contains quality gates, result processing, and metrics logic used

by the backend. It has no deployable runtime boundary.

• The main productization risk is schema drift: CI results, adapter results,

API responses, and dashboard displays need to agree on status, severity, and
evidence fields.

Good now: keep it internal, document its public functions or models, and add
unit tests around gate decisions and result transformation.

Great follow-on: publish the quality-gate contract as a small internal SDK
or shared model package if other services begin consuming it directly.

services/core/ - shared service kernel

Call: useful but too broad. It is not independently productizable.

Evidence and risks:

• The folder owns config, logging, connection pools, telemetry, SLOs, scoring,

registries, artifact storage, and orchestration helpers.

• services/core/qf_config.py defaults CORS broadly and allows optional secrets

that should be mandatory in production.

• services/core/connection_pools.py catches initialization failures and can

mark the pool manager initialized even when backing services are unavailable.

• services/core/artifact_storage.py builds local and S3 storage paths but does

not itself enforce tenant authorization; callers must do that consistently.

Good now: keep it in-process, but harden the production config path,
dependency initialization, artifact URL contracts, and tenant-sensitive helper
APIs.

Great follow-on: split config, storage, telemetry, and orchestration into
clear packages once the API service has one stable runtime path.

services/database/ - persistence library

Call: close to good as a library; unsafe as currently wired for production.

Evidence and risks:

• The folder provides SQLAlchemy models, database helpers, auth service logic,

and initialization scripts. It is not a database service; Postgres/Redis are
external dependencies.

• Defaults include local development credentials and automatic schema creation.

init_db() explicitly notes Alembic should be used for production.

• Productization is blocked by the JWT secret split: token verification and

token issuance read different environment variable names in different layers.

Good now: use one canonical JWT secret setting, require production database
URLs and secrets, disable direct create_all fallback in production, and make
Alembic migration success a release gate.

Great follow-on: add migration drift checks, seed-data policy, and rollback
runbooks for schema changes.

services/orchestration/ - execution orchestration library

Call: not a standalone service today. It is a worker candidate.

Evidence and risks:

• The folder contains orchestration and Kubernetes execution logic used by the

backend, but there is no independent deployment definition under this folder.

• The product risk is operational: Kubernetes job permissions, namespace

isolation, cleanup, timeout behavior, cost controls, and failure propagation.

Good now: keep it as a library, document the required Kubernetes RBAC and
namespace assumptions, and test job lifecycle cleanup and timeout paths.

Great follow-on: extract it into a dedicated worker when queue-backed test
execution, concurrency controls, and independent scaling are required.

services/template_tracker/ - broken service candidate

Call: not good. Either finish it or remove it from the service inventory.

Evidence and risks:

• The folder has Dockerfile.collector and Dockerfile.maturity, but the

referenced modules services.template_tracker.collector and
services.template_tracker.maturity are absent.

• Dockerfile.maturity references a collector group that is not created in

that Dockerfile.

• Existing Python files model event schemas and a Redis publisher. That is

library functionality, not a deployable service.

Good now: quarantine or delete the Dockerfiles from productized deployment
claims, or implement the missing collector and maturity modules with health
checks, compose/k8s manifests, and tests.

Great follow-on: build it as an event-processing service if template
maturity tracking is still product direction; otherwise keep only the event
schema/publisher library.

services/webhooks/ - webhook subsystem library

Call: not a separate service. It belongs to the API service for now.

Evidence and risks:

• Webhook management code lives in services/webhooks/, while HTTP routes live

under services/api/routers/webhooks.py.

• The main productization risks are signing, replay protection, retry/backoff,

dead-letter behavior, tenant isolation, and delivery observability.

Good now: consolidate naming, make the API router the only HTTP boundary,
and add tests for signature validation, delivery failure, retries, and tenant
scoping.

Great follow-on: extract to a delivery worker if webhook volume, retries, or
isolation needs exceed the API process.

quality_fabric/ - compatibility shim

Call: not a service and not a productization target.

Evidence and risks:

• It exists to preserve old import paths into the new module layout.
• Keeping it forever makes package ownership ambiguous and can hide stale

downstream imports.

Good now: keep only until import validation and downstream compatibility are
complete.

Great follow-on: remove it after a deprecation window and explicit import
graph check.

frontend/dashboard/ - standalone frontend service

Call: real service, but not good yet for Phase I unless the frontend is in
scope. It should be productized separately from the backend.

Evidence and risks:

• frontend/dashboard/Dockerfile runs npm ci --only=production before

npm run build, but the build relies on dev dependencies such as Vite and
TypeScript.

• The nginx config proxies /api/ and /ws/ to local-service, which points

back into the nginx container rather than a backend service in normal
multi-container deployment.

• API and WebSocket defaults drift between 8000 and 9800 across frontend source

files.

• A tracked .env exists under the dashboard folder and should be reviewed or

removed according to the repo secret policy.

Good now: fix the Docker build stage, configure backend upstreams via env or
deployment config, align API/WebSocket defaults, remove or sanitize tracked
runtime .env, and rename remaining Maestro IDE branding where inappropriate.

Great follow-on: add dashboard contract tests against the backend OpenAPI
schema and end-to-end smoke checks for deployed environments.

Deployment/support folders

Call: not services, but they decide whether the services are actually
deployable.

Evidence and risks:

• Root Dockerfiles and compose files disagree on app entrypoint, probe path, and

runtime assumptions.

• docker-compose.production.yml defines an nginx reverse proxy and two

quality-fabric backend instances, but also references
run_self_improvement.py, which is absent from the repo.

• Kubernetes and Terraform content should be treated as deployment artifacts for

the API and dashboard, not as separate services.

• Alembic is a release-critical migration asset and should be part of the

backend readiness gate.

• test-instances/* folders are fixtures and sample evidence packs, not

independently deployable services.

Good now: reduce root Dockerfiles to one supported backend image plus
clearly labeled demos, align health probes with the selected API app, require
real production secrets, remove or repair missing compose commands, and make
migration success mandatory before readiness.

Great follow-on: add environment-specific deployment runbooks, rollback
procedures, and post-deploy smoke tests for API, dashboard, migrations, and
background execution.

Gateway/reverse-proxy references

Call: not a clean-room service. Treat this as stale documentation plus
reverse-proxy deployment support.

Evidence and risks:

• There is no gateway/ folder and no current services/core/maestro_api_gateway.py

file in the repo.

• Historical docs and reports still reference maestro_api_gateway.py and an

external gateway:8080 service. Those references should not be used as proof
that a gateway service exists.

• The only concrete gateway-like runtime in clean-room is nginx in

docker-compose.production.yml and root nginx.conf.

Good now: remove or archive stale gateway docs, decide whether nginx is the
supported ingress path, and align deployment manifests around that choice.

Great follow-on: introduce an actual API gateway only if there is a product
need for cross-service routing, auth centralization, policy enforcement, or
multi-service traffic shaping.

Outstanding evidence list (Phase 8 V1–V5)

• V1 cold-start: NOT_ATTEMPTED. Compose path requires external network.
• V2 import-graph: 1 known regression (federated_learning_engine).
• V3 API contract probe: NOT_ATTEMPTED.
• V4 test runner integration: NOT_ATTEMPTED in this pass.
• V5 doc reference resolution: 3 broken README links + 1 broken

testing_service_api.py reference.

All five must be re-run during the actual productization branch.

Recommended next pass (out of scope for this audit)

1. Treat the inner quality-fabric repo as authoritative. Continue from the
current productization branch or create a dated continuation branch from it.
2. Mark the score and Appendix A findings as source-inspection evidence until
V1-V5 have been re-run.
3. Land the JWT collapse first (Appendix A, "Highest-leverage single fix").
Every auth/RBAC/tenancy test downstream is currently testing a broken substrate.
4. Work the P0 → P1 → P2 list (Appendix A).
5. Re-run V1–V5 with recorded evidence IDs.

Appendix A — Multi-agent review findings (2026-05-08)

Three independent agents reviewed this gap doc. The verifier fact-checked every
claim; the auditor did a fresh independent pass; the severity ranker assessed
blast radius. Below is the reconciled output.

A.1 Corrections to the original analysis

A.2 New findings the original pass missed

P0 — Will fire immediately on prod deploy or compromise auth:

• JWT issuer/verifier secret split. services/api/middleware/auth.py:43 reads JWT_SECRET; services/database/auth_service.py:41 reads config.jwt_secret_key (env QUALITY_FABRIC_JWT_SECRET_KEY). Both modules independently jwt.encode/jwt.decode (auth.py:134,141; auth_service.py:140,154). Middleware imports the verifier from auth_service, so the request path uses the config secret while issuance can use either — silent token rejection in any deploy that only sets JWT_SECRET. (auditor)
• K8s probe-path drift = guaranteed CrashLoopBackoff. infrastructure/kubernetes/deployment.yaml:115,124,133 probes /health/liveness, /health/readiness, /health/startup. Code exposes only /health/live and /health/ready (services/api/routers/health.py:91,106); /health/startup does not exist. K8s has never successfully deployed this image. Worse, /health/ready returns 200 even when pools aren't initialized (health.py:91-103) — readiness gate is cosmetic. (auditor)
• K8s secret-name mismatch. deployment.yaml:89-93 injects MAESTRO_SECRET_KEY, but no production code reads that name. auth_service.py:37 will fail-fast on the missing QUALITY_FABRIC_JWT_SECRET_KEY. (severity, auditor)
• TEST_MODE auth-bypass kill switch with no env guard. services/api/dependencies.py:117-128 returns a fake admin user (roles: ["user","admin"]) whenever TEST_MODE=true or QUALITY_FABRIC_TEST_MODE=true. No production-environment refusal. .env.development:27 ships TESTING=true. Any operator who copies a dev env file to staging silently disables auth. (auditor)
• JWT_SECRET=maestro-dev-jwt-secret default in start.sh:11-12. Sets both JWT_SECRET and QUALITY_FABRIC_JWT_SECRET_KEY to the dev value. If start.sh runs anywhere outside dev, attackers can mint valid JWTs. (severity)

P1 — Will fire on first integration or operation:

• Deeper federated_learning_engine import not covered by stub — see A.1 #6.
• No /metrics ASGI mount despite Prometheus port 9090 in K8s manifest + prometheus.io/scrape: "true" annotation. Silent observability hole — alerts never fire. (severity)
• Frontend has two competing API clients on different ports: qualityFabricApi.ts:8 defaults to local-service; api.ts:3-5 defaults to local-service/api. No single backend port choice makes both work. (auditor)
• Frontend Dockerfile is copy-pasted from a different product — labeled "Maestro Frontend", creates user maestro/group maestro (frontend/dashboard/Dockerfile:1-2,24-26). The Quality-Fabric frontend has never had its own Dockerfile. (auditor)
• Test suite doesn't even collect cleanly. Repo's own committed .report.json (153 KB at root) shows summary: {failed: 18, total: 18}, exit code 1, all integration tests die at httpx.ConnectError. Integration tests have never been green here. (auditor)
• bandit-report.json shows 3 HIGH severity findings ignored: pyCrypto.AES import (services/core/enterprise_multi_tenant.py:21-22, B413 deprecated/insecure) and MD5-for-security in services/core/test_result_aggregator.py:690 (B324). Plus 22 MEDIUM findings. (auditor)
• shell=True subprocess in services/adapters/node_playwright_adapter.py:457 — partial input control, single shell-injection vector in the repo. (auditor)

P2 — Hygiene / dead code:

• Config has zero model_validator/field_validator; nothing fails closed in production (services/core/qf_config.py). (auditor)
• .env.demo:38,55 literal-default secrets ${POSTGRES_PASSWORD:-changeme_in_production} accepted as values when running docker compose --env-file .env.demo up. (auditor)
• template_tracker ghost Dockerfiles are referenced by nothing — zero hits in any compose / terraform / k8s / shell file. Pure dead weight. (severity)
• Alembic non-hex revision id in create_test_orchestration_schema.py:3 (revision_id = "test_orchestration_001"). Single head computes today; future alembic merge will produce surprising filenames. (auditor)

A.3 Highest-leverage single fix (auditor's top recommendation)

Collapse the JWT identity story to one module before anything else. Delete
create_access_token/decode_access_token from services/api/middleware/auth.py:120-146,
make middleware delegate exclusively to services/database/auth_service.py, and
make QUALITY_FABRIC_JWT_SECRET_KEY the single canonical runtime secret (with
JWT_SECRET accepted as a deprecation alias for one release if needed). ~50 LOC
of deletion that resolves: (a) silent-token-rejection bug, (b) MAESTRO_SECRET_KEY
k8s wiring confusion, (c) the start.sh default-secret risk. Without this, every
other auth/RBAC/tenant-isolation test in the suite is testing a broken substrate.

A.4 Revised effort estimate

Original doc claimed "~2 days of surgical fixes." After agent review: *4–6
working days* for one engineer, broken down as:

• ~2 days: JWT collapse + K8s probe-path/secret reconciliation + container default-entrypoint hardening (P0).
• ~1 day: Prometheus mount, port drift, federated_learning_engine deep-import, frontend Dockerfile rename (P1).
• ~1–2 days: V1–V5 evidence runs, governance files, doc reconciliation, dead-code removal (P2).

This estimate excludes test-suite revival; integration tests being all-red
likely needs another 2–3 days on top.

Tracking — JIRA epic and stories (CON project)

Epic and stories created 2026-05-08. CON project is canonical; the
parallel MD-* tickets (MD-8664..MD-8674) created earlier in the session are
superseded but were not auto-cancelled — owner decides whether to close or
delete them.

Appendix B — Legacy removal: what we'd lose if we delete testing_service_api.py today

In response to the question "can we remove all the legacy services, what
functionality do we lose?", the modular vs legacy diff:

B.1 Functionality the modular app does NOT replicate

B.2 Safe to delete with no caller migration

• services.automation.api_endpoints — referenced at testing_service_api.py:572 but services/automation/ does not exist. The import is already dead.
• Duplicate /api/test-plans, /, /health, /api/health, /api/adapters, /api/metrics/aggregated direct @app routes — modular has equivalents at /api/v1/. Two integration tests target the legacy /api/ paths and would need rewriting (tests/integration/test_ai_integration.py:92,103,119,129,134; tests/integration/test_api_endpoints_integration.py:61,81,86,95).
• The legacy uvicorn entrypoint baked into Dockerfile, Dockerfile.production, Dockerfile.demo, Dockerfile.cicd, Dockerfile.nexus, run_server.py:69 — already covered by CON-3038.

B.3 Recommendation — two-step removal, not a single delete

A naked rm services/api/testing_service_api.py today would silently lose the
entire documented quickstart surface (/api/instances/* and the predictive
endpoints) — the exact deletion-without-callsite-scan anti-pattern the skill's
gate 5/10 names.

Step 1 (~2–3 extra days, must precede the delete):

• Mount instance_router in services/api/main.py (preserve /api/instances path so docs/getting-started/quickstart.md keeps working; also expose under /api/v1/instances going forward).
• Mount testing_execution_router and testing_predictive_router under /api/v1 and add the AI-engine + testing_state setup to modular's lifespan.
• Port OpenTelemetry initialization from testing_service_api.py:339-356 to services/api/main.py.
• Add /api/v1/reports/{execution_id} and /api/v1/categories (or formally deprecate them — they have no live callers in tests/frontend).

Step 2 (covered by CON-3044 once Step 1 lands):

• Delete services/api/testing_service_api.py, run callsite-clearance grep, update the two integration tests.

If CON-3044 is rewritten to
include the four carve-outs above, the legacy can be removed cleanly.

B.4 Revised epic-level effort estimate (with full legacy removal)

The original CON-3034 → CON-3044 list assumed legacy is retired or
feature-flagged, not deleted. Adding full removal lifts the estimate from
4–6 days → 7–9 days for one engineer (additional Step 1 carve-out work).
A new story under CON-3034 should be added for the carve-outs before the
existing CON-3044 (delete) becomes a clean single commit.