Runbooks¶
Operational procedures, incident playbooks, and governance runbooks for Private AI Platform Kit. Start with the incident-response index during an incident, or upgrade for the most common day-2 change.
Setup & day-2 operations¶
| Runbook | Purpose |
|---|---|
| Validation toolchain | Install and verify the local/strict validation CLIs |
| Upgrade & rollback | Promote a new release tag, test before promote, and roll back |
| GPU capacity | GPU scheduling diagnostics and capacity validation |
| Policy-blocked deploy | Diagnose and resolve Kyverno-blocked deployments |
| Agent workspaces | Design and operate locked-down coding-agent workspaces |
| Tenant labs | Onboard a tenant lab with quotas, egress, and isolation |
| API access | API-key access model and rotation |
| External / managed stores | Swap the bundled Redis, Qdrant, and Loki footprints for external/HA services |
Runtime, RAG & guardrails¶
| Runbook | Purpose |
|---|---|
| RAG service | Operate the RAG retrieval service |
| Vector RAG | The Qdrant vector-store retrieval profile |
| Qdrant migration | Collection migration with dry-run and rollback |
| Guardrails | Prompt and secret-detection guardrails |
| Traceability sandbox | Sandbox trace contract and request correlation |
| Budget controls | Redis-backed sandbox spend/abuse budgets |
| Evaluation harness | Run and interpret eval suites |
| End-user chat UI | Put a self-hosted Open WebUI chat frontend in front of the gateway |
| Async batch API | Operate the async Files + Batch API and the batch-processor worker |
Governance & evidence¶
| Runbook | Purpose |
|---|---|
| Audit chain & SIEM forwarding | Verify the tamper-evident audit hash chain, anchor its head, and forward receipts to a SIEM |
| Model governance | Model lifecycle, promotion requests, and approval |
| Model provenance | Artifact provenance and reproducible digest verification |
| Model drift monitoring | Detect production model-quality drift via metrics and scheduled evals |
| Evidence pack | Customer-facing evidence pack generation |
| Release gates | Release-gate thresholds and strict evidence |
| SLO & error budget | SLO objectives and error-budget review |
| Quota & chargeback | Quota plans and chargeback labelling |
| Data retention | Retention and privacy controls |
| Egress governance | Approved-egress catalog and enforcement |
| Scorecard triage | OpenSSF Scorecard finding triage |
Incidents & resilience¶
| Runbook | Purpose |
|---|---|
| Incident response | Severity tiers, escalation, and the incident index |
| Disaster recovery | Single-cluster DR: RPO/RTO, restore order, operator-owned scope |
| Failure modes & degradation | Consolidated dependency failure-mode and graceful-degradation matrix |
| Inference runtime incident | Gateway / vLLM / Ollama runtime outages |
| Chaos drills | Rollout/recovery and fault-injection drills |
| Restore drill | Backup restore-tooling smoke and real data-recovery drill |
| Runtime threat detection | Optional Falco/Tetragon detective layer for hijacked agents |
| OIDC / JWKS rotation | Identity-provider key rotation |
For the documentation map (setup, customer handoff, contracts), see docs/README.md.