Evidence
What needs checking against a control
Orchestrator agent
Drafts a finding, pulling controls through MCP
Adversarial auditor agent
Cross-examines each finding on its own
Human approval gate
LangGraph interrupt() holds the run until a person approves
- compliance-kb-mcppgvector embeddings plus BM25 keyword ranking over 324 NIST 800-53 controls
- Promptfoo + RAGASCI fails on any drop of more than 2% across 100 hand-labeled cases
- Langfuse + OpenTelemetryEvery agent run traced, prompt versions pulled at runtime
- 5
- MCP servers published
- 551
- tests green at last full run
- 100
- hand-labeled eval cases
- 16
- architecture decision records
- Published compliance-kb-mcp to PyPI and npm. It finds the right control among 324 NIST 800-53 controls with hybrid RAG retrieval, pairing pgvector embeddings with BM25 keyword ranking.
- Gated GitHub Actions CI on a Promptfoo and RAGAS eval suite of 100 hand-labeled cases, so any change that drops agent quality more than 2% fails the build.
- Traced every agent run in Langfuse and OpenTelemetry, with prompts versioned in Langfuse and pulled at runtime behind a local fallback.
- Ran the build through a spec-driven Claude Code workflow: sub-agent reviewers for architecture, security, database, and compliance language, plus hooks that block destructive git commands.
- Documented the major decisions in 16 ADRs so another team building human-gated, multi-agent compliance systems could adopt the pattern directly.
Stack
- LangGraph
- Pydantic AI
- MCP
- FastAPI
- Next.js
- Vercel AI SDK
- Neon Postgres
- pgvector
- Promptfoo
- RAGAS
- Langfuse
- OpenTelemetry
- Cloud Run