AuditPilot · Case study
How AuditPilot works
A SOC 2 readiness copilot that reads, drafts, and never touches
AuditPilot checks how a software company protects customer data, shows which SOC 2 controls pass or fail, and drafts the policies and questionnaire answers that are missing. Only a CPA firm can issue the real report, so this is the practice run before it. It reads GitHub with a read-only key, and a person approves every draft.
- Open source, Apache 2.0
- 2 agents and 1 human gate
- 5 MCP servers
- 324 NIST controls
- 16 decision records
It does
- Reads your GitHub settings with a read-only key
- Shows which controls pass, partly pass, or fail
- Drafts policies and fills a questionnaire for you to download
- Queues suggested fixes in Pending Actions
It never
- Changes a setting in GitHub or any other tool
- Sends email or posts to Slack
- Submits a questionnaire to the customer
- Issues the SOC 2 report, which only a CPA firm can
01The parts
What the system is made of
A Next.js app, a FastAPI server with two agents and an approval gate, and five MCP servers behind them. Each part is listed with the name it has in the code.
In the browser
Web app
Next.js + React
Sign-in, the repo picker, the live control scoreboard, the Pending Actions list, and the review screen for drafts. Results stream in as the agents produce them.
On the server
API
FastAPI + SSE
Takes requests from the web app, queues slow jobs, and streams results back over server-sent events as each one is ready.
Organizer
AuditOrchestrator
The main agent, built on LangGraph and Pydantic AI. It calls the five tools, matches evidence to controls, and writes the drafts. It is the only agent allowed to write to the shared run state.
Skeptic
AdversarialAuditor
A second agent that plays a tough reviewer in a mock readiness challenge. It runs as its own service, can only read, and has a $0.50 budget cap per run. The Organizer reaches it over A2A.
Approval gate
HumanReviewGate
Pauses the run with a LangGraph interrupt() and saves its place through the PostgresSaver checkpointer. It can wait hours or days. Approve, edit, or reject resumes the run.
Scheduler
cron, every 6 hours
Starts a drift check on a timer, so nobody has to click anything.
Tools the server calls
GitHub
read-only OAuth
The source of evidence. The key has read-only scopes, so AuditPilot can see settings but cannot change them.
Gemini
Google's LLM
The model behind both agents.
Rulebook
compliance-kb
324 NIST 800-53 controls, plus a map from them to SOC 2's five trust categories. Retrieval pairs pgvector embeddings with BM25 keyword ranking. Published on PyPI and npm.
Evidence store
evidence-store
Saves what each scan found and searches it by meaning, not only by exact words.
Questionnaire filler
questionnaire
Opens the customer's spreadsheet, groups similar questions, and writes each answer back with its source cited. Low-confidence answers get flagged for a person.
Policy templates
policy-template
Policy templates with blank slots for citations, so each draft points at its evidence.
Drift watcher
drift-watcher
Compares today's snapshot with the last one and spots anything that quietly got worse.
Storage
Postgres, Redis, R2
Postgres with pgvector holds findings and run checkpoints, and row-level security keeps each user's rows private. Redis runs the queue for slow jobs. R2 holds the generated files people download.
02Walkthrough
Follow a run
Pick a scenario, then step through it. Filled boxes are working at the current step, and outlined boxes already did their part.
What starts it: You connect GitHub, pick repos, and start a scan
Step 1 of 6
Connect GitHub and pick the repos
The user signs in and grants a read-only GitHub key. Nothing is in scope until they choose repos, so the picker starts empty. ADR-0015 made that default-deny rule.
In the browser
- The userengineer or compliance lead, working at this step
- Web appNext.js + React, working at this step
On the server
- APIFastAPI + SSE
- OrganizerAuditOrchestrator
- SkepticAdversarialAuditor
- Approval gateHumanReviewGate
- Schedulercron, every 6 hours
Tools the server calls
- GitHubread-only OAuth
- GeminiGoogle's LLM
- Rulebookcompliance-kb
- Evidence storeevidence-store
- Questionnaire fillerquestionnaire
- Policy templatespolicy-template
- Drift watcherdrift-watcher
- StoragePostgres, Redis, R2
03Decisions
Four choices that shaped it
Read in, drafts out, a person decides
The GitHub key is read-only, and nothing in the system can write to a customer tool. A bug in a tool that can write could switch off a real control, and write scopes slow down every security review. Compliance calls need someone's judgment anyway. So AuditPilot points at problems and queues fixes, and people apply them.
Two agents and a gate, no more
The Organizer is the only agent that writes to shared state. The Skeptic can only read, and it runs as a separate service with a $0.50 cap per run. Essays from Anthropic, Cognition, and OpenAI all warn that crowds of agents step on each other. Adding any other agent needs a written decision record first.
Tools as standalone MCP servers
The five tools are separate packages, not functions buried in the app. Anything that speaks MCP can use them. compliance-kb is already on PyPI and npm.
Nothing is in scope until you pick it
The repo picker starts empty, and AuditPilot only reads the repos a person chooses. Postgres row-level security keeps each user's records private, so one account can't see another's.
04Checks
How I know it works
Most of this runs without me. None of it replaces the approval gate, which stays in place for anything the agents draft.
Tests
The suite had 551 tests passing at the last full run.
Reviewer agents
Six specialist reviewers read the code before each sprint closed, covering architecture, security, the database, and compliance language. In Sprint 10 they caught one of my own scripts about to open GitHub issues. That is a write action and breaks the rule, so I removed it.
CI
GitHub Actions reruns every check on each change, so a mistake cannot slip in quietly. The eval gate runs in the same pipeline.
Evals
A Promptfoo and RAGAS suite of 100 hand-labeled cases checks that evidence still maps to the right controls. CI fails any change that drops quality by more than 2%.
Judge exam
When one model grades another, the grader needs a test too. Humans labeled 50 cases, and I measure the judge against them: how often it catches real problems (TPR), how often it avoids false alarms (TNR), and how well it agrees with the humans beyond luck (Cohen's kappa).
Traces
Langfuse and OpenTelemetry trace every agent run, so a bad result can be replayed. Prompts are versioned in Langfuse and pulled at runtime, with a local fallback.
05Delivery
Built in layers
The work ran as numbered sprints, from the blueprint in Sprint 0 to the eval suite in Sprint 10, with two half sprints added along the way. Each sprint put one layer on top of the last.
Foundations
Sprint 0 · Blueprint
Product and system design docs, plus the first batch of ADRs.
Sprint 1 · Rulebook
The compliance-kb server, published.
Sprint 2 · Engine room
The backend, live streaming, a job queue for slow work, and monitoring.
Sprint 3 · Front door
The web app, sign-in, and the read-only GitHub connection.
Sprint 3.5 · Guardrail
A repo picker where nothing is in scope until you choose it.
The audit loop
Sprint 4 · First scan
The Organizer wired to the screen, the live control scoreboard, and the Pending Actions list.
Sprint 5 · Evidence
Real GitHub checks, evidence saved and searchable, and a cache so unchanged evidence skips the model call.
Sprint 6 · Policies
The approval gate and four draft policies.
Sprint 6.5 · Make it readable
A sidebar, an overview page, click-to-explain control details, and dark mode.
Sprint 7 · Questionnaires
Upload a spreadsheet, download it back filled in with cited answers.
Challenge, watch, prove
Sprint 8 · The Skeptic
The adversarial reviewer as its own service, and one command, make docker-up, that starts the whole system.
Sprint 9 · Watchtower
Drift detection every 6 hours, plus re-run, compare, and revert for scans.
Sprint 10 · Proof
The eval suite, the judge exam, versioned prompts, and the automatic checks on GitHub.