Sentinel

The problem

Most LLM demos stop at "the model produced a plausible answer". The harder question starts after that: how do you let an agent act on a system you care about, safely?

Sentinel is an AI engineer that audits a Python codebase and opens fix pull requests. Three specialised agents investigate in parallel, a deterministic pipeline decides what is risky, and for every fixable issue Sentinel generates a patch, validates it against the real codebase and opens a real GitHub PR with the evidence, the reasoning, the fix and the validation results. Nothing ever merges without a human.

The rule: the LLM proposes, code enforces

Only 4 of the 9 nodes in the audit graph call a model: the three analysts and the fix proposer. Everything else is plain, auditable code.

• Risk tiers are assigned by rule, not by model judgment. Security findings are risky, quality and test findings are mechanical. The model cannot decide something is safe enough to skip review.
• Every consequential action (a branch, a commit, a shell command, a file read) goes through code the model can request but never bypass.
• LLM output is never trusted for authorization. Paths are checked for traversal before any read, and a patch is rejected unless its target snippet appears exactly once in the file.

What an audit does

1. Ingest. Clones a repo by GitHub URL or reads a local path, then chunks Python files by function and class using the AST, rather than raw line windows, and embeds the chunks into pgvector.
2. Investigate. A Security analyst runs real semgrep and pip-audit and interprets their output instead of pattern-matching on source. A Quality analyst finds unused imports. A Test analyst finds functions never referenced by any test. Findings stream live to the dashboard over WebSocket.
3. Diagnose. Findings get their risk tier, are deduplicated against near-identical past findings through a semantic cache, and are saved to a queryable history.
4. Propose. One structured LLM call per fixable finding returns a minimal old and new snippet, not a raw diff the model could get line offsets wrong on.
5. Validate and open a PR. The patch is tried locally, checked, reverted, and the PR is created through the GitHub API.

From finding to pull request

Patches are applied to a local clone just long enough to run ruff and pytest, then reverted, so the clone stays a clean read-only mirror. The branch, commit and PR happen entirely through the GitHub API, so no push credentials are needed beyond a token.

Mechanical fixes open as normal PRs. Risky security fixes open as drafts labelled needs-security-review, whatever the validation result, because a draft PR resists accidental fast-merging in a way a normal PR does not. There is simply no code path in Sentinel that merges anything.

Bugs found, and what they changed

The design notes in the repo are a log of real problems, and several of them shaped the safety rails.

• An effectively unbounded agent loop. The Security analyst spun through 50+ tool calls with no error and no stop. The installed LangGraph's default recursion limit turned out to be 10007. Every agent now runs with an explicit step limit, and a per-audit cost cap backs it up.
• A correct reason, an incorrect finding. The Quality analyst reported two security-looking constants as unused, while its own explanation said that was out of scope. Sentinel opened two wrong PRs proposing to delete them. Nothing was harmed, since both sat open and unmerged, which is exactly what the human-review boundary is for. The prompt now carries a hard scope constraint, and the incident stays in the record as the argument for the PR-only design.
• Duplicate PRs on re-runs. The dedup key included free text from the LLM, which is never reproduced byte for byte. It is now keyed on analyst, file and symbol only. Repeat live runs show every previously fixed finding skipped and zero new PRs.
• Lint as an all-clean gate would have blocked valid fixes. Real repos carry unrelated lint debt. Validation now compares the error count before and after the patch, so a fix passes if it does not make things worse.
• A pasted GitHub URL is untrusted input. git's own URL syntax includes an ext:: transport that runs arbitrary shell commands. URLs are validated against a strict pattern before they ever reach a subprocess, and branch names that could be read as flags are rejected.

Production hardening

• Spend caps that actually stop the run. A callback raises the moment a run's cost crosses the cap, rather than logging it afterwards. A daily cap is checked before any audit starts. Combined with a durable checkpointer, a cutoff mid-run still keeps the findings already committed.
• A kill switch held in Redis and checked before every analyst call and before propose, validate and PR.
• A narrow distributed lock. Only the git-writing step is locked per repo. Analysis is read-only and runs fully concurrently, so a repo-wide lock would have cost speed for no extra safety. Verified with real concurrent worker processes and no duplicate PRs.
• Fallbacks. A model failure retries once on a cheaper model, and failures in network tools retry with exponential backoff.
• Two caches. A content-hash cache means a repeat ingest of an unchanged repo makes zero embedding calls. A semantic cache dedups near-identical findings by cosine similarity.
• A task queue with a horizontally scalable worker pool, and a Postgres-backed checkpointer for recovery if a worker dies.

The dashboard

A Next.js app is the product surface: a live multi-agent timeline, a findings table with expandable diffs and validation status, an audit-history trend chart, and a kill switch wired to the real backend flag.

Stack

LangGraph and LangChain for orchestration, OpenAI models (a mid-size model for the analysts and fixer, a smaller one as fallback), Postgres with pgvector, Redis for the queue, caches, locks and budgets, LangSmith for tracing, and Prometheus with Grafana for metrics. The frontend uses Next.js, TypeScript, Tailwind, shadcn/ui and Recharts, and the infrastructure runs under Docker Compose.

Limits

The audit scope is Python only, because the real security tools used are Python-native. Writing new tests is deliberately left out of auto-fix, since a trustworthy test is a harder problem than a scoped snippet replacement. The deployment is a written plan and nothing has been deployed to the cloud.