# Shopify's app security check, on every PR

> Shopify now ships a security scanner in the CLI. Wire it into GitHub Actions so it blocks risky merges on its own, and hand its judgment calls to an agent before you release.

**Field Guide** · **Updated:** 2026-10-15 · **Tags:** ci-cd, github-actions, shopify-cli, security-review, ai, automation

_Source: https://verybadpanda.dev/field-guide/shopify-app-security-check-ci_

---

![Bandit the masked raccoon perched on a towering stack of paper, squinting through a magnifying glass at a page of code held up in one paw](https://verybadpanda.dev/assets/bandit-reading-code.webp)

*Shopify handed Bandit the checklist. Now he reads every page of your app before it ships, on every PR.*

Shopify has recently released a [dedicated security skill](https://shopify.dev/changelog/blog/build-more-secure-apps-with-shopify-s-ai-toolkit-security-check)
in their AI Toolkit. You can run it by hand with the `/shopify` slash command, but let's automate
it in our CI/CD, like we did with a similar dedicated tool, [Strix](https://verybadpanda.dev/field-guide/ai-security-harness-ci).

## How the check works

It has two halves, and they want different jobs in CI:

| | **Deterministic scan** | **Agent checks** |
|---|---|---|
| What it does | Matches known-bad patterns in code and config | Investigates what a pattern can't settle (missing authz, tenant isolation) |
| Same code, same answer? | Yes | No, an LLM reasons about it |
| Cost | Free, seconds | Tokens, minutes |
| Needs a secret? | No | An LLM key (or a cloud IAM role) |
| Writes | `deterministic-findings.json` + `agent-checks.json` | `agent-findings.json`, via `shopify app security record` |
| Run it | Every PR | On release, or when a label says so |

Everything lands in `.shopify/app-security/<results key>/`. The results key is your app config
name without `.toml` (or `--client-id`, if you pass one). Then `shopify app security review`
merges both halves into one report.

Three details make it a good fit for CI:

- **No login.** The scan runs on the runner and never calls home for auth. The exception is
  `--client-id`, which is checked against your Shopify account.
- **You choose what fails the build.** By default it *never* fails on findings. Pass
  `--blocking high` (or `medium`, or `low`) and it exits `1` when something meets the bar.
- **It doesn't prompt in CI.** On a non-interactive terminal it skips the "copy the agent
  instructions?" prompt, unless you ask with `--print-instructions`.

```mermaid
flowchart TD
    PR([PR opened / pushed]) --> SCAN[shopify app security check<br/>--blocking high]
    SCAN -->|exit 1: high finding| BLOCK[Merge blocked]:::bad
    SCAN -->|exit 0| MERGE[Merge allowed]:::good
    PR --> LBL{security-review label<br/>or release?}
    LBL -->|no| SKIP[Agent review skipped]:::good
    LBL -->|yes| AGENT[Agent runs agent-checks.json<br/>→ security record]
    AGENT --> REVIEW[shopify app security review<br/>--blocking high]
    REVIEW -->|high from scan or agent| BLOCK
    REVIEW -->|clean or advisory| MERGE
    classDef bad fill:#1a0f0f,stroke:#FF5F56,color:#FF5F56;
    classDef good fill:#0d1a12,stroke:#7CF9A0,color:#7CF9A0;
```

## Before you wire it up

Run it locally once, from your app directory, so CI doesn't surprise you:

```bash
npm install -g @shopify/cli@latest   # the check ships in 4.9.4+
shopify app security check
```

Fix every high before you turn on blocking. Otherwise, the first PR to run it gets blamed for
someone else's debt.

You don't need to touch `.gitignore`: the CLI writes its own `.shopify/.gitignore` that keeps
the results out of Git.

A few repo shapes need a flag:

- **More than one `shopify.app*.toml`?** CI can't stop and ask which one. Pass `--config production`
  (or whichever), and run one job per config if you ship several.
- **Backend outside the app directory?** Add it with `--include-dir ../backend`. Repeat the flag
  for more directories.
- **Generated code or fixtures making noise?** `--exclude '**/generated' --exclude 'test/fixtures/**'`.
  Quote the globs, or your shell expands them first.
- **Not sure what it'll read?** `--list-files` prints the file list and stops.

The check uses Git to respect `.gitignore` and needs Git 2.38 or later. GitHub's hosted runners
already have it.

Watch the **Coverage gaps** section of the report too. Files over the size limit are skipped
with "The file was not inspected because it was too large." That's usually bundles and
generated code, but check, because a skipped file is an unscanned file.

## Job 1: the scan on every PR

```yaml
# .github/workflows/shopify-app-security.yml
name: shopify-app-security

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review, labeled]
  push:
    branches: [main]
  release:
    types: [published]
  workflow_dispatch:

permissions:
  contents: read

jobs:
  scan:
    name: shopify-security-scan
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies   # the CLI loads your extensions before it scans
        run: npm ci                  # or pnpm install --frozen-lockfile, etc.

      - name: Install Shopify CLI
        run: npm install -g @shopify/cli@4.9.4

      - name: Deterministic scan
        run: shopify app security check --no-color --skip-instructions --blocking high

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: shopify-security-scan-${{ github.run_id }}
          path: .shopify/app-security/
          include-hidden-files: true
          retention-days: 14
```

What each piece is doing:

- **Install dependencies first.** The check loads your whole app before it scans, including
  type checks on UI extensions. Skip the install and it dies with `Type reference for
  admin.order-details.block.render could not be found`, which looks like a version problem
  but isn't. Use whatever install your repo already runs; `--ignore-scripts` is fine.
- `--blocking high` makes the job fail on a high finding. That exit code is the whole gate.
- `--skip-instructions` skips the agent instructions, because this job never runs an agent.
- `include-hidden-files: true` matters because `.shopify/` is a dotfolder, and `upload-artifact`
  silently skips those by default. Without this flag, you get an empty artifact.
- **Pin the CLI version.** New CLI releases add checks, and a new check can fail a build that
  passed yesterday. Bump the pin on purpose, in its own PR.

Mark **shopify-security-scan** as a required check in your branch ruleset.

<Callout level="info" title="Why this gate doesn't fail open">
The [Strix gate](https://verybadpanda.dev/field-guide/ai-security-harness-ci#turn-findings-into-issues--and-a-merge-gate)
fails *open* when the scanner errors, because you don't want merges held hostage by a model
provider's uptime. This scan has no provider. It's local and deterministic, and it runs every
time. If it fails, it found something (or your CLI install broke, and you want to know that
too). Let it block.
</Callout>

## Job 2: the agent checks, before you ship

Some checks can't be settled with pattern matching, like "does this route actually confirm the
caller owns this shop?" The CLI writes those to `agent-checks.json`, along with a full set of
instructions for whatever agent you hand them to. You don't need the Shopify AI Toolkit in CI:
`shopify app security instructions` prints the same workflow.

This job uses the [Claude Code GitHub Action](https://github.com/anthropics/claude-code-action).
Add it to the same workflow:

```yaml
  agent-review:
    name: shopify-security-agent-review
    if: >
      github.event_name == 'release' ||
      github.event_name == 'workflow_dispatch' ||
      (github.event.pull_request.draft == false &&
       contains(github.event.pull_request.labels.*.name, 'security-review'))
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies
        run: npm ci

      - name: Install Shopify CLI
        run: npm install -g @shopify/cli@4.9.4

      - name: Agent checks
        uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          prompt: |
            Review this Shopify app for security issues.

            Run `shopify app security instructions` and follow the workflow it prints,
            step by step. Record your findings with `shopify app security record`.

            Report issues only. Don't edit any file in the app.
          claude_args: >-
            --allowedTools "Bash(shopify app security:*),Bash(cat:*),Read,Glob,Grep,Write"

      - name: Combined review
        shell: bash   # turns on pipefail, so tee can't swallow the exit code
        run: shopify app security review --no-color --blocking high | tee "$RUNNER_TEMP/review.txt"

      - name: Job summary
        if: always()
        shell: bash
        run: |
          {
            echo '## Shopify app security review'
            echo '```'
            cat "$RUNNER_TEMP/review.txt" 2>/dev/null || echo 'No review output.'
            echo '```'
          } >> "$GITHUB_STEP_SUMMARY"

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: shopify-security-review-${{ github.run_id }}
          path: .shopify/app-security/
          include-hidden-files: true
          retention-days: 90
```

How it decides:

- **The agent doesn't gate anything.** It writes a findings document. `record` validates it
  all-or-nothing and rejects anything malformed. Then `review --blocking high` decides, using
  the scan's findings *and* the agent's. The exit code still comes from Shopify's CLI, not from
  the agent's opinion of its own work.
- **`shell: bash` isn't decoration.** GitHub's default shell doesn't set `pipefail`, so
  `review | tee` would report `tee`'s success and wave a high finding through.
- **It runs on release, on demand, or on a `security-review` label.** That's the same opt-in
  logic as the [Strix quick pass](https://verybadpanda.dev/field-guide/ai-security-harness-ci#two-scans-not-one): most
  PRs don't touch the trust boundary, and tokens aren't free.

<Callout level="crit" title="The agent is reading attacker-controlled code">
The agent reads the PR's code, and that code can contain text written to steer it. So keep the
tool list tight. This one can run `shopify app security` commands and read files. It can write
(the findings document needs it), but it can't run arbitrary shell, push, or comment. Keep the
`pull_request` trigger, so fork PRs never get your key. Never swap in `pull_request_target`. The
[Harden the harness](https://verybadpanda.dev/field-guide/ai-security-harness-ci#harden-the-harness) checklist applies
here too: pin actions to SHAs, use least-privilege `permissions:`, and pass PR fields through
`env:`.
</Callout>

Same advice on models and hosting as the Strix guide: the
[inference host](https://verybadpanda.dev/field-guide/ai-security-harness-ci#choosing-the-inference-host) decides
where your source code goes. The Claude Code Action can reach Claude through Amazon Bedrock or
Vertex AI instead of a raw API key. This job is code *review*, not an offensive pentest, so the
[guardrail trap](https://verybadpanda.dev/field-guide/ai-security-harness-ci#the-guardrail-trap-your-pentester-may-refuse-to-pentest)
bites less often here. Still, test the model you pin against your own repo.

## Where it fits next to Strix

These aren't competing tools. They catch different things:

| | Shopify's check | [Strix](https://verybadpanda.dev/field-guide/ai-security-harness-ci) |
|---|---|---|
| Knows Shopify | Deeply: app proxies, webhooks, session tokens, theme extensions, scopes | Generically: it's a web app pentester |
| Proves the bug | No, it points at a file and line | Yes, runs a real proof-of-concept exploit |
| Deterministic layer | Yes | No |
| Cost per PR | Free (scan) / tokens (agent) | Tokens, budget-capped |
| Best at | Known Shopify footguns, before they merge | Logic and auth bugs a pattern can't see |

Run Shopify's scan on every PR as the floor. Run Strix on labeled PRs and on a schedule as the
hunter. When either one confirms a real bug, write a
[deterministic test](https://verybadpanda.dev/field-guide/authorization) so it can't come back.

<Callout level="warn" title="What a clean report doesn't mean">
A passing check is evidence, not proof, and Shopify says the same. Some deterministic checks
only run on apps built from the React Router template. On Rails, Laravel, Django, or custom
stacks, those checks move to the agent, so Job 2 matters more for you. And it only reads code.
It can't see what your server sends to a storefront at runtime. A "custom JS" setting that's
empty in your repo and full of malware in your database passes a code scan.
</Callout>

## The checklist

- [ ] Ran `shopify app security check` locally and fixed every high before turning on blocking
- [ ] **shopify-security-scan** runs on every PR with `--blocking high`, and it's a required check
- [ ] Shopify CLI is **pinned**, and bumps land in their own PR
- [ ] `upload-artifact` has `include-hidden-files: true`
- [ ] `--config` is set if the repo has more than one `shopify.app*.toml`
- [ ] Agent review runs on **release**, **manual dispatch**, or a **`security-review` label**
- [ ] The agent's tools are allowlisted, and the review step uses `shell: bash`
- [ ] Fork PRs never get the key (`pull_request`, not `pull_request_target`)
- [ ] [Strix](https://verybadpanda.dev/field-guide/ai-security-harness-ci) still runs on top, for the bugs a pattern can't find

## References

- [Build more secure apps with Shopify's AI Toolkit security check](https://shopify.dev/changelog/blog/build-more-secure-apps-with-shopify-s-ai-toolkit-security-check). Shopify's launch post, from October 15, 2026.
- [App security checks](https://shopify.dev/docs/apps/build/security/app-security-checks). The workflow, scan scope, and every check ID by severity, each with its own fix page.
- [Shopify AI Toolkit](https://shopify.dev/docs/apps/build/ai-toolkit). Install it for local use, so `/shopify Check my app for security vulnerabilities` works in your editor.
- [App security engine source](https://github.com/Shopify/cli/blob/main/packages/app/src/cli/services/app-security-engine/). The check is open source, so read the rules it runs.
- [Claude Code GitHub Action](https://github.com/anthropics/claude-code-action). The action that runs the agent checks in Job 2.
- [GitHub Actions secure use reference](https://docs.github.com/en/actions/reference/security/secure-use-reference). Covers script injection, token permissions, and the `pull_request_target` trap.
- [Security reviews in CI: an AI pentester on every PR](https://verybadpanda.dev/field-guide/ai-security-harness-ci). Our Strix guide, which covers the hunter that runs on top of this floor.