Shopify's app security check, on every PR

Bandit the masked raccoon perched on a towering stack of paper, squinting through a magnifying glass at a page of code held up in one paw — Shopify handed Bandit the checklist. Now he reads every page of your app before it ships, on every PR.

Shopify has recently released a dedicated security skill in their AI Toolkit. You can run it by hand with the /shopify slash command, but let’s automate it in our CI/CD, like we did with a similar dedicated tool, Strix.

How the check works#

It has two halves, and they want different jobs in CI:

	Deterministic scan	Agent checks
What it does	Matches known-bad patterns in code and config	Investigates what a pattern can’t settle (missing authz, tenant isolation)
Same code, same answer?	Yes	No, an LLM reasons about it
Cost	Free, seconds	Tokens, minutes
Needs a secret?	No	An LLM key (or a cloud IAM role)
Writes	`deterministic-findings.json` + `agent-checks.json`	`agent-findings.json`, via `shopify app security record`
Run it	Every PR	On release, or when a label says so

Everything lands in .shopify/app-security/<results key>/. The results key is your app config name without .toml (or --client-id, if you pass one). Then shopify app security review merges both halves into one report.

Three details make it a good fit for CI:

No login. The scan runs on the runner and never calls home for auth. The exception is --client-id, which is checked against your Shopify account.
You choose what fails the build. By default it never fails on findings. Pass --blocking high (or medium, or low) and it exits 1 when something meets the bar.
It doesn’t prompt in CI. On a non-interactive terminal it skips the “copy the agent instructions?” prompt, unless you ask with --print-instructions.

flowchart TD
    PR([PR opened / pushed]) --> SCAN[shopify app security check<br/>--blocking high]
    SCAN -->|exit 1: high finding| BLOCK[Merge blocked]:::bad
    SCAN -->|exit 0| MERGE[Merge allowed]:::good
    PR --> LBL{security-review label<br/>or release?}
    LBL -->|no| SKIP[Agent review skipped]:::good
    LBL -->|yes| AGENT[Agent runs agent-checks.json<br/>→ security record]
    AGENT --> REVIEW[shopify app security review<br/>--blocking high]
    REVIEW -->|high from scan or agent| BLOCK
    REVIEW -->|clean or advisory| MERGE
    classDef bad fill:#1a0f0f,stroke:#FF5F56,color:#FF5F56;
    classDef good fill:#0d1a12,stroke:#7CF9A0,color:#7CF9A0;

Before you wire it up#

Run it locally once, from your app directory, so CI doesn’t surprise you:

npm install -g @shopify/cli@latest   # the check ships in 4.9.4+
shopify app security check

Fix every high before you turn on blocking. Otherwise, the first PR to run it gets blamed for someone else’s debt.

You don’t need to touch .gitignore: the CLI writes its own .shopify/.gitignore that keeps the results out of Git.

A few repo shapes need a flag:

More than one shopify.app*.toml? CI can’t stop and ask which one. Pass --config production (or whichever), and run one job per config if you ship several.
Backend outside the app directory? Add it with --include-dir ../backend. Repeat the flag for more directories.
Generated code or fixtures making noise? --exclude '**/generated' --exclude 'test/fixtures/**'. Quote the globs, or your shell expands them first.
Not sure what it’ll read? --list-files prints the file list and stops.

The check uses Git to respect .gitignore and needs Git 2.38 or later. GitHub’s hosted runners already have it.

Watch the Coverage gaps section of the report too. Files over the size limit are skipped with “The file was not inspected because it was too large.” That’s usually bundles and generated code, but check, because a skipped file is an unscanned file.

Job 1: the scan on every PR#

# .github/workflows/shopify-app-security.yml
name: shopify-app-security

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review, labeled]
  push:
    branches: [main]
  release:
    types: [published]
  workflow_dispatch:

permissions:
  contents: read

jobs:
  scan:
    name: shopify-security-scan
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies   # the CLI loads your extensions before it scans
        run: npm ci                  # or pnpm install --frozen-lockfile, etc.

      - name: Install Shopify CLI
        run: npm install -g @shopify/[email protected]

      - name: Deterministic scan
        run: shopify app security check --no-color --skip-instructions --blocking high

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: shopify-security-scan-${{ github.run_id }}
          path: .shopify/app-security/
          include-hidden-files: true
          retention-days: 14

What each piece is doing:

Install dependencies first. The check loads your whole app before it scans, including type checks on UI extensions. Skip the install and it dies with Type reference for admin.order-details.block.render could not be found, which looks like a version problem but isn’t. Use whatever install your repo already runs; --ignore-scripts is fine.
--blocking high makes the job fail on a high finding. That exit code is the whole gate.
--skip-instructions skips the agent instructions, because this job never runs an agent.
include-hidden-files: true matters because .shopify/ is a dotfolder, and upload-artifact silently skips those by default. Without this flag, you get an empty artifact.
Pin the CLI version. New CLI releases add checks, and a new check can fail a build that passed yesterday. Bump the pin on purpose, in its own PR.

Mark shopify-security-scan as a required check in your branch ruleset.

INFO Why this gate doesn't fail open

The Strix gate fails open when the scanner errors, because you don’t want merges held hostage by a model provider’s uptime. This scan has no provider. It’s local and deterministic, and it runs every time. If it fails, it found something (or your CLI install broke, and you want to know that too). Let it block.

Job 2: the agent checks, before you ship#

Some checks can’t be settled with pattern matching, like “does this route actually confirm the caller owns this shop?” The CLI writes those to agent-checks.json, along with a full set of instructions for whatever agent you hand them to. You don’t need the Shopify AI Toolkit in CI: shopify app security instructions prints the same workflow.

This job uses the Claude Code GitHub Action. Add it to the same workflow:

  agent-review:
    name: shopify-security-agent-review
    if: >
      github.event_name == 'release' ||
      github.event_name == 'workflow_dispatch' ||
      (github.event.pull_request.draft == false &&
       contains(github.event.pull_request.labels.*.name, 'security-review'))
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies
        run: npm ci

      - name: Install Shopify CLI
        run: npm install -g @shopify/[email protected]

      - name: Agent checks
        uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          prompt: |
            Review this Shopify app for security issues.

            Run `shopify app security instructions` and follow the workflow it prints,
            step by step. Record your findings with `shopify app security record`.

            Report issues only. Don't edit any file in the app.
          claude_args: >-
            --allowedTools "Bash(shopify app security:*),Bash(cat:*),Read,Glob,Grep,Write"

      - name: Combined review
        shell: bash   # turns on pipefail, so tee can't swallow the exit code
        run: shopify app security review --no-color --blocking high | tee "$RUNNER_TEMP/review.txt"

      - name: Job summary
        if: always()
        shell: bash
        run: |
          {
            echo '## Shopify app security review'
            echo '```'
            cat "$RUNNER_TEMP/review.txt" 2>/dev/null || echo 'No review output.'
            echo '```'
          } >> "$GITHUB_STEP_SUMMARY"

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: shopify-security-review-${{ github.run_id }}
          path: .shopify/app-security/
          include-hidden-files: true
          retention-days: 90

How it decides:

The agent doesn’t gate anything. It writes a findings document. record validates it all-or-nothing and rejects anything malformed. Then review --blocking high decides, using the scan’s findings and the agent’s. The exit code still comes from Shopify’s CLI, not from the agent’s opinion of its own work.
shell: bash isn’t decoration. GitHub’s default shell doesn’t set pipefail, so review | tee would report tee’s success and wave a high finding through.
It runs on release, on demand, or on a security-review label. That’s the same opt-in logic as the Strix quick pass: most PRs don’t touch the trust boundary, and tokens aren’t free.

CRITICAL The agent is reading attacker-controlled code

The agent reads the PR’s code, and that code can contain text written to steer it. So keep the tool list tight. This one can run shopify app security commands and read files. It can write (the findings document needs it), but it can’t run arbitrary shell, push, or comment. Keep the pull_request trigger, so fork PRs never get your key. Never swap in pull_request_target. The Harden the harness checklist applies here too: pin actions to SHAs, use least-privilege permissions:, and pass PR fields through env:.

Same advice on models and hosting as the Strix guide: the inference host decides where your source code goes. The Claude Code Action can reach Claude through Amazon Bedrock or Vertex AI instead of a raw API key. This job is code review, not an offensive pentest, so the guardrail trap bites less often here. Still, test the model you pin against your own repo.

Where it fits next to Strix#

These aren’t competing tools. They catch different things:

	Shopify’s check	Strix
Knows Shopify	Deeply: app proxies, webhooks, session tokens, theme extensions, scopes	Generically: it’s a web app pentester
Proves the bug	No, it points at a file and line	Yes, runs a real proof-of-concept exploit
Deterministic layer	Yes	No
Cost per PR	Free (scan) / tokens (agent)	Tokens, budget-capped
Best at	Known Shopify footguns, before they merge	Logic and auth bugs a pattern can’t see

Run Shopify’s scan on every PR as the floor. Run Strix on labeled PRs and on a schedule as the hunter. When either one confirms a real bug, write a deterministic test so it can’t come back.

WARN What a clean report doesn't mean

A passing check is evidence, not proof, and Shopify says the same. Some deterministic checks only run on apps built from the React Router template. On Rails, Laravel, Django, or custom stacks, those checks move to the agent, so Job 2 matters more for you. And it only reads code. It can’t see what your server sends to a storefront at runtime. A “custom JS” setting that’s empty in your repo and full of malware in your database passes a code scan.

The checklist#

Ran shopify app security check locally and fixed every high before turning on blocking
shopify-security-scan runs on every PR with --blocking high, and it’s a required check
Shopify CLI is pinned, and bumps land in their own PR
upload-artifact has include-hidden-files: true
--config is set if the repo has more than one shopify.app*.toml
Agent review runs on release, manual dispatch, or a security-review label
The agent’s tools are allowlisted, and the review step uses shell: bash
Fork PRs never get the key (pull_request, not pull_request_target)
Strix still runs on top, for the bugs a pattern can’t find

References#

Build more secure apps with Shopify’s AI Toolkit security check. Shopify’s launch post, from October 15, 2026.
App security checks. The workflow, scan scope, and every check ID by severity, each with its own fix page.
Shopify AI Toolkit. Install it for local use, so /shopify Check my app for security vulnerabilities works in your editor.
App security engine source. The check is open source, so read the rules it runs.
Claude Code GitHub Action. The action that runs the agent checks in Job 2.
GitHub Actions secure use reference. Covers script injection, token permissions, and the pull_request_target trap.
Security reviews in CI: an AI pentester on every PR. Our Strix guide, which covers the hunter that runs on top of this floor.