Animation: a push starts the pipeline. Checkout, install packages, install browsers, run tests and upload report light up one after another, then a green check appears on the pull request

# CI/CD and Scaling Course: [Playwright](https://www.devtutorials.ai/learn/playwright) · Section: Debugging, Reporting and CI Source: https://www.devtutorials.ai/learn/playwright/debug-report-ci/ci-cd-and-scaling > Why tests run on every push, a GitHub Actions workflow for Playwright, CI settings in the config (forbidOnly, retries, workers, reporters), environment variables and secrets, workers and parallel tests, sharding across machines and merging blob reports, running in Docker, other CI systems, and choosing which tests run when. ## CI/CD and Scaling - Tests help most when they run **every time** someone changes the code, not only when someone remembers. - **CI** (continuous integration) is a server that runs your tests on every push and pull request. **CD** (continuous delivery) ships the app when they pass. - In this lesson, you will set up Playwright for CI, and make a big suite fast with workers and sharding. **Simple lesson:** CI runs your tests for you, on every change, on a clean machine. ### A Real-World Example - A developer added `test.only` to focus on one test and forgot to remove it. Locally, only one test runs, and it passes. - On CI, that would hide every other test. The config's `forbidOnly: !!process.env.CI` stops it. - The command sets `CI=true`, like a CI server does. Run it. - Remove `CI=true` from the command and run again. Locally the focus is allowed, so only the login test runs. ### What You Will Learn 1. What a CI pipeline does, step by step. 2. A GitHub Actions workflow for Playwright. 3. CI settings in the config, environment variables and secrets. 4. Workers and parallel tests. 5. Sharding across machines and merging reports. 6. Docker, other CI systems, and which tests to run when. ## A CI Pipeline ### The Steps - Every Playwright pipeline does the same five things on a fresh machine: 1. Get the code. 2. Install Node.js and the project's packages with `npm ci`. 3. Install the browsers with `npx playwright install --with-deps`. 4. Run `npx playwright test`. 5. Upload the report and traces, even when tests fail.

A push starts the pipeline. Each step runs on a clean machine, and the result shows on the pull request.

### A GitHub Actions Workflow - `npm init playwright@latest` offers to create this file: `.github/workflows/playwright.yml`. ```yaml name: Playwright Tests on: push: branches: [main] pull_request: branches: [main] jobs: test: timeout-minutes: 60 runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: lts/* - name: Install dependencies run: npm ci - name: Install Playwright Browsers run: npx playwright install --with-deps - name: Run Playwright tests run: npx playwright test env: BASE_URL: ${{ vars.BASE_URL }} SHOP_PASSWORD: ${{ secrets.SHOP_PASSWORD }} - uses: actions/upload-artifact@v4 if: ${{ !cancelled() }} with: name: playwright-report path: playwright-report/ retention-days: 30 ``` - `if: ${{ !cancelled() }}` uploads the report even when tests fail. That is exactly when you need it. - `--with-deps` also installs the system libraries browsers need on Linux. - The Playwright docs advise **not** caching browsers. Downloading them is about as fast as restoring a cache. **Simple lesson:** checkout, install, install browsers, test, upload. Upload even when tests fail. ## The Config on CI ### process.env.CI - Most CI systems set the `CI` environment variable. The config reads it to behave differently there. - The config made by `npm init playwright` already does this: - Open `playwright.config.ts` in the Explorer. On CI: `forbidOnly` is on, `retries` is 2 and `workers` is 1. - With `CI=true`, the test fails once and passes on retry, so it is **flaky**. Remove `CI=true` and it simply fails. ### Environment Variables and Secrets - Point tests at different servers with a variable: `baseURL: process.env.BASE_URL ?? 'http://localhost:3000'`. - Passwords and tokens go in the CI system's **secrets**, never in the repository. The workflow passes them in with `env:`. - Remove the `BASE_URL=...` part and run. The test tries `localhost:3000`, which the lab cannot reach. Never print secrets in tests or logs. CI systems hide known secrets in logs, but a changed or encoded copy can still leak. **Simple lesson:** `process.env.CI` switches CI settings. URLs and secrets come from environment variables. ## Running Tests in Parallel ### Workers - A **worker** is a separate process with its own browser. Tests in different workers run at the same time. - By default, Playwright uses half your CPU cores. `--workers=4` or `workers: 4` sets it. `workers: '50%'` uses half the cores. - Files run in parallel. Tests **inside** one file run one after another, unless you turn on `fullyParallel: true` or use `test.describe.configure({ mode: 'parallel' })`. - The first line says how many workers ran. Try `--workers=1` and `--workers=6`. ### Why CI Often Uses One Worker - CI machines are often small. Too many workers fight for CPU and make tests slower and flakier. - The default config uses `workers: process.env.CI ? 1 : undefined`. On a big CI machine, raise it. - To go faster than one machine allows, use **sharding**. ## Sharding ### Splitting the Suite Across Machines - `--shard=1/3` runs the first third of the tests. Three CI machines run `1/3`, `2/3` and `3/3` at the same time. - With `fullyParallel: true`, Playwright splits by test. Without it, it splits by file. - Change it to `--shard=2/3` and `--shard=3/3`. Each shard runs different tests, and together they run all six. ### Merging Shard Reports - Each shard writes only part of the results. To get one HTML report, each shard uses the **blob** reporter, and a last job merges them: ```yaml jobs: test: strategy: fail-fast: false matrix: shardIndex: [1, 2, 3, 4] shardTotal: [4] runs-on: ubuntu-latest steps: # ... checkout, npm ci, install browsers ... - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} - uses: actions/upload-artifact@v4 if: ${{ !cancelled() }} with: name: blob-report-${{ matrix.shardIndex }} path: blob-report merge-reports: if: ${{ !cancelled() }} needs: [test] runs-on: ubuntu-latest steps: # ... checkout, npm ci ... - uses: actions/download-artifact@v4 with: path: all-blob-reports pattern: blob-report-* merge-multiple: true - run: npx playwright merge-reports --reporter html ./all-blob-reports ``` - In the config, use `reporter: process.env.CI ? 'blob' : 'html'`. Shards finish only as fast as the slowest one. Keep test files similar in size, or turn on `fullyParallel` so Playwright can balance single tests. **Simple lesson:** workers split tests on one machine. Shards split them across machines. Blob reports merge back into one. ## Docker and Other CI Systems ### The Official Docker Image - Microsoft publishes Docker images with Node.js, the browsers and their system libraries already installed. - The image version must match your `@playwright/test` version, for example `mcr.microsoft.com/playwright:v1.64.0-noble`. - Running in Docker gives everyone the same fonts and system, which keeps screenshot baselines stable. ```bash docker run --rm --ipc=host -v "$PWD":/work -w /work \ mcr.microsoft.com/playwright:v1.64.0-noble \ /bin/bash -c "npm ci && npx playwright test" ``` - `--ipc=host` is recommended for Chromium, so it does not run out of shared memory. When you update `@playwright/test`, update the Docker image tag in the same pull request. A mismatch is a common cause of "Executable doesn't exist" errors on CI. ### Other CI Systems - **Azure Pipelines**, **GitLab CI**, **Jenkins** and **CircleCI** follow the same steps: install, install browsers, test, publish the report. - The Playwright docs have a ready-made file for each one. - Use the `junit` reporter where the CI system shows JUnit results in its own test tab. ## Choosing What Runs When ### Fast Feedback First - Not every test must run on every push. Many teams split them: - On every pull request: a fast **smoke** set, tagged `@smoke`, with `--grep @smoke`. - Before merging, or every night: the full suite on all browsers. - `--last-failed` reruns only what failed last time. `--only-changed` runs tests affected by changed files, compared with git. - `maxFailures: 10` stops a broken run early, so the team gets feedback faster. **Simple lesson:** smoke tests on every change, the full suite before merging or overnight. ## Try It in Your Project - These steps take the lesson from the lab to your own app. - Add the GitHub Actions workflow and make it upload the report even on failure. - Run `@smoke` tests on pull requests and the full suite before merging. - When the suite passes 10 minutes, add sharding. **Simple lesson:** use one idea from this lesson on your own app this week. ## TL;DR - CI runs tests on every change: checkout, `npm ci`, `npx playwright install --with-deps`, test, upload the report. - `process.env.CI` turns on `forbidOnly`, retries and fewer workers. - URLs and secrets come from environment variables, never from the code. - Workers run tests in parallel on one machine. `fullyParallel` splits single tests. - `--shard=x/y` splits across machines. The blob reporter and `merge-reports` make one report. - Docker images give everyone the same browsers. Run smoke tests often and the full suite before merging. ## Common mistakes | Mistake | What happens | Fix | |---|---|---| | Uploading the report only when tests pass | No report when you need it | `if: ${{ !cancelled() }}` | | Forgetting `--with-deps` on Linux | Browsers fail to start with missing libraries | `npx playwright install --with-deps` | | A Docker image version that does not match the package | Browsers are missing or the wrong version | Use the same version as `@playwright/test` | | Many workers on a small CI machine | Slow, flaky runs | One or two workers, then shard | | Sharding with the html reporter | Each shard has a partial report | Use `blob` and `merge-reports` | ## Check Yourself ## Exercises ## What to Remember - CI runs tests on every change, on a clean machine. Upload the report even when tests fail. - `process.env.CI` switches on `forbidOnly`, retries and fewer workers. - Secrets and URLs come from environment variables. - Workers and `fullyParallel` for one machine. `--shard` and `merge-reports` for many. - Docker gives everyone the same browsers. Smoke tests first, the full suite later. Next: **Playwright Library and Advanced Browser Control**