Skip to content

perf(e2e): Run specs in parallel workers and overlap CI setup - #1269

Open
utkarshcloudinary wants to merge 5 commits into
developfrom
perf/e2e-parallel-workers
Open

perf(e2e): Run specs in parallel workers and overlap CI setup#1269
utkarshcloudinary wants to merge 5 commits into
developfrom
perf/e2e-parallel-workers

Conversation

@utkarshcloudinary

@utkarshcloudinary utkarshcloudinary commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Improves e2e test runtime in CI (no linked issue).

Approach

  • The e2e job took 5.5 to 6.5 min: ~2.5 min for the Playwright run with a single worker, ~2 min for setup run strictly in sequence. This PR makes the analytics specs worker-safe so most of the suite runs in parallel, and overlaps the independent setup steps.
  • Per-worker analytics capture log. The mu-plugin reads a cld_e2e_worker cookie (browser and REST traffic) or a CLD_E2E_WORKER env var (WP-CLI via docker exec) and appends to analytics-capture-<worker>.log. Unmarked requests keep using the shared log. A cookie rather than extraHTTPHeaders because the header is attached cross-origin and forced CORS preflights that third parties (fonts inside the Cloudinary player iframe) rejected.
  • New tests/e2e/fixtures.js sets both markers per worker; all specs import test/expect from it.
  • workers: 3. Specs that mutate site-wide state (wizard-setup, connection-analytics, deactivation-analytics, image and video delivery) are tagged @serial. npm run test:e2e runs the parallel set, then the serial set with one worker, in the same wp-env.
  • settings-analytics flips image_quality instead of image_format so it no longer races media-analytics on the same key.
  • Admin API calls made with the fake e2e cloud are answered locally with a 401. Connect::history() issued seven ~1s real round-trips per dashboard load, which pushed page=cloudinary past the navigation timeout.
  • cache-analytics: create the cache point after the admin page load, and clear the stale parent plus the asset lock before the single-purge test. activate_parents() deletes the unconfigured parent and leaves a 10s _asset_lock transient that blocks the tracked branch in rest_purge_all().
  • CI, e2e job: wp-env start runs in the background while browsers install and assets build; Playwright browsers cached by version; Setup PHP and a Composer cache added before npm ci. Without Setup PHP the e2e job's npm ci took 2.5 to 4 min on the same runs where every job with it took ~35s.
  • CI, unit job: install only @wordpress/env (391 packages, ~15s) at the lockfile version into a scratch prefix instead of the full npm ci; Composer deps installed directly. The wp-env sources cache is removed: once it actually restored, it carried the previous run's install state, the plugin activation hook did not run, and PHPUnit flagged two tests risky for a missing relationships table.
  • Tried and reverted: npm ci --ignore-scripts made the install step take 4 to 7 min with npm 10.

CI timings, latest run on this branch vs median of the 5 runs before it:

step before after
E2E job total 363s (327 to 391) 302s
npm ci 40s 39s
setup (browsers, build, wp-env) 127s 104s
Run E2E tests 147s 119s (parallel 50s + serial 66s)
Unit job total 187s 121s

Adding new specs:

  • Parallel (default). Import test/expect from ./fixtures; nothing else is needed. Use fakeCloudinaryConnected(), assert only on findAnalyticsEvents( events, 'name' ) rather than the total event count, and own the settings keys the spec flips (check rg "image_settings\[" tests/e2e before picking one).

  • Serial. Tag the describe block and say why in a comment; npm run test:e2e runs tagged specs in a second, single-worker pass after the parallel set:

    // @serial: writes real credentials to `cloudinary_connect`.
    test.describe( 'Real sync flow', { tag: '@serial' }, () => {

    Tag it if the spec writes or deletes cloudinary_connect (any ensureCloudinaryConnected() or wp cloudinary sync), deactivates or uninstalls the plugin, or flips a global setting another spec reads without restoring it. Serial specs may still run alongside nothing else, so they are free to assume the plugin is active and connected as they left it.

  • Tests within one file always run in order in both modes (fullyParallel is off), so per-file beforeAll/afterAll state is safe.

  • Run one phase with npm run test:e2e:parallel or npm run test:e2e:serial; one file with npx playwright test --config tests/e2e/playwright.config.js my.spec.js.

QA notes

  • Compare the E2E job timings on this PR against a recent develop run (gh run view <id> --json jobs).
  • Locally: npm run env:start && npm run build && npm run test:e2e. Both phases should pass. docker exec <tests-cli> ls wp-content/uploads/ should show analytics-capture-w0.log through w2.log.
  • Run npm run test:e2e twice in a row to confirm the cache-analytics ordering fix holds across runs.
  • Open the video delivery test with --headed or check the Playwright trace: no CORS errors for fonts.gstatic.com in the console.

The e2e suite ran with a single Playwright worker because every spec
shared one analytics capture log and several specs mutate site-wide
state. Test execution took ~2.5 min of a ~6 min CI job.

Test side:
- Give each Playwright worker its own analytics capture log. The
  mu-plugin derives the log path from an X-CLD-E2E-Worker header
  (browser and REST traffic) or a CLD_E2E_WORKER env var (WP-CLI via
  docker exec). Unmarked requests keep using the shared log.
- Add tests/e2e/fixtures.js which sets both markers per worker; all
  specs import test/expect from it.
- Raise workers to 3 and tag the five specs that mutate global state
  (connection, plugin activation, real-credential delivery) with
  @serial. `npm run test:e2e` runs the parallel set first, then the
  serial set with one worker in the same wp-env.
- settings-analytics flips image_quality instead of image_format so it
  no longer races media-analytics on the same key.
- Short-circuit Admin API calls made with the fake e2e cloud with a
  local 401. Connect::history() issued seven ~1s real round-trips on
  every dashboard load, pushing page loads past the navigation timeout.
- cache-analytics: create the cache point after the admin page load,
  and clear the stale parent plus the asset lock before the single
  purge test. activate_parents() deleted the unconfigured parent and
  left a 10s lock that blocked the tracked branch in rest_purge_all().

CI side:
- Run wp-env start in the background while browsers install and
  assets build.
- Cache Playwright browsers keyed on the @playwright/test version.
- npm ci --ignore-scripts in the e2e job; composer and patch-package
  are not needed to run the suite.
- Pin WP_ENV_HOME in the unit job so the wp-env sources cache path is
  actually the one wp-env writes to.

Local run: parallel phase 20 tests in ~23s, serial phase 12 tests in
~1.3 min, down from ~2.5 min for 32 tests serially.
With npm 10 (Node 22) the flag made the install step take 4 to 7
minutes in two consecutive attempts, against ~36s for the plain
npm ci in the other jobs of the same run. The postinstall hook it
was meant to skip costs about 7s.
The unit job pulled all ~2,200 npm packages to get one CLI. Install
@wordpress/env alone, at the version pinned in package-lock.json,
into a scratch prefix so npm does not reconcile against the repo
lockfile. Composer deps are installed directly.

Remove the wp-env sources cache. Once WP_ENV_HOME made it work, the
restored directory carried the previous run's install state, the
plugin activation hook did not run, and PHPUnit flagged two tests
risky for a missing relationships table.
extraHTTPHeaders is attached to every request the browser context
makes, including cross-origin ones. The custom header forced a CORS
preflight that third parties (fonts inside the Cloudinary player
iframe) rejected. A cookie is scoped to the site under test and is
shared by page.request, so REST calls still carry it.
Every job with setup-php completes npm ci in ~35s; the e2e job,
without it, took 2.5 to 4 minutes on the same runs with the same npm
and the same npm cache hit. Aligning the job's prelude with the
others is the one difference left to test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants