perf(e2e): Run specs in parallel workers and overlap CI setup - #1269
Open
utkarshcloudinary wants to merge 5 commits into
Open
perf(e2e): Run specs in parallel workers and overlap CI setup#1269utkarshcloudinary wants to merge 5 commits into
utkarshcloudinary wants to merge 5 commits into
Conversation
The e2e suite ran with a single Playwright worker because every spec shared one analytics capture log and several specs mutate site-wide state. Test execution took ~2.5 min of a ~6 min CI job. Test side: - Give each Playwright worker its own analytics capture log. The mu-plugin derives the log path from an X-CLD-E2E-Worker header (browser and REST traffic) or a CLD_E2E_WORKER env var (WP-CLI via docker exec). Unmarked requests keep using the shared log. - Add tests/e2e/fixtures.js which sets both markers per worker; all specs import test/expect from it. - Raise workers to 3 and tag the five specs that mutate global state (connection, plugin activation, real-credential delivery) with @serial. `npm run test:e2e` runs the parallel set first, then the serial set with one worker in the same wp-env. - settings-analytics flips image_quality instead of image_format so it no longer races media-analytics on the same key. - Short-circuit Admin API calls made with the fake e2e cloud with a local 401. Connect::history() issued seven ~1s real round-trips on every dashboard load, pushing page loads past the navigation timeout. - cache-analytics: create the cache point after the admin page load, and clear the stale parent plus the asset lock before the single purge test. activate_parents() deleted the unconfigured parent and left a 10s lock that blocked the tracked branch in rest_purge_all(). CI side: - Run wp-env start in the background while browsers install and assets build. - Cache Playwright browsers keyed on the @playwright/test version. - npm ci --ignore-scripts in the e2e job; composer and patch-package are not needed to run the suite. - Pin WP_ENV_HOME in the unit job so the wp-env sources cache path is actually the one wp-env writes to. Local run: parallel phase 20 tests in ~23s, serial phase 12 tests in ~1.3 min, down from ~2.5 min for 32 tests serially.
With npm 10 (Node 22) the flag made the install step take 4 to 7 minutes in two consecutive attempts, against ~36s for the plain npm ci in the other jobs of the same run. The postinstall hook it was meant to skip costs about 7s.
The unit job pulled all ~2,200 npm packages to get one CLI. Install @wordpress/env alone, at the version pinned in package-lock.json, into a scratch prefix so npm does not reconcile against the repo lockfile. Composer deps are installed directly. Remove the wp-env sources cache. Once WP_ENV_HOME made it work, the restored directory carried the previous run's install state, the plugin activation hook did not run, and PHPUnit flagged two tests risky for a missing relationships table.
extraHTTPHeaders is attached to every request the browser context makes, including cross-origin ones. The custom header forced a CORS preflight that third parties (fonts inside the Cloudinary player iframe) rejected. A cookie is scoped to the site under test and is shared by page.request, so REST calls still carry it.
Every job with setup-php completes npm ci in ~35s; the e2e job, without it, took 2.5 to 4 minutes on the same runs with the same npm and the same npm cache hit. Aligning the job's prelude with the others is the one difference left to test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves e2e test runtime in CI (no linked issue).
Approach
cld_e2e_workercookie (browser and REST traffic) or aCLD_E2E_WORKERenv var (WP-CLI viadocker exec) and appends toanalytics-capture-<worker>.log. Unmarked requests keep using the shared log. A cookie rather thanextraHTTPHeadersbecause the header is attached cross-origin and forced CORS preflights that third parties (fonts inside the Cloudinary player iframe) rejected.tests/e2e/fixtures.jssets both markers per worker; all specs importtest/expectfrom it.workers: 3. Specs that mutate site-wide state (wizard-setup,connection-analytics,deactivation-analytics, image and video delivery) are tagged@serial.npm run test:e2eruns the parallel set, then the serial set with one worker, in the same wp-env.settings-analyticsflipsimage_qualityinstead ofimage_formatso it no longer racesmedia-analyticson the same key.Connect::history()issued seven ~1s real round-trips per dashboard load, which pushedpage=cloudinarypast the navigation timeout.cache-analytics: create the cache point after the admin page load, and clear the stale parent plus the asset lock before the single-purge test.activate_parents()deletes the unconfigured parent and leaves a 10s_asset_locktransient that blocks the tracked branch inrest_purge_all().wp-env startruns in the background while browsers install and assets build; Playwright browsers cached by version;Setup PHPand a Composer cache added beforenpm ci. WithoutSetup PHPthe e2e job'snpm citook 2.5 to 4 min on the same runs where every job with it took ~35s.@wordpress/env(391 packages, ~15s) at the lockfile version into a scratch prefix instead of the fullnpm ci; Composer deps installed directly. The wp-env sources cache is removed: once it actually restored, it carried the previous run's install state, the plugin activation hook did not run, and PHPUnit flagged two tests risky for a missing relationships table.npm ci --ignore-scriptsmade the install step take 4 to 7 min with npm 10.CI timings, latest run on this branch vs median of the 5 runs before it:
npm ciRun E2E testsAdding new specs:
Parallel (default). Import
test/expectfrom./fixtures; nothing else is needed. UsefakeCloudinaryConnected(), assert only onfindAnalyticsEvents( events, 'name' )rather than the total event count, and own the settings keys the spec flips (checkrg "image_settings\[" tests/e2ebefore picking one).Serial. Tag the describe block and say why in a comment;
npm run test:e2eruns tagged specs in a second, single-worker pass after the parallel set:Tag it if the spec writes or deletes
cloudinary_connect(anyensureCloudinaryConnected()orwp cloudinary sync), deactivates or uninstalls the plugin, or flips a global setting another spec reads without restoring it. Serial specs may still run alongside nothing else, so they are free to assume the plugin is active and connected as they left it.Tests within one file always run in order in both modes (
fullyParallelis off), so per-filebeforeAll/afterAllstate is safe.Run one phase with
npm run test:e2e:parallelornpm run test:e2e:serial; one file withnpx playwright test --config tests/e2e/playwright.config.js my.spec.js.QA notes
developrun (gh run view <id> --json jobs).npm run env:start && npm run build && npm run test:e2e. Both phases should pass.docker exec <tests-cli> ls wp-content/uploads/should showanalytics-capture-w0.logthroughw2.log.npm run test:e2etwice in a row to confirm the cache-analytics ordering fix holds across runs.--headedor check the Playwright trace: no CORS errors forfonts.gstatic.comin the console.