Introduce ability to shuffle AB Test spaces - #16561
Draft
Jakeii wants to merge 4 commits into
Draft
Conversation
Jakeii
force-pushed
the
jlk/ab-test-shuffle
branch
3 times, most recently
from
August 20, 2026 09:09
384b07b to
2568ad2
Compare
Jakeii
force-pushed
the
jlk/ab-test-shuffle
branch
from
August 21, 2026 12:35
2568ad2 to
b257acf
Compare
Jakeii
force-pushed
the
jlk/ab-test-shuffle
branch
from
August 25, 2026 14:08
00e005a to
7e3aae9
Compare
Jakeii
force-pushed
the
jlk/ab-test-shuffle
branch
from
August 25, 2026 14:15
7e3aae9 to
6602316
Compare
🚀 Image pushed to AWS ECRImage digest: 🐛 Run the image locallyThe following can be used to run the image locally: # Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:e3237a8097fe7a98488189ba558751700a80633cc543a88775cbf9ac9407bc48"
# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-jlk-ab-test-shuffle"
# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-30379"
# Refer to image via the GitHub commit SHA tag
# IMAGE_IDENTIFIER=":sha-66023162144ae2515b242aa8f3d0e095f53cb3ca"
# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"
IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/deployTools --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"
# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY
# Pull the image
docker pull $IMAGE
# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Add a configuration to shuffle the AB test audience spaces so that when tests are allocated, they are not allocated linearly from 0->999, but randomly.
A super simple way to do this is to store the desired order for each space in an array and lookup the mvtid to use. Using a straight random number each time would be
Why?
The current allocation setup:

Hopefully this demonstrates the issue, that tests are allocate to spaces linearly, this causes tests variants overlap, something we have been conscious of as of late, especially with the addition of more test spaces.
This could cause tests to influence each other.
With shuffling:
