Skip to content

Introduce ability to shuffle AB Test spaces - #16561

Draft
Jakeii wants to merge 4 commits into
mainfrom
jlk/ab-test-shuffle
Draft

Introduce ability to shuffle AB Test spaces#16561
Jakeii wants to merge 4 commits into
mainfrom
jlk/ab-test-shuffle

Conversation

@Jakeii

@Jakeii Jakeii commented Aug 14, 2026

Copy link
Copy Markdown
Member

What does this change?

Add a configuration to shuffle the AB test audience spaces so that when tests are allocated, they are not allocated linearly from 0->999, but randomly.

A super simple way to do this is to store the desired order for each space in an array and lookup the mvtid to use. Using a straight random number each time would be

Why?

The current allocation setup:
ab-test_spaces_before

Hopefully this demonstrates the issue, that tests are allocate to spaces linearly, this causes tests variants overlap, something we have been conscious of as of late, especially with the addition of more test spaces.

This could cause tests to influence each other.

With shuffling:
ab_test_spaces_after

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown

@Jakeii
Jakeii force-pushed the jlk/ab-test-shuffle branch 3 times, most recently from 384b07b to 2568ad2 Compare August 20, 2026 09:09
@Jakeii Jakeii changed the title Introduced addtional AB test spaces and test space shuffling Introduced ability to shuffle AB Test spaces Aug 21, 2026
@Jakeii
Jakeii force-pushed the jlk/ab-test-shuffle branch from 2568ad2 to b257acf Compare August 21, 2026 12:35
@Jakeii
Jakeii changed the base branch from main to jlk/audience-space-d-and-e August 21, 2026 12:36
@Jakeii Jakeii changed the title Introduced ability to shuffle AB Test spaces Introduce ability to shuffle AB Test spaces Aug 25, 2026
@Jakeii
Jakeii force-pushed the jlk/ab-test-shuffle branch from 00e005a to 7e3aae9 Compare August 25, 2026 14:08
Base automatically changed from jlk/audience-space-d-and-e to main August 25, 2026 14:10
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

@Jakeii
Jakeii force-pushed the jlk/ab-test-shuffle branch from 7e3aae9 to 6602316 Compare August 25, 2026 14:15
@github-actions

Copy link
Copy Markdown

🚀 Image pushed to AWS ECR

Image digest: sha256:e3237a8097fe7a98488189ba558751700a80633cc543a88775cbf9ac9407bc48

🐛 Run the image locally

The following can be used to run the image locally:

# Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:e3237a8097fe7a98488189ba558751700a80633cc543a88775cbf9ac9407bc48"

# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-jlk-ab-test-shuffle"

# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-30379"

# Refer to image via the GitHub commit SHA tag
# IMAGE_IDENTIFIER=":sha-66023162144ae2515b242aa8f3d0e095f53cb3ca"

# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"

IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/deployTools --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"

# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY

# Pull the image
docker pull $IMAGE

# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE

From guardian/actions-publish-image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants