Repository navigation
ingest: remove .gz/.xz files and date_submitted column by TBD #320
Description
Context
Follow up on #317:
In ~1 month, I'll remove the gz/xz file uploads and delete them from S3 so that we don't continue to upload duplicate files with different compressions.
Follow up on #319:
Preserving the date_submitted column for now to give users a chance to switch over to the new column. I plan to remove the date_submitted column by 28 July 2025.
TODO
By TBD
- Write blog post about changes in mpox nextstrain.org#1192
- remove gz/xz file uploads from nextstrain-automation's files_to_upload config - gz/xz files no longer updated as part of Ingest from Pathoplexus, adapt phylogenetic to use Pathoplexus accessions #342
- manually delete gz/xz files on S3
- revert bb47794 to remove the
date_submittedcolumn in metadata.tsv on S3 - column no longer exists as part of Ingest from Pathoplexus, adapt phylogenetic to use Pathoplexus accessions #342
Activity
[-]ingest: remove .gz/.xz files from S3[/-][+]ingest: remove .gz/.xz files and `date_submitted` column by 28 July 2025[/+]joverlee521 commentedon Jul 30, 2025
And of course I missed my own deadline...Pushing this back a bit to announce the changes more widely.
[-]ingest: remove .gz/.xz files and `date_submitted` column by 28 July 2025[/-][+]ingest: remove .gz/.xz files and `date_submitted` column by TBD[/+]joverlee521 commentedon Nov 13, 2025
Upstream data source was changed to PPX in #342.
As part of this change, gz/xz files are no longer updated and the
date_submittedcolumn has been removed.joverlee521 commentedon Nov 13, 2025
Based on the forked NW-PaGe/mpox, the WA DOH folks are still using the gz/xz files, so I've notified them on Slack about switching from gz/xz files to zst files.