Skip to content
elixir-europePublic

About

CLI for validating and submitting ISA-JSON multi-omics studies to repositories like BioSamples and ENA.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

MARS-CLI

The MARS-CLI tool is a powerful interface for submitting metadata and associated files to various biological repository services like ENA, BioSamples, and MetaboLights. This command-line tool is useful for managing and validating metadata submissions in a ISA-JSON, as well as for automating aspects of repository submissions.

Main Steps of MARS-CLI

  1. Ingesting and Validating the ISA-JSON
    MARS-CLI requires certain mandatory fields beyond the standard ISA specification (e.g., target_repository as a comment). Upon ingestion, the ISA-JSON is loaded into memory and validated using Pydantic to ensure it meets these requirements.

  2. Identifying the Target Repositories
    The order of submission depends on the repositories specified in the ISA-JSON. MARS-CLI determines the correct sequence for submitting metadata and data.

  3. Registering Samples in BioSamples
    MARS-CLI first submits the ISA-JSON to BioSamples via a newly developed API. BioSamples accessions are crucial since other repositories reuse them. After submission, BioSamples returns a receipt containing accessions for Source and Sample, mapped to Source characteristics and Sample characteristics, respectively.

  4. Filtering the ISA-JSON
    Once updated with BioSamples accessions, the ISA-JSON is filtered for each target repository. This ensures that only relevant metadata is submitted to each repository. The filtering is based on the target repository attribute present in the ISA-JSON assays.

  5. Submitting Data to Target Repositories
    MARS-CLI uploads data using FTP. Some repositories require that data files are present in their upload space before metadata submission. This step ensures that data availability and checksum validation are completed before sending metadata.

  6. Registering ISA-JSON at the Target Repositories
    The filtered ISA-JSON is submitted to repositories that accept ISA-JSON metadata, such as ENA. The MARS project helps out with the adaptation of ISA-JSON by the repositories using the so called adapters.

  7. Processing Repository Receipts and Errors
    After submission, each repository returns a receipt in a standardized format defined for MARS (see repository-api info).

  • The receipt includes the paths of objects within the ISA-JSON and their assigned accession numbers.
  • Errors encountered during submission are processed and reported accordingly.
  1. Updating BioSamples External References
    The Data Broker retrieves the BioSamples JSONs of the submitted samples using its accession numbers and updates the External References schema by adding the accession numbers assigned by other target repositories.

  2. Generating an Updated ISA-JSON with Repository Information
    Based on the information in repository receipts, the ISA-JSON is updated with accession numbers linked to submitted metadata objects. This final, enriched ISA-JSON can then be provided as output.

Credential management

MARS-CLI comes with the functionality to interact with your device's keychain backend in order to fetch the necessary credentials. This allows the user to set credentials in a safe way.

Data submission

MARS-CLI is not a platform to host data and will not store the data after submission to the target repository. This should be handled by the Data broker platform where MARS-CLI is installed. The ISA-JSON provided to the application will be updated and stored in the BioSamples repository as an External Reference, but is otherwise considered as ephemeral.

Installation

This installation procedure describes a typical Linux installation. This application can perfectly work on Windows and MacOS but some of the steps might be different.

This project is managed with uv. Install uv first if you don't have it yet.

Installing the mars-cli from source as a command line tool:

cd mars-cli # Assuming you are in the root folder of this project
uv tool install .

If you want to contribute to the project, set up a development environment with all testing dependencies instead:

uv sync
uv run pytest

Commands like uv run mars-cli --help, uv run black, uv run ruff and uv run mypy then run inside that environment.

Running the MARS-cli for the first time will by default create a .mars directory in the home directory to store settings and log files. If you wish to create the .mars directory in another place, you must specify the MARS_SETTINGS_DIR variable and set it to the desired path:

export MARS_SETTINGS_DIR=<path/to/parent_folder/containing/.mars>

If you want to make it permanent, you can run to following commands in the terminal. Note: replace .bashrc by the config file of your shell.

echo '# Add MARS setting directory to PATH' >> $HOME/.bashrc
echo 'export MARS_SETTINGS_DIR=<path/to/parent_folder/containing/.mars>' >> $HOME/.bashrc

Once installed, the CLI application will be available from the terminal.

Configuration

Running this application for the first time will generate a settings.ini file in $HOME/.mars/. To regenerate the default settings, delete the settings.ini file and run the CLI again.

[logging]
log_level = INFO
log_file = /my/logging/directory/.mars/app.log
log_max_size = 1024
log_max_files = 5

Repository services

To configure MARS for submissions, modify the configuration file settings.ini located at ~/.mars/settings.ini. Ensure the following content is set:

[webin]
development-url = https://wwwdev.ebi.ac.uk/ena/dev/submit/webin/auth
development-token-url = https://wwwdev.ebi.ac.uk/ena/dev/submit/webin/auth/token
production-url = https://www.ebi.ac.uk/ena/submit/webin/auth
production-token-url = https://www.ebi.ac.uk/ena/submit/webin/auth/token

[ena]
development-url = http://localhost:8042/isaena
development-submission-url = http://localhost:8042/isaena/submit
development-data-submission-url = webin2.ebi.ac.uk
production-url = https://www.ebi.ac.uk/ena/submit/webin-v2/
production-submission-url = https://www.ebi.ac.uk/ena/submit/drop-box/submit/?auth=ENA
production-data-submission-url = webin2.ebi.ac.uk

[biosamples]
development-url = http://localhost:8032/isabiosamples
development-submission-url = http://localhost:8032/isabiosamples/submit
production-url = https://www.ebi.ac.uk/biosamples/samples/
production-submission-url = https://www.ebi.ac.uk/biosamples/samples/

Logging

The MARS-CLI will automatically log events to a .log file.

log_level: The verbosity of logging can be set to three different levels

  • CRITICAL: Only critical messages will be logged. Not recommended!
  • ERROR: Errors and critical messages will be logged.
  • WARNING: Warnings, errors and critical messages will be logged.
  • INFO: All events are logged.
  • DEBUG: For debugging purpose only. Not recommended as it might log more sensitive information! The default setting is INFO. So all events are logged, except debug messages. Existing settings files keep their current value, so change log_level there if you want the new default.

log_file: The path to the log file. By default this will be in $HOME/.mars/app.log.

log_max_size: The maximum size in kB for the log file. By default the maximum size is set to 1024 kB or 1 MB.

log_max_files: The maximum number of old log files to keep. By default, this is set to 5

Target repository settings

Each of the target repositories have a set of settings:

  • development-url: URL to the development server when performing a health-check
  • development-submission-url: URL to the development server when performing a submission
  • production-url: URL to the production server when performing a health-check
  • production-submission-url: URL to the production server when performing a submissionW

ENA and MetaboLights also have a development-study-page-url and production-study-page-url: the base URL of their study pages, used to link the submitted studies to the BioSamples records (see Extending BioSamples' records). They default to the ENA browser (https://www.ebi.ac.uk/ena/browser/view/) and MetaboLights (https://www.ebi.ac.uk/metabolights/) pages.

Usage

If you wish to use a different location for the `.mars' folder:

export MARS_SETTINGS_DIR=<path/to/parent_folder/containing/.mars>
mars-cli [options] <command> ARGUMENT

The mars-cli's help text can be found from the command line as such:

mars-cli --help

Output:

➜ mars-cli --help
Usage: mars-cli [OPTIONS] COMMAND [ARGS]...

Options:
  -d, --development  Boolean indicating the usage of the development
                     environment of the target repositories. If not present,
                     the production instances will be used.
  --help             Show this message and exit.

Commands:
  check-data-files   Check that the data files in the ISA JSON match the files...
  health-check       Check the health of the target repositories.
  set-password       Store a password in the keyring.
  submit             Start a submission to the target repositories.
  validate-isa-json  Validate the ISA JSON file.

or for a specific command:

mars-cli submit --help

Output:

➜ mars-cli submit --help
Usage: mars-cli submit [OPTIONS] ISA_JSON_FILE

  Start a submission to the target repositories.

Options:
  --output TEXT
  --submit-to-metabolights BOOLEAN
                                  Submit to Metabolights.
  --data-folder DIRECTORY         Path to a folder holding the files to upload.
                                  All files in the folder (recursively) are
                                  included. Can be passed multiple times.
  --file-transfer TEXT            provide the name of a file transfer
                                  solution, like ftp or aspera
  --submit-to-ena BOOLEAN         Submit to ENA.
  --submit-to-biosamples BOOLEAN  Submit to BioSamples.
  --credentials-file FILENAME     Name of a credentials file
  --username-credentials TEXT     Username from the keyring
  --credential-service-name TEXT  service name from the keyring
  --help                          Show this message and exit.

IMPORTANT: All the data files in the ISA-JSON should have a UNIQUE name.

Development vs production

By default, the mars-CLI will try to submit the ISA-JSON's metadata towards the repositories' production servers. Passing the development flag will run it in development mode and substitute the production servers with the development servers.

Health check repository services

You can check whether the supported repositories are healthy, prior to submission, by doing a health-check.

mars-cli health-check

Output:

➜ mars-cli health-check
############# Welcome to the MARS CLI. #############
Running in Production environment
Checking the health of the target repositories.
Service 'webin' healthy and ready to use!
Service 'ena' healthy and ready to use!
Service 'biosamples' healthy and ready to use!

Credential management

This CLI application comes with functionality to interact with your device's keychain backend in order to fetch the necessary credentials.

Store a password

You can add a password to keychain:

mars-cli set-password set-password [OPTIONS] USERNAME

  Store a password in the keyring.

Options:
  --service-name TEXT  You are advised to include service name to match the
                       credentials to. If empty, it defaults to "mars-
                       cli_{DATESTAMP}"
  --password TEXT      The password to store. Note: You are required to
                       confirm the password.
  --help               Show this message and exit.

Options

Biosamples submissions

--submit-to-biosamples: By default set to True. Will try submit ISA-JSON metadata towards Biosamples. Setting it to False will skip sending the ISA-JSON's metadata to Biosamples.

Note: Following command line will avoid submission to Biosamples repository:

mars-cli submit  --submit-to-biosamples False my-credentials my-isa-json.json

ENA submissions

--submit-to-ena: By default set to True. Will try submit ISA-JSON metadata towards ENA. Setting it to False will skip sending the ISA-JSON's metadata to ENA.

Note: Following command line will avoid submission to ENA repository:

mars-cli submit --submit-to-ena False my-credentials my-isa-json.json

--file-transfer: Provide the name of a file transfer solution, like ftp or aspera

--data-folder: Path to a folder holding the files to upload. All files in the folder (recursively) are included. Can be passed multiple times. Only folders are accepted, not individual files.

Note: Following command line will submit isa-file and data files using FTP solution to Biosamples and ENA:

mars-cli submit --submit-to-metabolights False --file-transfer ftp --data-folder ../data my-credentials my-isa-json.json

Metabolights submissions

Status: 🚧 To Be Developed

--submit-to-metabolights: By default set to True. Will try to submit ISA-JSON metadata towards Metabolights. Setting it to False will skip sending the ISA-JSON's metadata to Metabolights.

Following command line will avoid submission to metabolights repository:

mars-cli submit --submit-to-metabolights False my-credentials my-isa-json.json

--output: By default "output_{datetime.now()}", the name of the isa final output.

mars-cli submit --output final_isa my-credentials my-isa-json.json

Feature: Check data files against the ISA JSON

Check that every data file in the ISA-JSON is present in the data folder(s), without submitting anything. Files are matched on file name, folders are searched recursively and --data-folder can be passed multiple times.

mars-cli check-data-files --data-folder ../data my-isa-json.json
Data file check for ISA JSON with investigation 'My investigation':
- Data files matched to a data file in the ISA JSON: 12
- Data files not referenced in the ISA JSON: 1
- ISA JSON data files without a matching data file: 1
Not referenced in the ISA JSON: ../data/not-in-isa.fastq
Missing data file: rna-seq-data-3.fastq

Unreferenced files are skipped during submission. The command exits with a non-zero status if a data file is missing, or if more than one file in the data folders has the name of the same data file in the ISA-JSON. submit runs the same check first and stops before anything is sent if either happens.

Feature: Extending BioSamples' records

After a submission to ENA and/or MetaboLights, MARS-CLI links the new studies to the BioSamples records of the samples they were made from, as externalReferences:

  • ENA: every sample used in an ENA assay gets a link to the ENA study of that assay (e.g. https://www.ebi.ac.uk/ena/browser/view/ERP123456, which shows the PRJEB... project).
  • MetaboLights: every sample used in a MetaboLights assay gets a link to the MetaboLights study (e.g. https://www.ebi.ac.uk/metabolights/MTBLS1234).

The records are updated with the Webin credentials, so they must belong to the same Webin account. The submission has already succeeded at this point: if a record cannot be updated, a warning is logged and the submission continues.

To summarize, the steps of the code are:

  1. Collect, per BioSamples accession, the study links of the assays the sample is used in
  2. Validate them against mars_lib/biosamples-input-schema.json
  3. For each BioSamples accession, download its JSON record from BioSamples
  4. Extend the record's externalReferences with the new links, skipping the ones already present
  5. Submit the extended JSON to BioSamples to replace the existing one

Examples

Submit isa-json to biosamples

After configuring the settings.ini file, you can run the MARS CLI tool to submit the isa-json:

python mars_cli.py --development submit --submit-to-metabolights False --submit-to-ena False --credential-service-name <biosamples> --username-credentials <username> MARS/test-data/ena-minimal.json
  • Replace <biosamples> with the appropriate service name.
  • Replace <username> with your BioSamples username.
  • Adjust the submission file path (MARS/test-data/ena-minimal.json) as needed.

Alternatively, you can also use a credentials file to authenticate to the services. An example can be found here: https://github.com/elixir-europe/mars-cli/blob/main/tests/test_credentials_example.json

Run the MARS CLI tool to submit the isa-json using credentials file:

python mars_cli.py --development submit --submit-to-metabolights False --submit-to-ena False --credentials-file <path_to_your_credentials_file.json> MARS/test-data/ena-minimal.json

Submit data files and isa-json and to biosamples and ENA

python mars_cli.py --credential-service-name biosamples  --username-credentials <username> --file-transfer ftp --data-folder ../data --submit-to-metabolights False --output final-isa ../data/ena-minimal.json

Submit data files and isa-json to Metabolights

Work in progress, currently data transfer is not supported yet.

python mars_cli.py --credential-service-name metabolights  --username-credentials <username> --file-transfer ftp --data-folder ../data --submit-to-biosamples False --submit-to-ena False --output final-isa ../data/metabolights-lcms.json

Deploy repository services

To set up and run the MARS tool locally using Docker, follow these steps

About

CLI for validating and submitting ISA-JSON multi-omics studies to repositories like BioSamples and ENA.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages