Piles of reads, piles of transcripts, piles of alignments, piles of evidence...
This repository contains scripts and pipeline for processing NGS and RNAseq data. The aim here is to standardize naming and file location conventions for working with sequences, transcriptomes, alignments across species and samples.
There should be a directory where all the workspace and reference files are
stored. Set the following variables on the host, then use docker-compose to
run commands.
PILE_WORKSPACES_DIR=...
All other commands require a workspace to be set using the PILE_WORKSPACE
environment variable, which is passed to the container via
docker-compose.yml.
The following alias is useful
alias pile-py='docker-compose -f pile/docker-compose.yml run --rm pile python3'
alias pile-run='docker-compose -f pile/docker-compose.yml run --rm pile'
For clustering sequences, get the MMSeqs2 docker image: docker pull ghcr.io/soedinglab/mmseqs2
Pile works with the following data types
-
reference: files associated with a reference genome, each with a NCBI accession
-
workspace
- reads: NGS sequencing reads, named with SRA accessions or other unique identifiers
- transcriptomes: uniquely named transcriptomes, each with .fna, .cds.fna, .faa
- alignments: sample reads aligned against a transcriptome, named as -.bam
- transcripts: a transcript's alignments from those of a sample against a transcriptome
For more details on the file naming conventions, see the pile.Defaults class.
Fetching SRA reads
PILE_WORKSPACE=PM32426508 pile-py pile/sra_fetch.py \
SRR9331959
This may be helpful to download all the SRR numbers in bulk
pip3 install pysradb
pysradb metadata PRJNA591730 > metadata.tsv
Quantification requires a transcriptome. For transcriptome files downloaded from NCBI or paper, cluster them first using MMSeqs2, then put the following files in the transcriptome directory
transcripts.fnatranscripts.unclustered.fnatranscript_clusters.tsvproteins.faa
If creating a transcriptome from scratch using Trinity, follow the Trinity workflow steps below.
If samples are sequenced with multiple species, they should be counted with concatenated transcriptomes. First, use the following script to add species prefix to accessions
tangle-py ~/git/tangle/tangle/scripts/fasta-pool.py --database a_tenuis --delim _ transcripts.fna.gz
Then
PILE_WORKSPACE=PM32426508 pile-py pile/transcriptome_concat.py \
a b c
Create Salmon index for the transcriptome
PILE_WORKSPACE=PM32426508 pile-py pile/quant_index.py \
GCA_947184155.2
Then quant a SRR sample against the transcriptome with
PILE_WORKSPACE=PM32426508 pile-py pile/quant_reads.py \
GCA_947184155.2 SRR9331959
The above are equivalent of running the following inside a container
# in docker, in transcriptome directory
salmon index -t transcripts.fna -i transcripts.salmon_index
salmon quant -i transcripts.salmon_index \
-p 2 -l A --validateMappings \
-o ../../quants/<transcriptome_name>/SRR6255880
-1 ../../reads/SRR6255880_1.fastq -2 ../../reads/SRR6255880_2.fastq
A transcriptome should have these files in its transcriptome directory
transcripts.fna- transcripts, likely clusteredproteins.faa- proteins associated with the transcriptsproteins.gff3- GFF mapping transcripts to proteins
To filter reads for Trinity assembly
PILE_WORKSPACE=PM32426508 pile-py pile/assembly_filter_reads.py \
a_tenuis SRR9331965 \
--capture GCA_014633955.1 \
--remove GCA_947184155.2
If you need to pool reads, pool them now. It is important to POOL BEFORE NORMALIZATION.
Then run bbnorm.sh to normalize depth so we don't run out of memory during
the Trinity step.
bbnorm.sh \
in1=pooled_1.fastq \
in2=pooled_2.fastq \
out1=norm_1.fastq \
out2=norm_2.fastq \
target=40 \
min=6 \
bits=16 \
prefilter=true \
passes=1
To make a Trinity assembly, after you have filtered the reads, run the following (the last two arguments help the run fit under the memory constraint)
# inside docker, in transcriptome directory
Trinity --seqType fq --max_memory 20G --CPU 8 \
--left norm_1.fastq \
--right norm_2.fastq \
--output trinity_transcript \
--min_kmer_cov 2
TrinityStats.pl transcript.Trinity.fasta
Use the following script to add species prefix to accessions
tangle-py ~/git/tangle/tangle/scripts/fasta-pool.py --database a_tenuis --delim _ transcripts.fna.gz
If you have mmseqs Docker image, then use the following to cluster transcripts. Note that clustered transcripts should only be used for certain differential expression analyses where you don't care about alleles and want to maximize signal. For most genetics work, unclustered transcripts may be best.
# on host, in transcriptome directory
<pile_repo_directory>/scripts/mmseqs-cluster-trinity-transcripts \
transcripts.fna
mv transcripts.fna transcripts.unclustered.fna
mv transcripts.fna_rep_seq.fna.gz transcripts.fna.gz
mv transcripts.fna_cluster.tsv transcript_clusters.tsv
Once you have clustered, the downstream workflows -- transdecoder, quantification, classification etc -- all use clustered transcriptome. This consistency is important.
Use TransDecoder to predict ORFs
# inside docker, in transcriptome directory
TransDecoder.LongOrfs -t transcripts.fna -m 100
cp transcripts.fna.transdecoder_dir/longest_orfs.pep proteins.faa
cp transcripts.fna.transdecoder_dir/longest_orfs.gff3 proteins.gff3
rm -rf transcripts.fna.transdecoder_dir
Converting transcriptome GFF file, from transdecoder, into detected table (you need to run TransDecoder first)
PILE_WORKSPACE=PM32426508 pile-py pile/transdecoder_to_detected.py \
--database-name=EXP_PM32426508 \
c_goreaui
Use the following to make a bowtie index, on a path off the top workspaces directory (i.e. not specific to a workspace). E.g.
pile-py pile/bowtie_index.py ncbi/GCA_014633955.1/genomic.fna
Index a transcriptome using the following
PILE_WORKSPACE=PM32426508 pile-py pile/transcriptome_index.py \
SRR9331959_algae_denovo
Align a sample against a transcriptome
PILE_WORKSPACE=PM32426508 pile-py pile/transcriptome_align.py \
SRR9331959_algae_denovo SRR9331961 \
--cpus 6
Extract a specific transcript's alignments from sample alignments (see above)
PILE_WORKSPACE=PM32426508 pile-py pile/alignment_extract.py -a \
SRR9331959_algae_denovo SRR9331961 TRINITY_DN7562_c0_g1_i1
Create alignment pileups of transcripts in a second transcriptome, releated to a transcript from a first transcriptome
PILE_WORKSPACE=PM32426508 pile-run bash -c \
"pile/transcriptome_find.py GCA_947184155.2 CAL1161012.1 | \
pile/transcriptome_search.py SRR9331959_algae_denovo - | \
pile/alignment_extract.py SRR9331959 SRR9331959_algae_denovo -"
Same as above, but find the protein sequence instead of creating pileups
PILE_WORKSPACE=PM32426508 pile-run bash -c \
"pile/transcriptome_find.py GCA_947184155.2 CAL1161012.1 | \
pile/transcriptome_search.py SRR9331959_algae_denovo - | \
pile/transcriptome_find.py -f -p SRR9331959_algae_denovo -"
Search for a sequence
# inside docker
bowtie2 --local -f -p 8 \
-x /users/pile/PM32426508/transcriptomes/SRR9331959_algae_denovo/transcripts.fna \
-U query.fasta \
-S results.sam
Using samtools mpileup feature to generate position vectors
bowtie2 --local --very-sensitive-local -p 8 \
-x /users/pile/PM34593802/transcriptomes/orbicella_faveolata_symbiodinium_a3/transcripts.fna.gz \
-1 /users/pile/PM34593802/reads/SRR6255844_1.fastq.gz \
-2 /users/pile/PM34593802/reads/SRR6255844_2.fastq.gz | \
samtools sort -@ 4 -O BAM | \
samtools mpileup -Q 20 -d 100000 \
-f /users/pile/PM34593802/transcriptomes/orbicella_faveolata_symbiodinium_a3/transcripts.fna - \
> /users/pile/PM34593802/alignments/SRR6255844.orbicella_faveolata_symbiodinium_a3.pos.txt
Then parse the position vectors to generate a summary of the read coverage, using PELT
grep orbicella_host_TRINITY_DN216370_c1_g2_i10 \
/users/pile/PM34593802/alignments/SRR6255844.orbicella_faveolata_symbiodinium_a3.pos.txt > transcript_posvec.txt
pile-py pile/posvec_pelt.py -i transcript_posvec.txt -t orbicella_host_TRINITY_DN216370_c1_g2_i10
extract-protein-dna.py extracts the transcript DNA encoding a 1-based,
inclusive protein interval. It resolves the protein-to-CDS and CDS-to-transcript
mappings in a detected TSV. The transcript FASTA may be plain text or gzip
compressed. If the coding region is on the reverse strand, the script prints
the reverse complement and descending transcript coordinates in the header.
python3 scripts/extract-protein-dna.py \
transcripts.fna.gz detected.tsv protein_accession start_aa stop_aa \
> protein-region.fasta
The output record has this form:
>protein_accession_protein_start_protein_stop|transcript_accession_dna_start_dna_stop
DNA_SEQUENCE
extract-transcript-dna.py performs the same extraction for multiple protein
regions. It uses the detected TSV to translate each protein interval to
transcript coordinates. The CSV or TSV of regions must have these columns:
protein_accession,start,stop
The start and stop values are 1-based, inclusive amino-acid coordinates, with
start <= stop. Reverse-strand coding regions produce reverse-complemented DNA
and descending transcript coordinates. The transcript FASTA may be plain text
or gzip compressed.
python3 scripts/extract-transcript-dna.py \
transcripts.fna.gz detected.tsv protein-regions.csv \
> transcript-regions.fasta
Each output record has this form:
>protein_accession_protein_start_protein_stop|transcript_accession_dna_start_dna_stop
DNA_SEQUENCE