Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Pile

Piles of reads, piles of transcripts, piles of alignments, piles of evidence...

This repository contains scripts and pipeline for processing NGS and RNAseq data. The aim here is to standardize naming and file location conventions for working with sequences, transcriptomes, alignments across species and samples.

Setup

There should be a directory where all the workspace and reference files are stored. Set the following variables on the host, then use docker-compose to run commands.

PILE_WORKSPACES_DIR=...

All other commands require a workspace to be set using the PILE_WORKSPACE environment variable, which is passed to the container via docker-compose.yml.

The following alias is useful

alias pile-py='docker-compose -f pile/docker-compose.yml run --rm pile python3'
alias pile-run='docker-compose -f pile/docker-compose.yml run --rm pile'

For clustering sequences, get the MMSeqs2 docker image: docker pull ghcr.io/soedinglab/mmseqs2

Data Model and Directory Setup

Pile works with the following data types

  • reference: files associated with a reference genome, each with a NCBI accession

  • workspace

    • reads: NGS sequencing reads, named with SRA accessions or other unique identifiers
    • transcriptomes: uniquely named transcriptomes, each with .fna, .cds.fna, .faa
    • alignments: sample reads aligned against a transcriptome, named as -.bam
    • transcripts: a transcript's alignments from those of a sample against a transcriptome

For more details on the file naming conventions, see the pile.Defaults class.

Quantification workflow

Fetching SRA reads

PILE_WORKSPACE=PM32426508 pile-py pile/sra_fetch.py \
  SRR9331959

This may be helpful to download all the SRR numbers in bulk

pip3 install pysradb
pysradb metadata PRJNA591730 > metadata.tsv

Quantification requires a transcriptome. For transcriptome files downloaded from NCBI or paper, cluster them first using MMSeqs2, then put the following files in the transcriptome directory

  • transcripts.fna
  • transcripts.unclustered.fna
  • transcript_clusters.tsv
  • proteins.faa

If creating a transcriptome from scratch using Trinity, follow the Trinity workflow steps below.

Combining transcriptomes

If samples are sequenced with multiple species, they should be counted with concatenated transcriptomes. First, use the following script to add species prefix to accessions

tangle-py ~/git/tangle/tangle/scripts/fasta-pool.py --database a_tenuis --delim _ transcripts.fna.gz

Then

PILE_WORKSPACE=PM32426508 pile-py pile/transcriptome_concat.py \
  a b c

Quantification

Create Salmon index for the transcriptome

PILE_WORKSPACE=PM32426508 pile-py pile/quant_index.py \
  GCA_947184155.2

Then quant a SRR sample against the transcriptome with

PILE_WORKSPACE=PM32426508 pile-py pile/quant_reads.py \
  GCA_947184155.2 SRR9331959

The above are equivalent of running the following inside a container

# in docker, in transcriptome directory
salmon index -t transcripts.fna -i transcripts.salmon_index
salmon quant -i transcripts.salmon_index \
  -p 2 -l A --validateMappings \
  -o ../../quants/<transcriptome_name>/SRR6255880 
  -1 ../../reads/SRR6255880_1.fastq -2 ../../reads/SRR6255880_2.fastq

Building a transcriptome

A transcriptome should have these files in its transcriptome directory

  • transcripts.fna - transcripts, likely clustered
  • proteins.faa - proteins associated with the transcripts
  • proteins.gff3 - GFF mapping transcripts to proteins

Trinity assembly

To filter reads for Trinity assembly

PILE_WORKSPACE=PM32426508 pile-py pile/assembly_filter_reads.py \
  a_tenuis SRR9331965 \
  --capture GCA_014633955.1 \
  --remove GCA_947184155.2

If you need to pool reads, pool them now. It is important to POOL BEFORE NORMALIZATION.

Then run bbnorm.sh to normalize depth so we don't run out of memory during the Trinity step.

bbnorm.sh \
  in1=pooled_1.fastq \
  in2=pooled_2.fastq \
  out1=norm_1.fastq \
  out2=norm_2.fastq \
  target=40 \
  min=6 \
  bits=16 \
  prefilter=true \
  passes=1

To make a Trinity assembly, after you have filtered the reads, run the following (the last two arguments help the run fit under the memory constraint)

# inside docker, in transcriptome directory
Trinity --seqType fq --max_memory 20G --CPU 8 \
        --left norm_1.fastq \
        --right norm_2.fastq \
        --output trinity_transcript \
        --min_kmer_cov 2

TrinityStats.pl transcript.Trinity.fasta

Use the following script to add species prefix to accessions

tangle-py ~/git/tangle/tangle/scripts/fasta-pool.py --database a_tenuis --delim _ transcripts.fna.gz

Transcriptome Clustering

If you have mmseqs Docker image, then use the following to cluster transcripts. Note that clustered transcripts should only be used for certain differential expression analyses where you don't care about alleles and want to maximize signal. For most genetics work, unclustered transcripts may be best.

# on host, in transcriptome directory
<pile_repo_directory>/scripts/mmseqs-cluster-trinity-transcripts \
  transcripts.fna
mv transcripts.fna transcripts.unclustered.fna
mv transcripts.fna_rep_seq.fna.gz transcripts.fna.gz
mv transcripts.fna_cluster.tsv transcript_clusters.tsv

Once you have clustered, the downstream workflows -- transdecoder, quantification, classification etc -- all use clustered transcriptome. This consistency is important.

TransDecoder ORF and protein prediction

Use TransDecoder to predict ORFs

# inside docker, in transcriptome directory
TransDecoder.LongOrfs -t transcripts.fna -m 100
cp transcripts.fna.transdecoder_dir/longest_orfs.pep proteins.faa
cp transcripts.fna.transdecoder_dir/longest_orfs.gff3 proteins.gff3
rm -rf transcripts.fna.transdecoder_dir

Converting transcriptome GFF file, from transdecoder, into detected table (you need to run TransDecoder first)

PILE_WORKSPACE=PM32426508 pile-py pile/transdecoder_to_detected.py \
  --database-name=EXP_PM32426508 \
  c_goreaui

Reviewing/debugging transcripts

Use the following to make a bowtie index, on a path off the top workspaces directory (i.e. not specific to a workspace). E.g.

pile-py pile/bowtie_index.py ncbi/GCA_014633955.1/genomic.fna

Index a transcriptome using the following

PILE_WORKSPACE=PM32426508 pile-py pile/transcriptome_index.py \
  SRR9331959_algae_denovo

Align a sample against a transcriptome

PILE_WORKSPACE=PM32426508 pile-py pile/transcriptome_align.py \
  SRR9331959_algae_denovo SRR9331961 \
  --cpus 6

Extract a specific transcript's alignments from sample alignments (see above)

PILE_WORKSPACE=PM32426508 pile-py pile/alignment_extract.py -a \
  SRR9331959_algae_denovo SRR9331961 TRINITY_DN7562_c0_g1_i1

Create alignment pileups of transcripts in a second transcriptome, releated to a transcript from a first transcriptome

PILE_WORKSPACE=PM32426508 pile-run bash -c \
  "pile/transcriptome_find.py GCA_947184155.2 CAL1161012.1 | \
   pile/transcriptome_search.py SRR9331959_algae_denovo - | \
   pile/alignment_extract.py SRR9331959 SRR9331959_algae_denovo -"

Same as above, but find the protein sequence instead of creating pileups

PILE_WORKSPACE=PM32426508 pile-run bash -c \
  "pile/transcriptome_find.py GCA_947184155.2 CAL1161012.1 | \
   pile/transcriptome_search.py SRR9331959_algae_denovo - | \
   pile/transcriptome_find.py -f -p SRR9331959_algae_denovo -"

Search for a sequence

# inside docker
bowtie2 --local -f -p 8 \
  -x /users/pile/PM32426508/transcriptomes/SRR9331959_algae_denovo/transcripts.fna \
  -U query.fasta \
  -S results.sam

Using samtools mpileup feature to generate position vectors

bowtie2 --local --very-sensitive-local -p 8 \
  -x /users/pile/PM34593802/transcriptomes/orbicella_faveolata_symbiodinium_a3/transcripts.fna.gz \
  -1 /users/pile/PM34593802/reads/SRR6255844_1.fastq.gz \
  -2 /users/pile/PM34593802/reads/SRR6255844_2.fastq.gz | \
samtools sort -@ 4 -O BAM | \
samtools mpileup -Q 20 -d 100000 \
  -f /users/pile/PM34593802/transcriptomes/orbicella_faveolata_symbiodinium_a3/transcripts.fna - \
  > /users/pile/PM34593802/alignments/SRR6255844.orbicella_faveolata_symbiodinium_a3.pos.txt

Then parse the position vectors to generate a summary of the read coverage, using PELT

grep orbicella_host_TRINITY_DN216370_c1_g2_i10 \
  /users/pile/PM34593802/alignments/SRR6255844.orbicella_faveolata_symbiodinium_a3.pos.txt > transcript_posvec.txt
pile-py pile/posvec_pelt.py -i transcript_posvec.txt -t orbicella_host_TRINITY_DN216370_c1_g2_i10

Extracting DNA associated with proteins

extract-protein-dna.py extracts the transcript DNA encoding a 1-based, inclusive protein interval. It resolves the protein-to-CDS and CDS-to-transcript mappings in a detected TSV. The transcript FASTA may be plain text or gzip compressed. If the coding region is on the reverse strand, the script prints the reverse complement and descending transcript coordinates in the header.

python3 scripts/extract-protein-dna.py \
  transcripts.fna.gz detected.tsv protein_accession start_aa stop_aa \
  > protein-region.fasta

The output record has this form:

>protein_accession_protein_start_protein_stop|transcript_accession_dna_start_dna_stop
DNA_SEQUENCE

extract-transcript-dna.py performs the same extraction for multiple protein regions. It uses the detected TSV to translate each protein interval to transcript coordinates. The CSV or TSV of regions must have these columns:

protein_accession,start,stop

The start and stop values are 1-based, inclusive amino-acid coordinates, with start <= stop. Reverse-strand coding regions produce reverse-complemented DNA and descending transcript coordinates. The transcript FASTA may be plain text or gzip compressed.

python3 scripts/extract-transcript-dna.py \
  transcripts.fna.gz detected.tsv protein-regions.csv \
  > transcript-regions.fasta

Each output record has this form:

>protein_accession_protein_start_protein_stop|transcript_accession_dna_start_dna_stop
DNA_SEQUENCE

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages