Heap: (noun) a disorderly collection of objects placed haphazardly on top of each other, or (verb) put objects or loose substances on top of each other.
In this case: loosely organized collection of probabilistic annotations for transcripts and proteins, to help figure out what these transcripts and proteins do.
Install hmmer3 package: e.g. on MacOS run brew install hmmer
Download the following profiles
-
KEGG KO profile HMMs:
https://www.genome.jp/ftp/db/kofam/profiles.tar.gz- Then concatenate all the profiles together:
cat profiles/*.hmm > ko.hmm- Remove all entries that are RNA or small RNAs: make a list of KOs to include, then use
hmmfetch -fto create a new.hmmfile - Run
hmmfetch --index ko.hmm - Run
hmmpress ko.hmm - If you run the above two out of order, may need to
mv ko.hmm.h3m.ssi ko.hmm.ssi
- Remove all entries that are RNA or small RNAs: make a list of KOs to include, then use
- Also create
ko_thresholds.tsvfrom KEGG FTP sitehttps://www.genome.jp/ftp/db/kofam/ko_list.gz- Remove RNA entries
- For rows withput a threshold, set threshold value to "-1", from "-"
- Change "knum" column header to "model"
- Then concatenate all the profiles together:
-
Pfam HMM profiles:
https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.hmm.gz- Run
hmmpress Pfam-A.hmm
- Run
Put these files in the same directory then set the following environment variable
HMM_DB_DIR=</host/hmm_dir>
docker pull ghcr.io/steineggerlab/foldseek:latest
Then download databases to a host directory mounted onto the container
docker run -v /host/db_dir:/app --rm \
ghcr.io/steineggerlab/foldseek databases \
Alphafold/Swiss-Prot /app/afdb-swissprot /tmp
docker run -v /host/db_dir:/app --rm \
ghcr.io/steineggerlab/foldseek databases \
PDB /app/pdb /tmp
docker run -v /host/db_dir:/app --rm \
ghcr.io/steineggerlab/foldseek databases \
ProstT5 /app/prost-t5-weights /tmp
Then, to use foldseek scripts, set the following environment variables
FOLDSEEK_DB_DIR=</host/db_dir>
This repo
docker build --platform linux/amd64 -t us-east1-docker.pkg.dev/needle-489321/tangle-docker/heap:latest .
docker push us-east1-docker.pkg.dev/needle-489321/tangle-docker/heap:latest
Foldseek image
(Note you may need to cleanup existing non-amd64 versions of the foldseem image first, using docker rmi)
docker pull --platform linux/amd64 ghcr.io/steineggerlab/foldseek:latest
docker tag ghcr.io/steineggerlab/foldseek:latest us-east1-docker.pkg.dev/needle-489321/tangle-docker/foldseek:amd64
docker push us-east1-docker.pkg.dev/needle-489321/tangle-docker/foldseek:amd64
And, to use these images on Google Cloud, make sure everything under FOLDSEEK_DB_DIR or HMM_DB_DIR are synced to Google Cloud storage, into a bucket.
Following script can be used to search for Pfam domains
PYTHONPATH=. python3 scripts/hmmscan-pfam.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498 \
query.faa test_result.tsv
The query database name should uniquely identify the source of the query accessions in the query fasta file.
This script uses hmmscan with GA cutoff scores as the reporting threshold, so reported domains are likely correct.
To prepare and submit this job to run on Google Cloud, use the following script
to create a run directory under runs (or whatever value for --run-dir), and
then follow instructions in the README file in that run directory.
PYTHPATH=. python3 gcloud/hmmscan-pfam/setup.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498.SRR9331959_algae_denovo \
--run-dir=runs \
proteins.faa.gz
You can search against KO HMM profile similarly,
PYTHONPATH=. python3 scripts/hmmscan-ko.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498 \
query.faa test_result_unassigned.tsv
To prepare and submit this job to run on Google Cloud, use the following script
to create a run directory under runs (or whatever value for --run-dir), and
then follow instructions in the README file in that run directory.
PYTHPATH=. python3 gcloud/hmmscan-ko/setup.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498.SRR9331959_algae_denovo \
--run-dir=runs \
proteins.faa.gz
But because KO HMM profiles do not use GA scores, but rather a separate threshold file, a second script is needed to make assignments.
PYTHONPATH=. python3 scripts/ko-assign.py \
test_result_unassigned.tsv test_result_assigned.tsv
Use the --scoring-ratio-min option to change the min ratio of bitscore over
threshold used to filter the results. Default is 0.99. If you change to 0.7,
for example, the resulting file may include putative matches worth
investigating.
Use scripts/hmmscan.py and scripts/ko-assign (with --threshold-file
option) to perform scans with other databases.
python3 scripts/hmmscan.py \
--query-database-name _ \
--target-database-name sieve_nfkb_ikb \
nfkb_ikb.hmm needle_proteins.faa needle_nfkb_ikb.tsv
python3 scripts/ko-assign.py \
--threshold-file nfkb_ikb_thresholds_low.csv \
needle_nfkb_ikb_assigned.tsv \
needle_nfkb_ikb.tsv
The threshold file needs to have the following headers
model,threshold,score_type,profile_type,f_measure,nseq,nseq_used,alen,mlen,eff_nseq,re_pos,definition
Use the following script to search the SwissProt database
PYTHONPATH=. python3 scripts/foldseek-swissprot.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498 \
query.faa test_result.tsv
The query database name should uniquely identify the source of the query accessions in the query fasta file.
From a .cif file, you can use the Docker image directly
docker run --rm -v /host/db-dir:/db -v /host/cif-file-dir:/app \
ghcr.io/steineggerlab/foldseek easy-search \
/app/fold_2026_03_12_14_08_model_0.cif /db/pdb /app/res.tsv /tmp \
--format-output "query,target,prob,evalue,bits,fident,qstart,qend,tstart,tend"
To prepare and submit this job to run on Google Cloud, use the following script
to create a run directory under runs (or whatever value for --run-dir), and
then follow instructions in the README file in that run directory.
PYTHPATH=. python3 gcloud/foldseek-swissprot/setup.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498.SRR9331959_algae_denovo \
--run-dir=runs \
proteins.faa.gz
The output file from above needs to be converted to the standard TSV format,
use the scripts/foldseek-swissprot.py script with --input-from-foldseek
option, e.g.
PYTHONPATH=. python3 scripts/foldseek-swissprot.py \
--query-database-name exp-doi:10.1126_sciadv.aba2498 \
--input-from-foldseek \
sequence_fs_raw.tsv sequence_fs.tsv
Using foldseek to create a 3Di database, with Prost-T5
docker run --rm --platform linux/amd64 \
-v <hmm_dir>:/hmm \
-v <foldseek_dir>:/foldseek \
-v <3di_output_dir>:/output \
ghcr.io/steineggerlab/foldseek createdb \
/hmm/ko.consensus.faa /output/ko.3di \
--prostt5-model /foldseek/prost-t5-weights --threads 6
# two steps to dump db to fasta file of 3Di sequences
docker run --rm --platform linux/amd64 \
-v <3di_output_dir>:/output \
ghcr.io/steineggerlab/foldseek \
lndb /output/ko.3di_h /output/ko.3di_ss_h
docker run --rm --platform linux/amd64 \
-v <3di_output_dir>:/output \
ghcr.io/steineggerlab/foldseek \
convert2fasta /output/ko.3di_ss /output/ko.3di.faa