Skip to content

Let skew take a directory of FASTQs - #14

Draft
micah-olivas wants to merge 2 commits into
mainfrom
worktree-skew-fastq-dir
Draft

micah-olivas wants to merge 2 commits into
mainfrom
worktree-skew-fastq-dir

Conversation

@micah-olivas

Copy link
Copy Markdown
Contributor

What

usortm skew now accepts a directory of FASTQs, not just a single file.

The reads argument resolves three ways: a single FASTQ, a directory searched
recursively for .fastq/.fq files (gzipped or not), or a mix of both. Files
found in a directory are counted together as one library, so a basecaller's
fastq_pass/ tree can be passed as-is with no concatenation step.

How

  • collect_fastqs() in usortm/qc/counting.py does the resolution. Non-FASTQ
    files in the tree are ignored; a directory holding no reads raises rather
    than silently counting nothing.
  • count_variant_reads() passes every file to a single minimap2 invocation.
    minimap2 accepts any number of query files and emits each read's alignments
    consecutively, in input order, so the per-read grouping logic is unchanged
    and no temporary concatenated copy of the reads is written.
  • count_fastq_reads() sums across files for the progress total.
  • The file list is recorded in skew_report.json under fastq_files, and in
    the project file under measured_skew. A directory listing can change
    between runs, so the report states which files were actually counted.

Tests

Full suite passes (262). Seven new tests cover resolution of files,
directories, mixed input, and the empty-directory error. The load-bearing one
splits a library FASTQ into a plain half and a gzipped half in a subdirectory
and asserts the per-variant counts match the single-file run exactly, so
recursion, gzip handling, and pooling are covered end to end.

Note on the first commit

This branch carries two commits. 9d5929e is a snapshot of the skew feature
that was still uncommitted in the working tree when this change was written;
it is the base so that 588c41a shows the directory-input change on its own.

Snapshot of work in progress, committed as the base for the FASTQ
directory-input change that follows.
The reads argument now accepts a directory, searched recursively for
.fastq/.fq files, gzipped or not, so a basecaller's fastq_pass/ can be
passed without concatenating it first. Files are counted together as one
library: minimap2 takes any number of query files and emits their
alignments in input order, so the per-read grouping needs no change.

collect_fastqs resolves a file, a directory, or a mix of both, and
count_fastq_reads sums over whatever it returns. Which files were counted
is recorded in skew_report.json and in the project file under
measured_skew, since a directory listing can change between runs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant