Let skew take a directory of FASTQs - #14
Draft
micah-olivas wants to merge 2 commits into
Draft
micah-olivas wants to merge 2 commits into
micah-olivas wants to merge 2 commits into
Conversation
Snapshot of work in progress, committed as the base for the FASTQ directory-input change that follows.
The reads argument now accepts a directory, searched recursively for .fastq/.fq files, gzipped or not, so a basecaller's fastq_pass/ can be passed without concatenating it first. Files are counted together as one library: minimap2 takes any number of query files and emits their alignments in input order, so the per-read grouping needs no change. collect_fastqs resolves a file, a directory, or a mix of both, and count_fastq_reads sums over whatever it returns. Which files were counted is recorded in skew_report.json and in the project file under measured_skew, since a directory listing can change between runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
usortm skewnow accepts a directory of FASTQs, not just a single file.The reads argument resolves three ways: a single FASTQ, a directory searched
recursively for
.fastq/.fqfiles (gzipped or not), or a mix of both. Filesfound in a directory are counted together as one library, so a basecaller's
fastq_pass/tree can be passed as-is with no concatenation step.How
collect_fastqs()inusortm/qc/counting.pydoes the resolution. Non-FASTQfiles in the tree are ignored; a directory holding no reads raises rather
than silently counting nothing.
count_variant_reads()passes every file to a single minimap2 invocation.minimap2 accepts any number of query files and emits each read's alignments
consecutively, in input order, so the per-read grouping logic is unchanged
and no temporary concatenated copy of the reads is written.
count_fastq_reads()sums across files for the progress total.skew_report.jsonunderfastq_files, and inthe project file under
measured_skew. A directory listing can changebetween runs, so the report states which files were actually counted.
Tests
Full suite passes (262). Seven new tests cover resolution of files,
directories, mixed input, and the empty-directory error. The load-bearing one
splits a library FASTQ into a plain half and a gzipped half in a subdirectory
and asserts the per-variant counts match the single-file run exactly, so
recursion, gzip handling, and pooling are covered end to end.
Note on the first commit
This branch carries two commits.
9d5929eis a snapshot of theskewfeaturethat was still uncommitted in the working tree when this change was written;
it is the base so that
588c41ashows the directory-input change on its own.