Enable pwrite for raw FASTQ file output#700
Conversation
Reuse the existing ordered pwrite path for multi-threaded non-stdout FASTQ output so raw and gzip file writers share the same direct-write flow. Constraint: Preserve gzip compression behavior and stdout writer path Confidence: medium Scope-risk: narrow 💘 Generated with Crush Co-Authored-By: Crush <crush@charm.land>
|
Hi @KimYannn I tested this pr, but it didn't work. My command with a pair of SRA paired-end files. |
|
Hi @sfchen, thanks for testing. I reproduced your case and you're right — for raw (uncompressed) FASTQ output this PR gives no speedup, so I'd hold off merging it as-is. Repro (4M PE150 reads, gz input → raw
So output is correct (identical to the single-writer-thread path and to Why: for raw output the write stage isn't the bottleneck. A single writer thread doing pwrite only pays off for Suggest we close this PR. If you'd still like the unified write path for code-simplicity reasons (raw + gz sharing one flow) rather than performance, I can rescope and re-benchmark — let me know. |
|
ok, closing |
Summary
Test plan
make -j INCLUDE_DIRS=/opt/homebrew/include LIBRARY_DIRS=/opt/homebrew/lib./fastp test./fastp --version./fastp -i testdata/R1.fq -o /dev/nullgzip -t💘 Generated with Crush