feat: use multiple reference sequences for minimizer index generation#404
feat: use multiple reference sequences for minimizer index generation#404ivan-aksamentov wants to merge 7 commits into
Conversation
Background:
Some pathogen datasets have significant genetic diversity that a single reference sequence cannot fully represent. This limits the accuracy of dataset auto-detection when query sequences are distant from the chosen reference. By allowing multiple reference sequences per dataset, the minimizer index can capture broader sequence diversity and improve detection rates.
Implementation:
- Add optional `files.minimizerReferences` field in pathogen.json (array of FASTA file paths)
- New `get_minimizer_refs()` function reads sequences from all listed files, falls back to main reference if field is absent
- `make_ref_search_index()` combines minimizers from all references using set union; uses average length for normalization
- Backward compatible: existing datasets work unchanged
Usage:
In pathogen.json, add array of FASTA paths containing representative sequences for the dataset:
```json
{
"files": {
"reference": "reference.fasta",
"minimizerReferences": [
"clade_a.fasta",
"clade_b.fasta"
]
}
}
```
Each FASTA file can contain one or more sequences. All sequences across all files contribute minimizers to the dataset's index entry.
Co-Authored-By: Claude <noreply@anthropic.com>
ivan-aksamentov
temporarily deployed
to
refs/pull/404/merge
GitHub Actions
Inactive
|
I tested the multi-ref minimizer functionality in this branch with CVA16: https://github.com/nextstrain/nextclade_data/tree/cva16-testing-sample |
Current Nextclade versions reject array values in the `files` section of `pathogen.json` with `invalid type: sequence, expected a string`. The `DatasetFiles` struct catches unknown keys with `rest_files: BTreeMap<String, String>` (https://github.com/nextstrain/nextclade/blob/f7db57f31/packages/nextclade/src/io/dataset.rs#L567-L569), which only accepts string values. An array like `minimizerReferences` fails deserialization before reaching the `other: serde_json::Value` catch-all. Move the lookup to `.buildFiles`, a top-level key absorbed by `VirusProperties`'s `other: serde_json::Value` (https://github.com/nextstrain/nextclade/blob/1400012f7/packages/nextclade/src/analyze/virus_properties.rs#L137-L138), which accepts any JSON type. This makes datasets with multiple minimizer reference files loadable by both old and new Nextclade versions.
- Depends on: #404 Current Nextclade versions reject array values in the `files` section of `pathogen.json` with `invalid type: sequence, expected a string`. Datasets declaring `minimizerReferences` as an array inside `files` fail to load. Move `minimizerReferences` from `.files` to a new top-level `.buildFiles` key: ```json { "files": { "reference": "reference.fasta", "genomeAnnotation": "genome_annotation.gff3", "changelog": "CHANGELOG.md" }, "buildFiles": { "minimizerReferences": [ "minimizer_refs/additional_refs_B.fasta", "minimizer_refs/additional_refs_B1a.fasta" ] } } ``` Unknown top-level keys are silently ignored by Nextclade, so the dataset remains loadable by both old and new versions. The companion script change reading from `buildFiles` is in #404 (9fc7f3e, 9fc7f3e). ### Work items - [x] Move `minimizerReferences` from `files` to `buildFiles` in `pathogen.json`
# Conflicts: # data_output/minimizer_index.json # scripts/rebuild
ivan-aksamentov
had a problem deploying
to
refs/heads/feat/multiref
GitHub Actions
Error
ivan-aksamentov
temporarily deployed
to
refs/pull/404/merge
GitHub Actions
Inactive
…port Rewrite scripts/minimizer as a subcommand CLI (build, search) with support for multi-ref indexes, pre-built index loading, and sorted FASTA output. Move search algorithm (search_one_query, filter_matches, deserialize_ref_search_index) from the script into lib/minimizer.py to share with other consumers (rebuild, suggest_datasets).
ivan-aksamentov
had a problem deploying
to
refs/heads/feat/multiref
GitHub Actions
Error
| "name": name, | ||
| "length": len(ref.seq), | ||
| "nMinimizers": len(minimizers) | ||
| "length": int(avg_length), |
There was a problem hiding this comment.
@nneune here the length of the reference is being set to the average length of over all minimizers
| all_minimizers = set() | ||
| total_length = 0 | ||
| for ref in ref_list: | ||
| minimizers = get_ref_search_minimizers(ref) |
There was a problem hiding this comment.
list of unique kmer hashes -> this means if the same kmer hash comes up in multiple references it still is only added once to the list
|
One thing we noticed is that when multiple references are merged, using literal merged In the multi-reference case, this assumption changes: a query close to reference A is not expected to hit all minimizers contributed by A+B+C after merging (especially with deduplication), which can make the denominator too large and lower otherwise good matches. We suggest fixing this at minimizer generation time (no schema change): for multi-reference datasets, set A practical approximation is: (or more exactly: This keeps Nextclade code/schema unchanged, preserves backward compatibility, and naturally reduces to current behavior for single-reference datasets. Notes on this matter here: https://docs.google.com/document/d/1EGIaSpCdSfgnPO0m4h9_OweGnw8wSGCio3ATK-xfof8/edit?tab=t.0#heading=h.tetsokd7p2ro |
- Support multiple reference sequences per dataset for minimizer index - Use expectedMinimizerHits as score denominator instead of union size - Add search_one_query() to reproduce client scoring in Python - Fix get_hash() sentinel to work above any cutoff - Rename pathogen.json key: minimizerIndex.references (was buildFiles.minimizerReferences)
4 participants
Data-side changes for multi-reference minimizer scoring.
See nextstrain/nextclade#1771 for the full description, design rationale, and CLI verification results.