Researchers need a local triage tool that flags uploaded DNA sequence files (FASTA,…

Researchers need a local triage tool that flags uploaded DNA sequence files (FASTA, FASTQ, JSON) which may contain ASCII text encoded into DNA, without decoding or displaying that text. Goal: give a fast, honest first-pass verdict (Needs manual recheck / Worth a closer look / Looks OK) with candidate region coordinates and scores. Success metric: a 10 M base file finishes a full scan in under 5 minutes (about 300 seconds) on a standard laptop, and all tests pass (38 existing plus new ones).

A single local researcher. No accounts, no roles, no sharing.

Default: One researcher using the app on their own machine.

1. Researcher starts the app locally (127.0.0.1:8501); the start page shows the three limits (10 MB file size, 10 M total bases, 10 M windows per record) as three plain-text lines directly above the upload control.
2. Uploads a FASTA, FASTQ or JSON file (max 10 MB). The file is kept in memory for the session.
3. App parses strictly; malformed input shows a parse error screen.
4. After the file is chosen and before Run scan is selected, if the file or any record is above 90% of a limit, the near-limit warning is shown ("File is above 90% of the 10 MB size limit", "File is above 90% of the 10 M base limit" or "Record X is above 90% of the windows-per-record limit"). The scan still runs when Run scan is selected.
5. App runs composition score, ASCII triage and, when a reference is available, the experimental matched-reference comparison, then optional host-specific threshold. Marker screening is not active and no marker-match result is shown.
6. Results page shows, in order: verdict card, reference notices (if any), plain-language explanation, candidate table in collapsed technical details, then limitations last.
7. During a session the researcher can change only the reference (NCBI organism name or uploaded reference FASTA) and re-run the scan on the file already in memory, without re-uploading. The caps and warning thresholds stay fixed.

Must-haves:

• Strict parsing of FASTA, FASTQ, JSON with ParseError on malformed input.
• 10 MB upload cap, 10 M total base cap, 10 M windows-per-record cap, each shown as a plain-text line on the start page directly above the upload control.
• Near-limit warning, shown after the file is chosen and before Run scan, when the file is above 90% of the 10 MB size limit, above 90% of the 10 M base limit, or any single record is above 90% of the 10 M windows-per-record limit. Wording: 'File is above 90% of the 10 MB size limit', 'File is above 90% of the 10 M base limit' or 'Record X is above 90% of the windows-per-record limit'. The scan still runs.
• ASCII triage over 4 base frames, with XOR 0x3C profile evaluated together with the raw stream.
• Verdict card with the 'triage hint only, not proof of safety' wording directly under the Looks OK label only; limitations list at the bottom of the results page; no decoded text anywhere.
• Optional host-specific threshold from NCBI or uploaded reference FASTA.
• Re-run on demand: the researcher can change only the reference and re-run the scan on the file already held in memory for the session. Caps and warning thresholds are not changeable.
• Candidate table shows only start, end and score columns, sorted by score highest first, with start and end ascending as tie-breakers.

Start page

• Three plain-text lines directly above the upload control: 10 MB file size, 10 M total bases, 10 M windows per record.
• Streamlit version check: if the installed version is below the pinned minimum, a clear error is shown on the start page and no scan runs.

Results page (top to bottom)

1. Verdict card: verdict label; for 'Looks OK' only, 'triage hint only, not proof of safety' directly under the label; one reference line naming the source only ('Reference used: host-specific NCBI reference', 'Reference used: uploaded reference FASTA' or 'Reference used: generalized background model only'), with no sequence count. Amber and Red verdicts show no 'triage hint only' line.
2. Reference notices, if any, above the explanation, each with its exact wording; the verdict is still produced from the generalized background model.
3. Plain-language explanation.
4. Candidate table inside collapsed technical details: columns start, end and score only, sorted by score highest first, with start and end ascending as tie-breakers. Inside the same collapsed technical details: the matched-reference comparison, marked EXPERIMENTAL, with the 1% and 5% cutoffs and the 20-window persistence rule, each labelled PROVISIONAL, and no sensitivity, precision or false-positive rate claimed until held-out validation is complete.
5. Limitations list, always shown last, below the candidate table, stating the scan is a triage hint and cannot confirm a file is safe.

No CSV download button in v1. Marker screening is not active in v1: the results page states this and shows no marker-match result.

Component library
Streamlit built-in components (st.file_uploader, st.error, st.metric, st.expander, st.dataframe, st.download_button, st.container, st.fragment). No custom CSS or external component library.

Colors (hex)

• Verdict green: #2E7D32
• Verdict amber: #F9A825
• Verdict red: #C62828
• Background light mode: #FFFFFF
• Background dark mode: #0E1117
• Text dark mode: #FAFAFA
• Text light mode: #262730

Font
System default (Streamlit default sans-serif), base font 16 px.

Corner radius
8 px on cards and verdict containers.

spacing
16 px between major sections; 8 px between table rows.

Light/dark mode
Follow Streamlit theme; verdict colours meet WCAG AA contrast in both modes.

Tone
Plain language, cautious, no safety claims. Phrases like "Looks OK" mean triage hint only, not proof of safety.

Reference look
Existing Streamlit localhost app (see HANDOVER.md and current app.py). Suggest attaching screenshots of apps you like for visual comparison.

---

What was asked about

• Preserve current visual style; do not introduce a new design system.
• Keep the collapsed technical-details container and verdict card layout.

NCBI reference cache (local folder)
A folder of downloaded reference FASTA files plus a JSON index. No database.

index.json fields

A second scan of the same organism reads from this cache and makes no network request. Sequence contents are never written to the index or logs.

NCBI E-utilities

• Direct retrieval: when every uploaded record header contains the same supported versioned nucleotide accession (NC, NG, NM, NR, NT, NW, NZ, AC, CP, CM, AE, AL), the app fetches that accession with EFetch. Mixed or missing accessions fall back to organism lookup.
• Organism lookup: the organism name from the header is searched on nuccore for a RefSeq chromosome or complete-genome title excluding plasmids, then one result is fetched.
• Sent to NCBI: only the accession (when used) and the organism label. No uploaded sequence content is sent.
• Optional API key: read from the NCBI_API_KEY environment variable only; never written to disk, logs or the cache.
• Limits: 12-second timeout, at least 0.34 seconds between requests, downloads capped at 10 MiB and must parse as FASTA.
• Failure: network failure, rate limiting, HTTP error, invalid response, oversized reference and no search result each show a distinct notice above the explanation and fall back to the generalized background model; the app never crashes.
• Cache: valid references stored under ~/.polycheck/references as FASTA plus index.json; a cache write failure is logged and does not discard a fetched reference.
• Uploaded reference FASTA stays local.
• Assembly accessions (GCA/GCF) are not resolved directly in v1.

Not needed: a single local researcher on 127.0.0.1 with no accounts, roles or sharing. The only external access is the optional NCBI E-utilities lookup, which is unauthenticated unless an optional API key is set in an environment variable.

• Near-limit warning: after the file is chosen and before Run scan, shown above 90% of the 10 MB size limit, 10 M base limit or 10 M windows-per-record limit, with the exact wording from edge_cases; the scan still runs when Run scan is pressed.
• Matched-reference comparison (EXPERIMENTAL, in technical details): the 1% and 5% decision cutoffs and the 20-window persistence rule are labelled PROVISIONAL and shown to the researcher. No sensitivity, precision or false-positive rate is claimed until held-out validation is complete.
• Runtime: a 10 M base file scan is measured as the median of three runs on a named laptop, recording model, CPU, RAM and OS. Result: UNVERIFIED until the measurement is written here.
• Test suite: python -m unittest discover -s tests must be run after the latest accession and matched-reference changes; the pass count (38 existing plus new tests) is reported. Any failure blocks readiness. Result: NOT YET RUN.

• Upload above 90% of the 10 MB size limit, above 90% of the 10 M base limit, or any single record above 90% of the 10 M windows-per-record limit: show the warning after the file is chosen and before Run scan, using the exact wording 'File is above 90% of the 10 MB size limit', 'File is above 90% of the 10 M base limit' or 'Record X is above 90% of the windows-per-record limit'; the scan still runs.
• Upload over 10 MB, over 10 M bases, or any record over 10 M windows: reject with a clear message before scanning.
• No organism name in the FASTA header: show the notice 'No organism name found in the FASTA header, so no host-specific reference was used' above the explanation; use the generalized background model only and still produce the verdict.
• Organism not found in NCBI: show the notice 'Organism not found in NCBI, so no host-specific reference was used' above the explanation; use the generalized background model only and still produce the verdict.
• NCBI unreachable, timeout or HTTP 429: show the notice that the host-specific reference was unavailable above the explanation; use the generalized background model only and still produce the verdict; never crash.
• No NCBI reference available: the background is the internal generalized model only, never derived from the uploaded file.
• Streamlit below the pinned minimum at startup: show a clear error on the start page naming the installed and required versions; do not run the scan.
• Cache file corrupted or index.json unreadable: ignore the cache entry, re-download or fall back, and log a redacted message.
• FASTQ quality lines containing A, C, G or T: never passed to the DNA scoring path.

Language & runtime: Python 3.10+ (existing app).

Framework: Streamlit, pinned to the exact installed version in requirements.txt (e.g. streamlit>=1.49.0). At startup the app reads the installed version and, if it is below the pinned minimum, shows a clear error on the start page instead of failing silently when @st.fragment is used.

Libraries: biopython, pandas, numpy (already in requirements.txt).

Hosting: Local only, bound to 127.0.0.1:8501. No cloud, no managed host.

Storage: In-memory processing only. Optional NCBI reference cache on the local filesystem, stored as a folder of downloaded FASTA files plus a JSON index.

Constraints: Outbound network access limited to the NCBI E-utilities lookup (organism name only). No decoding or displaying of text. No encoding or reconstruction of DNA.

Entry point: app.py (about 1,000 lines). Extend in place; do not rewrite.

Key functions to preserve behaviour of: parse_fasta, parse_fastq, parse_json, parse_upload, acgt_runs, frame_bytes, byte_distribution_score, ascii_triage, find_orfs, region_overlaps_orf, synthetic_proxy_score, marker_match, extract_organism_name, fetch_ncbi_reference, compute_background_distribution, host_specific_threshold.

Do not change: the no-decode and no-encode rules, the localhost-only binding, the 10 MB / 10 M base / 10 M windows caps, the CSV export columns (start, end, score), the verdict wording, the 38 existing tests.

Tests: python -m unittest discover -s tests must keep all 38 passing; add tests for each new change.

1. FASTQ quality-line separation and NCBI cache with offline fallback. Strictly separate sequence lines from quality lines before scoring. Cache downloaded references as a folder of FASTA files plus a JSON index; on network failure, 429 or timeout, fall back to the internal generalized background model and show a visible notice.
2. Performance: vectorized sliding windows and ORF speedup. Use NumPy stride tricks for the 16-byte windows and replace the manual ORF search with a faster method. Confirm a 10 M base file scans in under 5 minutes.
3. Dual XOR 0x3C evaluation. Evaluate raw and XOR-masked streams in the same windowing pass and deduplicate candidates across both profiles.
4. Module split. Move engine code to engine.py, parsing to parsers.py, keep app.py as the presentation layer. Behaviour unchanged, 38 tests still passing.
5. @st.fragment rendering. Wrap the whole results pipeline (verdict, explanation, technical details, export) in one @st.fragment so presentation toggles never re-run the scan.

Each step adds new tests; the 38 existing tests stay unchanged and the full suite runs before finishing.

Environment: Local Windows machine (path C:\Users\nicor\OneDrive\Documents\Default Project\). Python 3.10+.

Start command: python -m streamlit run app.py --server.address 127.0.0.1 --server.port 8501

Test command: python -m unittest discover -s tests

Secrets: Optional NCBI API key, read from an environment variable only; never written to disk, logs or the cache.

Environments: Single local environment; no staging or production.

Confirmed defaults:

• App name: 'Polycheck - DNA triage scanner'.
• Users: one researcher using the app on their own machine, no accounts or roles.
• NCBI API key: not required; the app works without one using throttled requests and cache fallback. An optional key may be set via the NCBI_API_KEY environment variable only.
• XOR key: 0x3C.
• Streamlit: pinned to the installed version.

Open items blocking readiness:

• Runtime: a 10 M base file scan (median of three runs on a named laptop, recording model, CPU, RAM and OS) is UNVERIFIED until measured.
• Tests: python -m unittest discover -s tests has not been rerun after the latest accession and matched-reference changes; the pass count must be reported and any failure blocks readiness.