What a Genome Manager Needs to Do (and How to Build One Yourself)

A “genome manager” is the web portal a consumer sequencing company gives you to register your kit, track your sample through the lab, and read reports once they are ready. Dante Labs popularized the name. Nebula, Sequencing.com, and others ship the same thing under different labels. It is a file browser with a report store attached. If you intend to analyze your own data, the portal is a delivery mechanism, not a workspace: you download the FASTQ, BAM/CRAM, and VCF once, and after that the useful genome manager is the one you run yourself.

That is the shape of the problem. Below is what the vendor portal gives you, what it does not, and the setup we use to keep a person’s molecular data organized over years rather than over one download session.

What a vendor portal is and is not

The portal handles four things: kit registration and consent, sample status (received, extracted, libraries prepped, sequencing, analysis), file delivery via signed S3-style URLs, and an upsell catalog of interpretive reports. Reports are typically PDFs or HTML rendered from a fixed annotation pipeline run against your VCF.

What it does not do:

• Tell you the reference build. This matters more than anything else in the portal. A VCF against GRCh37/hg19 and one against GRCh38 have different coordinates for every variant, and most 2015–2020-era annotation databases are hg19-native while most current pipelines are GRCh38. Check the ##reference header line and the contig lengths (chr1 is 249,250,621 bp in hg19 and 248,956,422 bp in GRCh38).
• Tell you caller, pipeline version, or filtering. A VCF with every record marked PASS and no FILTER definitions in the header usually means filtering was never applied, not that every call is good.
• Give you the intermediate files. If you only get a VCF, you cannot re-call, you cannot inspect read support for a surprising variant, and you cannot re-align to a different build. Get the CRAM or BAM.
• Survive vendor churn. Portals go dark. Companies get acquired. Treat delivery as a one-time export.

The first hour after download

Verify before you analyze. Corrupt or truncated downloads are the most common failure mode and they fail silently for a long time.

# BGZF integrity, not just gzip integrity
bgzip -t sample.vcf.gz
bcftools index -t sample.vcf.gz

# Does the file agree with itself about the reference?
bcftools view -h sample.vcf.gz | grep -E '^##(reference|contig=<ID=chr1,)'

# Call counts by type — a WGS germline VCF for one person
# should land near 4.5-5.0M SNVs and 500-900k indels
bcftools stats sample.vcf.gz | grep -E '^SN'

If SNV count is under 4 million, you probably have an exome or a heavily filtered file. If it is over 6 million, either the sample is not of European-ish ancestry (non-reference variant counts rise with genetic distance from GRCh38) or filtering is absent and you are looking at noise.

For alignments:

samtools quickcheck -v sample.cram   # catches truncation, prints nothing if fine
samtools coverage sample.cram | head -25

Mean depth of 30x nominal often lands at 25–28x after duplicate removal. Below 20x, heterozygous sites start dropping out and your genotype calls get unreliable in a way no downstream tool will warn you about.

Then convert to CRAM with the exact reference FASTA you will keep forever, and store a checksum:

samtools view -T GRCh38_full_analysis_set_plus_decoy_hla.fa -C -@8 \
  -o sample.cram sample.bam
samtools index sample.cram
sha256sum sample.cram sample.cram.crai > sample.cram.sha256

CRAM is reference-dependent. Losing the exact FASTA means losing the data. Keep the FASTA, its .fai, and its checksum in the same archive tree as the CRAM.

A layout that survives five years

We use a flat, dated, immutable-raw layout. No database, no application server, nothing that needs maintenance.

profile/
  raw/                       # never edited, never regenerated
    2024-03-11_wgs_illumina/
      sample.cram
      sample.cram.crai
      sample.hard-filtered.vcf.gz
      MANIFEST.json          # vendor, kit, build, pipeline, checksums
    2024-03-11_rnaseq_pbmc/
      quant.sf               # salmon
      aux_info/
    2025-01-20_olink_plasma/
      npx_long.csv
    labs/
      2024-2026_panels.csv   # LOINC-coded, one row per analyte per draw
    cgm/
      2024-06_dexcom_g7.csv
  refs/
    GRCh38_full_analysis_set_plus_decoy_hla.fa{,.fai}
    gencode.v44.annotation.gtf.gz
  derived/                   # everything reproducible, safe to delete
  notebooks/
  Makefile

MANIFEST.json is the part people skip and regret. Minimum fields: assay, tissue, collection date, vendor, library kit, instrument, reference build, aligner and version, caller and version, and SHA-256 for every file. Write it by hand if the vendor will not give it to you. Two years later, when a variant looks interesting, the first question is always “what called this and against what.”

Everything in derived/ is produced by a Makefile or a Snakemake workflow, so it can be thrown away and rebuilt. Raw files go to two physical locations plus one cold copy. WGS CRAM is roughly 15–25 GB per genome, so this is a $10/month problem, not an infrastructure problem.

Annotation is the layer you maintain

Annotation goes stale fast. ClinVar ships weekly. gnomAD, dbNSFP, and ancestry-aware frequency panels update on their own cadence. A report generated in 2023 reflects a 2023 snapshot of the literature, which is why the portal’s PDF ages and your pipeline should not.

We use Ensembl VEP with a cached, versioned plugin set, pinned in the Makefile:

vep --cache --offline --dir_cache "$VEP_CACHE" \
  --assembly GRCh38 --fasta "$REF" \
  --input_file sample.vcf.gz --format vcf \
  --vcf --compress_output bgzip --output_file annotated.vcf.gz \
  --everything --pick_allele_gene \
  --custom clinvar_20260601.vcf.gz,ClinVar,vcf,exact,0,CLNSIG,CLNREVSTAT \
  --plugin CADD,whole_genome_SNVs.tsv.gz \
  --plugin SpliceAI,snv=spliceai_scores.snv.hg38.vcf.gz \
  --fork 8

Record the ClinVar release date in the output filename. When a friend or a clinician asks where a classification came from, you can answer precisely. Re-run quarterly. The diff between two runs is short and is the most informative thing in the whole setup: a handful of variants move between “uncertain significance” and something else each quarter, almost always because review status changed, not because the biology did.

Filter for review status, not just significance. CLNREVSTAT values of criteria_provided,_multiple_submitters,_no_conflicts and above (two stars) are worth attention. A single-submitter “pathogenic” with no assertion criteria is frequently wrong.

Nothing produced by this pipeline is a diagnosis. Variant classification for clinical purposes requires a CLIA/CAP-accredited lab and a clinician or genetic counselor to interpret it in the context of your personal and family history. Research-grade WGS is not confirmatory. If something looks actionable, the correct next step is orthogonal confirmation in an accredited lab, ordered by a physician.

Joining the genome to everything else

The genome is the static layer. The value compounds when the same manager holds the time series: RNA-seq counts, plasma proteomics NPX values, blood panels, CGM traces. Join keys are what make this work.

• Genes: use Ensembl stable IDs without version suffix (ENSG00000141510) as primary, HGNC symbol as display only. Symbols get renamed.
• Proteins: UniProt accession. Olink and SomaScan both map to it, imperfectly, and one assay target can cover multiple gene products.
• Lab analytes: LOINC codes, plus explicit units. Glucose in mg/dL versus mmol/L is an 18x error waiting to happen.
• Time: ISO 8601 with timezone, on every row. CGM at 5-minute resolution against a meal log is useless if either one has drifted.

Store all of it as long-format Parquet, one row per (subject, analyte, timestamp, value, unit, assay_batch). Batch is not optional. Proteomic panels and clinical chemistry both shift between lots, and a step change on a January draw is far more often a new reagent lot than a change in you.

Questions people also ask

What is the world’s largest DNA database? For raw sequence, the INSDC partnership (NCBI’s SRA, the European Nucleotide Archive, and DDBJ) is the largest public archive, holding tens of petabytes of reads across all organisms. For human genomes tied to health records, the UK Biobank (about 500,000 participants with whole-genome sequencing completed) and the All of Us Research Program are the largest broadly accessible resources. National forensic databases like the FBI’s CODIS are large by profile count but hold STR profiles, not sequence.

What is the largest biological database in the world? Depends on what you count. By bytes, the SRA. By number of records, ENA and GenBank sequence collections. By structural content, the AlphaFold Protein Structure Database holds over 200 million predicted structures, which dwarfs the experimentally determined PDB by count while being a different kind of object entirely.

What does a genomic analyst do? Runs and audits the path from reads to interpretation: QC on FASTQ and alignments, variant calling and filtering, annotation against ClinVar/gnomAD/functional predictors, and structured reporting. The bulk of the day is failure-mode work, checking coverage gaps, contamination estimates, sex-chromosome concordance, and sample swaps. In a clinical lab the analyst prepares evidence and a board-certified director signs the report.

Should I keep FASTQ or is CRAM enough? CRAM against a pinned reference is enough for nearly everything, including re-calling with a different caller. Keep FASTQ only if you expect to re-align to a future reference build (a pangenome graph reference, for example) and have the storage to spare. CRAM is roughly a third the size of BAM and a fraction of gzipped FASTQ.

Oak builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.