How to Query the Human Metabolome Database Programmatically
By the end of this you will have a local DuckDB file holding every HMDB metabolite record with indexes on accession, InChIKey, chemical formula, and monoisotopic mass, a Python function that takes a measured m/z and returns candidate identifications across common adducts, and a cached HTTP client for the handful of cases where you need a single live record. You need about 40 GB of free disk (the uncompressed metabolite XML is roughly 6–7 GB, the spectral archive is much larger), Python 3.11+, lxml, pyarrow, and duckdb. Budget 20–40 minutes for the first parse on a laptop. HMDB stands for the Human Metabolome Database; the metabolome is the set of small molecules in a biological sample, and metabolomics is the measurement of it. HMDB is a reference database describing those molecules, not a repository of experimental datasets, which is a distinction that trips people up and that we come back to at the end.
1. Understand what “the HMDB API” means before you write code
There are three distinct access paths, and the search results conflate them.
The first is the commercial REST API sold through the Wishart lab’s commercialization arm. It is a paid, licensed service. If you are building a product on top of HMDB, that is the path you want, and the terms are negotiated by email. Do not assume the free web endpoints are a substitute for a license.
The second is the set of per-record endpoints the website exposes for free. Any metabolite page can be requested as XML by appending an extension:
curl -s -A 'you@example.com hmdb-client/0.1' https://hmdb.ca/metabolites/HMDB0000122.xmlThis returns the same record structure as the bulk dump, one metabolite at a time. The Bioconductor package hmdbQuery is a thin wrapper over exactly this. It is fine for tens of lookups and wrong for thousands. There is no documented rate limit, which means you should impose your own (one request per second, a persistent cache, a real User-Agent with a contact address).
The third path is the bulk downloads, and this is the one we recommend for essentially all analysis work. HMDB publishes the full metabolite set as a single zipped XML file, plus biofluid-specific subsets (serum, urine, CSF, saliva, feces, sweat), a protein file, an SDF of structures, and separate archives of predicted and experimental MS/MS and NMR spectra. HMDB 5.0 contains roughly 217,000 metabolite entries, most of them predicted or expected rather than experimentally detected in a human sample, which matters a great deal for how you filter. Mirroring locally removes the rate limit question, makes your analysis reproducible against a pinned version, and turns mass searches from N HTTP requests into one SQL scan. Reviews of the metabolomics database ecosystem have made the same point for well over a decade: the reference resources are designed to be downloaded and integrated, and the integration work is where the analysis value sits.12
2. Download and pin a version
mkdir -p hmdb/v5 && cd hmdb/v5
curl -L -O https://hmdb.ca/system/downloads/current/hmdb_metabolites.zip
curl -L -O https://hmdb.ca/system/downloads/current/serum_metabolites.zip
curl -L -O https://hmdb.ca/system/downloads/current/structures.zip
shasum -a 256 *.zip > SHA256SUMS
unzip -o hmdb_metabolites.zip # -> hmdb_metabolites.xml
ls -lh hmdb_metabolites.xmlRecord the checksums and the download date in your repo. HMDB’s current URL moves when a new release ships, and metabolite records get merged, so an accession that resolved last year may be a secondary accession today. Pinning the checksum is the only way to make a result reproducible.
Skip hmdb_all_spectra.zip unless you are doing spectral matching. It is tens of gigabytes of individual peak-list XML files and you almost certainly want a purpose-built spectral library format instead.
3. Stream-parse the XML into Parquet
Do not call etree.parse() on a 7 GB file. Use iterparse, and clear each element plus its preceding siblings, otherwise lxml keeps the whole tree alive and you will OOM around the 40,000th record.
One quirk to know: the monoisotopic mass field is spelled monisotopic_molecular_weight in the schema. That typo has been in HMDB for many years and is stable enough to rely on, but guard for both spellings.
from lxml import etree
import pyarrow as pa, pyarrow.parquet as pq
NS = "{http://www.hmdb.ca}"
def text(el, tag):
child = el.find(NS + tag)
return child.text if child is not None and child.text else None
def texts(el, path):
return [e.text for e in el.findall(NS + path) if e.text]
def record(el):
mono = text(el, "monisotopic_molecular_weight") or text(el, "monoisotopic_molecular_weight")
return {
"accession": text(el, "accession"),
"name": text(el, "name"),
"formula": text(el, "chemical_formula"),
"mono_mass": float(mono) if mono else None,
"avg_mass": float(text(el, "average_molecular_weight") or "nan"),
"smiles": text(el, "smiles"),
"inchikey": text(el, "inchikey"),
"status": text(el, "status"), # detected / expected / predicted
"super_class": text(el, "taxonomy/" + NS.join(["", "super_class"]).lstrip()) ,
"kegg_id": text(el, "kegg_id"),
"chebi_id": text(el, "chebi_id"),
"pubchem_cid": text(el, "pubchem_compound_id"),
"drugbank_id": text(el, "drugbank_id"),
"foodb_id": text(el, "foodb_id"),
"synonyms": texts(el, "synonyms/" + NS + "synonym"),
"biospecimens": texts(el, "biological_properties/" + NS +
"biospecimen_locations/" + NS + "biospecimen"),
"secondary": texts(el, "secondary_accessions/" + NS + "secondary_accession"),
}
def rows(path):
ctx = etree.iterparse(path, events=("end",), tag=NS + "metabolite")
for _, el in ctx:
yield record(el)
el.clear()
while el.getprevious() is not None:
del el.getparent()[0]
buf, writer = [], None
for i, r in enumerate(rows("hmdb_metabolites.xml"), 1):
buf.append(r)
if len(buf) == 20_000:
table = pa.Table.from_pylist(buf)
writer = writer or pq.ParquetWriter("hmdb_metabolites.parquet", table.schema)
writer.write_table(table); buf.clear()
print(i, flush=True)
if buf:
table = pa.Table.from_pylist(buf)
writer = writer or pq.ParquetWriter("hmdb_metabolites.parquet", table.schema)
writer.write_table(table)
writer.close()The nested taxonomy lookup above is awkward because of the namespace prefix on every path segment. In production, define def find(el, *tags) that walks children by local name and stop fighting ElementPath.
Two nested blocks are worth a second table rather than a column: normal_concentrations and abnormal_concentrations. Each concentration entry carries biofluid, value, units, subject age, sex, condition, and a literature citation. Flatten those into hmdb_concentrations.parquet keyed by accession. They are the most useful and the most dangerous part of HMDB, for reasons in step 7.
4. Load into DuckDB and add the indexes you will use
-- hmdb.sql
CREATE TABLE metabolites AS SELECT * FROM 'hmdb_metabolites.parquet';
CREATE INDEX idx_acc ON metabolites(accession);
CREATE INDEX idx_ikey ON metabolites(inchikey);
CREATE INDEX idx_mass ON metabolites(mono_mass);
CREATE INDEX idx_form ON metabolites(formula);
-- exploded synonym table for name resolution
CREATE TABLE synonyms AS
SELECT accession, lower(unnest(synonyms)) AS syn FROM metabolites
UNION
SELECT accession, lower(name) FROM metabolites;
CREATE INDEX idx_syn ON synonyms(syn);duckdb hmdb.duckdb < hmdb.sqlThe resulting file is a few hundred megabytes and answers a name lookup in under a millisecond. A full-table ppm scan over 217,000 rows takes a few milliseconds even without the index.
5. Mass-based candidate lookup
Most people reach for HMDB because an untargeted LC-MS run produced a feature at some m/z and retention time and they want candidate identities. The query is: neutral mass = m/z minus the adduct shift, then all metabolites within a ppm window.
import duckdb
ADDUCTS = { # m/z shift relative to neutral monoisotopic mass
"[M+H]+": 1.007276,
"[M+Na]+": 22.989218,
"[M+NH4]+": 18.033823,
"[M-H]-": -1.007276,
"[M+Cl]-": 34.969402,
"[M+HCOO]-":44.998201,
}
con = duckdb.connect("hmdb.duckdb", read_only=True)
def annotate(mz, ppm=5, detected_only=True, adducts=ADDUCTS):
out = []
for name, shift in adducts.items():
neutral = mz - shift
tol = neutral * ppm / 1e6
q = """SELECT accession, name, formula, mono_mass, status,
1e6*(mono_mass-?)/? AS err_ppm
FROM metabolites
WHERE mono_mass BETWEEN ? AND ?"""
params = [neutral, neutral, neutral - tol, neutral + tol]
if detected_only:
q += " AND list_contains(biospecimens, 'Blood')"
out += [(name, *row) for row in con.execute(q, params).fetchall()]
return sorted(out, key=lambda r: abs(r[-1]))Two things about this function. First, the detected_only filter is doing more work than the ppm window. Without it, a 5 ppm search on a plasma feature routinely returns 40+ candidates, most of them predicted structures that have never been observed in a human. Filtering to metabolites with a blood biospecimen annotation, or to status = 'detected', cuts that by an order of magnitude and is the single highest-value filter in the whole database.
Second, mass alone does not identify anything. Isomers are indistinguishable by accurate mass, and a 5 ppm window at m/z 300 is 1.5 mDa, which is not enough to separate most formula candidates cleanly. You need MS/MS fragmentation, retention time against an authentic standard, or both. If you want adduct-aware, multi-database searching without building the combinatorics yourself, CEU Mass Mediator implements exactly this over HMDB, KEGG, LipidMaps, and others with a documented REST API, and is a reasonable thing to call rather than reimplement.3
6. The live endpoints, used sparingly
For interactive one-offs, or to check whether a pinned accession has been merged, hit the record endpoint with a cache:
import requests, requests_cache
from lxml import etree
requests_cache.install_cache("hmdb_http", expire_after=60*60*24*30)
S = requests.Session()
S.headers["User-Agent"] = "yourname@example.com hmdb-client/0.1"
def fetch(accession):
r = S.get(f"https://hmdb.ca/metabolites/{accession}.xml", timeout=30)
r.raise_for_status()
return etree.fromstring(r.content)The site’s search endpoint (/unearth/q?query=...&searcher=metabolites) returns HTML, not JSON, and scraping it at volume is both fragile and rude. Resolve names against your local synonyms table instead. Reserve HTTP for accessions you cannot find locally, which usually means your dump is stale.
7. Joining HMDB to your own measurements
The join key is InChIKey, not name. Metabolite names are a swamp: “glucose”, “D-glucose”, “dextrose”, and “alpha-D-glucopyranose” are four strings for overlapping chemical concepts, and vendor panels are inconsistent about stereochemistry. Normalize your panel names through RefMet (the Metabolomics Workbench nomenclature standard, which has a free REST API) to get structures, convert structures to InChIKey with RDKit, then join on the first 14 characters of the InChIKey if you want to collapse stereoisomers, or the full key if you do not.
from rdkit import Chem
from rdkit.Chem.inchi import MolToInchiKey
key = MolToInchiKey(Chem.MolFromSmiles("OC[C@H]1OC(O)[C@H](O)[C@@H](O)[C@@H]1O"))Once mapped, HMDB’s cross-references (kegg_id, chebi_id, pubchem_cid) are what feed pathway and enrichment analysis downstream. The identifier you need depends on the tool: KEGG compound IDs for KEGG pathway mapping, HMDB accessions for SMPDB and for MetaboAnalyst’s enrichment sets. Coverage and results vary substantially across enrichment tools even on the same input list, so pick one, understand its background set, and report which you used.45 Broader surveys of the tooling and package ecosystem are a good orientation if you are choosing for the first time.6
On the concentration tables: HMDB’s normal and abnormal concentration entries are literature-derived, pooled across studies with different assays, different populations, and different sample handling. They are useful for order-of-magnitude sanity checks (“is my reported plasma citrate in the tens of micromolar or the tens of millimolar?”). They are not reference intervals for you, and the disease associations attached to a metabolite record are literature co-occurrences, not diagnostic criteria. Interpreting your own values against a clinical question is a conversation with a physician, not a SQL query.
8. Where HMDB stops
HMDB describes molecules. It does not hold raw experimental datasets. If you want other people’s spectra and study designs to compare against, that is MetaboLights, which archives raw files, metadata, and study protocols under a standardized reporting format.7 If a chunk of your plasma metabolome looks microbial in origin (indoles, phenyl-sulfates, secondary bile acids), MiMeDB from the same group maps microbial metabolites to producing taxa and is the better reference for that subset.8 For questions that span metabolites, genes, proteins, and phenotypes at once, the knowledge-graph projects are ahead of any single database: KRAKEN integrates multi-omic and wellness data with provenance tracking on every edge, and ROBOKOP federates biomedical sources into a queryable graph.910 Provenance tracking matters more than it sounds. When an assertion propagates through three databases, you want to know which primary source it came from.
9. How to cite HMDB
Cite the most recent database-issue paper, not the URL. As of writing that is Wishart et al., “HMDB 5.0: the Human Metabolome Database for 2022,” Nucleic Acids Research, 2022 database issue. Also state the version and your download date in your methods, because “we searched HMDB” without a version is not reproducible. If you used MiMeDB for microbial metabolites, cite it separately.8
Common problems
Memory blowup during parsing. You called etree.parse(), or you called el.clear() without deleting preceding siblings. The while el.getprevious() is not None loop is the part people omit.
Empty results from XPath. Every element in the dump is in the http://www.hmdb.ca namespace. el.find("accession") returns None; el.find("{http://www.hmdb.ca}accession") works. Write a local-name helper early.
Missing monoisotopic masses. Roughly the entries without a resolved structure have nulls in both mass fields. Filter WHERE mono_mass IS NOT NULL before any ppm scan, or your tolerance arithmetic silently drops rows.
Accessions that 404. HMDB merges duplicate records and the loser becomes a secondary accession. Build a lookup from your secondary column to the primary accession and resolve through it before giving up.
Too many candidates per feature. Expected. Tighten ppm only if your instrument’s mass accuracy justifies it (5 ppm is realistic for a well-calibrated Orbitrap, 20+ ppm for many TOFs), then filter on biospecimen and detection status, then use isotope pattern and MS/MS. Mass plus database hit is a hypothesis with a confidence level, and the Metabolomics Standards Initiative identification levels exist precisely to make you state which level you reached.
Getting rate-limited or blocked. You looped HTTP requests without a delay. Mirror locally.
Commercial use. The free download is for academic and non-commercial use. If your work is commercial, license it properly before you ship anything derived from the dump.
Oak builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.
Footnotes
-
Eden P. Go. Database Resources in Metabolomics: An Overview. Journal of Neuroimmune Pharmacology, 2009. https://doi.org/10.1007/s11481-009-9157-3 ↩
-
JOHN L. MARKLEY, MARK E. ANDERSON, QIU CUI, et al. NEW BIOINFORMATICS RESOURCES FOR METABOLOMICS. Biocomputing 2007, 2006. https://doi.org/10.1142/9789812772435_0016 ↩
-
Alberto Gil-de-la-Fuente, Joanna Godzien, Sergio Saugar, et al. CEU Mass Mediator 3.0: A Metabolite Annotation Tool. Journal of Proteome Research, 2018. https://doi.org/10.1021/acs.jproteome.8b00720 ↩
-
Anna Marco-Ramell, Magali Palau-Rodriguez, Ania Alay, et al. Evaluation and comparison of bioinformatic tools for the enrichment analysis of metabolomics data. BMC Bioinformatics, 2018. https://doi.org/10.1186/s12859-017-2006-0 ↩
-
Luiz Gustavo Gardinassi, Jianguo Xia, Sandra E Safo, et al. Bioinformatics Tools for the Interpretation of Metabolomics Data. Current Pharmacology Reports, 2017. https://doi.org/10.1007/s40495-017-0107-0 ↩
-
Saurav Kumar Mishra, Hamadou Mamoudou, Pragya Rai, et al. Advancement in Metabolomics Research via Tools, Databases, and Packages: A Cancer Perspective. Archives of Computational Methods in Engineering, 2026. https://doi.org/10.1007/s11831-026-10517-7 ↩
-
Namrata S. Kale, Kenneth Haug, Pablo Conesa, et al. MetaboLights: An Open‐Access Database Repository for Metabolomics Data. Current Protocols in Bioinformatics, 2016. https://doi.org/10.1002/0471250953.bi1413s53 ↩
-
David S Wishart, Eponine Oler, Harrison Peters, et al. MiMeDB: the Human Microbial Metabolome Database. Nucleic Acids Research, 2022. https://doi.org/10.1093/nar/gkac868 ↩ ↩2
-
Amy K. Glen, Drew Witherington, Trent Leslie, et al. KRAKEN: A provenance-tracked knowledge graph for multiomic and wellness research. 2026. https://doi.org/10.64898/2026.08.18.745544 ↩
-
Chris Bizon, Steven Cox, James Balhoff, et al. ROBOKOP KG and KGB: Integrated Knowledge Graphs from Federated Sources. Journal of Chemical Information and Modeling, 2019. https://doi.org/10.1021/acs.jcim.9b00683 ↩