Mail (mailcrawl)
This page shows how to connect Gmail, IMAP, Maildir, or static .eml/.mbox email through the mailcrawl CLI. AutoRAG never opens archive.sqlite directly; the CLI owns the archive and the index.
Prerequisites
@nomadamas/mailcrawl@0.3.0or newer (Node.js 24+). Do not use 0.1.3 or earlier: a no-op sync followed byindexfails withtext array must be non-empty.- A Himalaya account configured separately, or
connector.source: "files"withconnector.pathsto index static.emlfiles,.mbox/.mbxarchives, Apple Mail bundles, or directories without Himalaya. - Email search is the sole mail path; the former
mail-exportconnector is retired.
Connect
Refresh runs mailcrawl sync, then mailcrawl index.
{
"datasources": {
"mailcrawl": {
"type": "mailcrawl",
"instanceId": "personal",
"description": "Personal Gmail inbox, last few years.",
"connector": {
"binaryPath": "mailcrawl",
"account": "personal",
"mailbox": "INBOX"
}
}
}
}| Connector key | Meaning |
|---|---|
binaryPath | Explicit mailcrawl path; default is a bare mailcrawl lookup. |
account / mailbox | Himalaya account and mailbox; they also narrow retrieval results. |
source | files indexes .eml/.mbox on disk; otherwise Himalaya sync. |
paths | Paths used with source: "files". |
dataDir | Explicit mailcrawl data directory; set it only to move the native archive. |
pageSize / concurrency / readTimeoutMs | IMAP window size, parallel reads, and per-message read deadline. |
includeCategories / excludeCategories | Categories re-included or excluded from sync. |
rebuildIndex | Forces a vector-store rebuild on index. |
timeoutMs / indexTimeoutMs | Interactive and sync/index budgets; indexTimeoutMs defaults to one hour. |
What is indexed
mailcrawl sync fetches envelopes and messages; mailcrawl index builds the search index. Semantic vectors live in a LanceDB table (<data-dir>/semantic.lance) with the embedder identity in semantic.identity.json; a mismatch rebuilds the table and a search against a different embedder fails loudly rather than returning wrong neighbours. With a shared endpoint (minsync serve daemon or a remote minSync.embedder), mailcrawl embeds through it; a cold cache otherwise downloads ONNX weights on the first index. Results map to opaque /mailcrawl/<instance>/chunks/<chunk-id> sources.
mailcrawl doctor
mailcrawl status
autorag refresh --method datasourcesSearch behavior
- Retrieval is hybrid-first: one
mailcrawl search --mode hybridper query, with BM25 only when hybrid cannot run (for example no usable semantic index).account/mailboxnarrow results after the CLI returns. - The native CLI exposes
bm25,semantic, andhybridmodes:mailcrawl search --mode hybrid --limit 20 --json "<query>". scopenarrows results; a time window is applied over the over-fetch (publishedAtfrom the messagedate). See Search semantics.
Troubleshooting
| Symptom | Cause / fix |
|---|---|
datasource-embedding-egress-rejected | A non-loopback embedding URL was set; AutoRAG forwards only loopback endpoints and refuses a remote before spawning. |
mailcrawl semantic index unavailable (warning) | index failed but sync succeeded; BM25 stays available. |
text array must be non-empty | mailcrawl 0.1.3 or earlier; upgrade to 0.3.0+. |
First index slow or downloading models | Cold cache building the semantic index; raise connector.indexTimeoutMs. |
| Stale vectors after switching embedder | Identity mismatch rebuilds on the next index, or force it with mailcrawl repair --semantic. |
Diagnostics use the datasource codes in Search semantics; mailcrawl's stderr reaches them verbatim.
