Adding a datasource
This page is the procedure for adding a built-in datasource skill to AutoRAG 2.0. Follow it in order; each step names the file it touches and the check that proves it.
Prerequisites
- Read Architecture and the datasource contract in
src/datasource/types.ts. - Choose the family your backend belongs to:
- Connector-backed (
ConnectorDatasourceSkill,src/datasource/connector-skill.ts): AutoRAG fetches documents and owns the chunk store — seegithubandrss. - Crawler-backed (
CrawlerDatasourceSkill,src/datasource/crawler-skill.ts): an external crawler CLI owns the archive — seenotion. - A custom
DatasourceSkillwhen neither base fits — seewombat,hister,clawgallery, andobsidian.
- Connector-backed (
- Keep the skill constructible only from trusted config. Model and tool arguments must never reach it, and a query
scopemay only narrow, never widen.
1. Create the skill directory
Put the implementation under src/datasource/skills/<name>/, with index.ts re-exporting the skill, its options, and any helpers. Split modules the way the family you chose does:
| File | Role |
|---|---|
skill.ts | The DatasourceSkill (or a subclass of a base skill). |
connector.ts | A DatasourceConnector for a connector-backed skill. |
client.ts | A process/HTTP client for a CLI- or server-backed skill. |
methods.ts | RetrievalMethod implementations when a base skill does not provide them. |
index.ts | The package surface: skill, options types, and any test seams. |
Keep one module focused: the connector/client must never throw, and returns an ok: false union with a coarse reason instead.
2. Implement the contract
DatasourceSkill (in src/datasource/types.ts) requires describe(), polling(), index(), retrievalMethods(), describeSources(), and skillManifest(). Hold to these rules:
index()never throws; report failures asDatasourceDiagnosticentries whosecodeis one ofDatasourceDiagnosticCode(for exampledatasource-auth-error,datasource-unavailable,datasource-empty).- Descriptor
capabilitiesandtagsare descriptive metadata. A method that accepts ascopemust advertisescoped; a method that cannot answer must throwdatasourceCliError(...)so the pipeline reports the surface as unsearched instead of returning an empty set. - Sources are opaque, slash-hierarchical paths (
/<name>/<instance>/chunks/<id>). Never emit an OS path for the agent tocat, and never emit a#fragment. describeSources()explains the data without leaking storage paths; theskillManifest()body stays path- and PII-opaque.
3. Register in the factory
src/datasource/skills/factory.ts materializes skills from the trusted datasources config section:
- Add the import for your skill.
- Add a
BUILDERSentry keyed by the template name; the key becomes the configtypeand the default connection id. - If the backend exists on one platform only, add it to
PLATFORM_ONLY_SKILLSso it is neither offered nor built elsewhere. - If you retire a template, add it to
RETIRED_TEMPLATESso a config entry using it is collected asretiredand skipped, never thrown.
builtinDatasourceSkillNames() then includes your name automatically, and buildDatasourceSkills reports unknown names without failing agent construction.
4. Wire the setup probe
src/cli/setup.ts decides what autorag setup reports:
BUILTIN_BINARIESmaps a CLI-backed template to its binary; a missing binary reportsskipped: missing <binary> binary.DEFAULT_CREDENTIALSlists the environment variables a non-CLI connector requires (for examplegithub→GITHUB_TOKEN); a CLI-backed datasource is never required to hold one.datasourceNames()is the set of built-ins probed even when not configured; add names that should always be offered.- A datasource with its own health story models it on
setupHisterand surfaces it onSetupReport.
Verify the wiring:
autorag setup --format jsonThe datasource must appear with configured, skipped, or blocked and an actionable reason.
5. Tests
- Add
test/datasource/skills/<name>.test.ts. Cover argv and environment isolation, output parsing, a missing binary, malformed output, indexing, retrieval, auth and rate-limit diagnostics, and#-fragment rejection. - A scope-capable datasource also needs multi-scope and user-scope narrowing tests.
- Run the focused files with vitest, for example
bunx vitest run test/datasource/skills/<name>.test.ts. test/docs/site-contract.test.tsfails when a built-in has nodocs/datasources/<name>.md, so land step 6 in the same change.
6. Documentation
- Write
docs/datasources/<name>.md. The site contract test requires this exact path for every built-in datasource. - Add exactly one entry to
docs/.vitepress/sidebar.mtsunder Datasources → Knowledge and code, withtextequal to the page'stitleandlink/datasources/<name>. - Follow the page format the docs tests enforce: YAML
titleanddescription, one# H1equal to the title, relative.mdlinks to other pages, absolute GitHub URLs for code, and a language on every fenced code block. See Page format and Contract tests.
7. Manual QA
- Add a harness under
scripts/manual-qa/run-qa-<name>-live.tswhen a local store exists, and record it in Manual QA and live E2E. - Run it inside the repository's isolated QA shell (
make qa-shell); never mount a real host home to make a native lane pass.
8. Issue template and labels
- Track the work with
.github/ISSUE_TEMPLATE/datasource-skill.md. - Labels:
datasource-skillandintegration, plus the데이터 소스 추가label; addmanual-qa-requiredwhen real credentials or local-app permissions are needed.
Acceptance criteria
- Every configured connection is searchable with no authorization step, and tool arguments cannot change trusted connector configuration.
- A query
scopenarrows and never widens; results are narrowed before merge and use slash-hierarchical sources with no#fragment. - Expected auth and API failures return non-throwing diagnostics; a retrieval method that cannot answer surfaces the CLI's own words.
- The datasource appears in
autorag setup, inautorag status --json, and asdatasource-<name>in the agent's skill list. - The docs page, sidebar entry, tests, and manual QA land in the same change.
