Working with datasets using AI assistants
Working with datasets using AI assistants
Datapressr is a skills repo — instead of a CLI, it provides AI assistant instructions that work across tools (Claude Code, Codex CLI, Gemini CLI, etc.).
Quick start
Make a folder to hold your datasets, install the skills into it and start your assistant there:
mkdir my-datasets && cd my-datasets
npx skills add datasets/datapressr
claude
The installer asks which skills and which agent to install for. To skip the prompts, pass them as flags: npx skills add datasets/datapressr --skill '*' -a claude-code -y installs all eight skills for Claude Code.
Then inside the session:
/init world-gdp
Your AI assistant creates my-datasets/world-gdp/ with datapackage.json, data/ and the validator, and explains next steps. Run later commands from my-datasets/ too, and each new dataset gets its own folder alongside world-gdp/.
Scope. By default the installer puts the skills in the project you ran it from (./.claude/skills/ for Claude Code), so they are available when the assistant is started in that folder. To install them once for every project, add -g (npx skills add datasets/datapressr -g).
(Claude Code in this repo also picks the skills up directly via .claude/skills/ symlinks, so no npx skills step is needed when working inside datapressr itself.)
AGENTS.md
Every dataset directory should contain AGENTS.md — a knowledge file that gives your AI assistant immediate context about dataset structure, conventions, and the publish workflow. The /init command copies it automatically.
AGENTS.md is the standard cross-tool instruction file:
| Tool | Reads |
|---|---|
| Claude Code | AGENTS.md + skills/ (via npx skills add, or .claude/skills/ symlinks inside this repo) |
| Codex CLI | AGENTS.md + skills/ (via npx skills add; not yet tested with DataPressr) |
| Gemini CLI | AGENTS.md + skills/ (via npx skills add; not yet tested with DataPressr) |
Skills
The skills/ directory holds one SKILL.md playbook per step, in the flat layout
npx skills expects — so they install
into any agent, not just Claude Code. See skills/README.md. In Claude Code the
judgement-heavy playbooks (capture, archive, structure) and the mechanical
last-mile steps below are all invocable as /<name> slash commands:
/init <name>
Scaffold a new dataset directory.
/init world-gdp
Creates:
world-gdp/
datapackage.json # dataset metadata and resource list
data/ # data files go here
.datahubignore # gitignore-style exclusions for dh publish
AGENTS.md # AI assistant context
scripts/validate-datapackage.mjs # the deterministic check /validate runs
/validate
Check datapackage.json for common issues before publishing.
Runs scripts/validate-datapackage.mjs (copied into the dataset by /init, zero dependencies, plain Node — no package.json needed to run it) and reports its output: errors (must fix), warnings (worth fixing) and notes (reminders that don't count as warnings). It checks the metadata and, by default, every CSV value against the resource's declared schema: types, real calendar dates, row width, primary key uniqueness, foreign keys and the house CSV format. --metadata-only skips the value checks; --json prints machine-readable output.
- Errors: missing file, invalid JSON, unsafe name, empty resources (allowed while
statusiscaptureorstub), a resource path that doesn't exist - Warnings: missing title/description, missing or unknown status, unlisted files in
data/, large files, missinglicenses/sourcespaststub, resources with no typedschemaor noprimaryKey - Notes: a blank
""title/description (the/initplaceholder) on acaptureorstub
The script itself has a test suite in the datapressr repo (npm test, using Node's built-in test runner against fixture datapackages in scripts/fixtures/) — it's the one piece of this project's tooling that's actually tested, rather than being an LLM re-deriving a checklist from prose each run.
/push
Publish the current dataset to DataHub. Publishing is optional: if dh isn't installed or you haven't signed in, the skill says so and stops, and your dataset lives on in Git.
The skill keeps the name push, but the command it runs is dh publish (the DataHub CLI renamed its push command to publish, and the old name no longer works):
dh publish . --publication datapressr
Install dh. It is a Go binary released from datopian/datahub-next. That repository is private, so you need GitHub access to it. Download the archive for your platform, check it and put dh on your PATH:
gh release download v0.1.0 -R datopian/datahub-next -p 'dh_darwin_arm64.tar.gz' -p checksums.txt
shasum -a 256 -c checksums.txt --ignore-missing # Linux: sha256sum -c --ignore-missing checksums.txt
tar -xzf dh_darwin_arm64.tar.gz && mv dh ~/bin/
Sign in. Run dh login once. It opens a browser, you sign in to datahub.io and authorize the CLI, and it saves a token locally. For CI, copy that token into a DATAHUB_API_TOKEN secret and set DATAHUB_API_URL=https://datahub.io. Treat the token as highly privileged.
Before publishing, the skill checks:
- A root
README.md. DataHub builds the dataset page from it; without one the page is a 404. Don't start it with a# Title, because DataHub shows the title already. /validatepasses.- Views DataHub can draw. Simple
linecharts take several series. Simplebarandlines-and-pointscharts need the x (group) field andseries[0]typedyear,yearmonth,dateornumber, notintegerorstring, and they plot onlyseries[0]. For anything else use avega-liteorplotview. .datahubignoreexcludesarchive/, scripts,node_modules/,package*.jsonandAGENTS.md. Every other non-hidden file is uploaded, and every other.mdfile becomes a sub-page.
Which publication. The skill always passes --publication explicitly: $DATAHUB_PUBLICATION if you set it, otherwise datapressr (https://datahub.io/datapressr). It never publishes to core (it refuses if $DATAHUB_PUBLICATION is core), whose datasets sync from GitHub, so the next sync can overwrite a direct upload.
After publishing, open the printed URL (https://datahub.io/<publication>/<name>) and check the page; it is processed in the background after the upload. Publishing again adds and overwrites files but never deletes them, so a file you removed locally stays online, and an existing dataset's title and description only change in the DataHub dashboard.
Publishing a story. Inside the DataPressr repo, /push also publishes a data story from site/stories/ as its own DataHub page (/push story oil-prices). It never uploads the story file in place. It builds a bundle with node scripts/bundle-story.mjs <story> --publication <pub>: the story as README.md without its h1 and internal "Friction notes", links made absolute, plus only the charts it uses. Then it runs the dh publish <bundle> --publication <pub> --name <slug> --title <title> --description <description> command the bundler prints. The slug and approval live in the story's frontmatter:
datahub:
slug: wti-went-negative
status: draft # approved, with an "approved: <date> <basis>" line, after the voice pass or an owner OK
The bundler refuses core, refuses publications with no recorded grant (only datapressr today; the DataHub blog comes later), refuses a draft unless the publication is datapressr and --allow-draft is passed with the owner's OK, and refuses a slug that is a dataset name, since stories and datasets share one namespace per publication. After publishing, check the page in a browser: title shown once, description present, every chart loads, no "Error parsing MDX", links work. Three DataHub limits apply: the title and description are fixed at first publish, the date shown is the first publish date rather than the frontmatter date, and deleted or renamed files stay online.