> ## Documentation Index > Fetch the complete documentation index at: https://puzzlet-9ba7bb98.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # CLI reference > Complete reference for all AgentMark CLI commands The AgentMark CLI (`@agentmark-ai/cli`) provides tools for developing, testing, and building your AI prompts. ## Commands at a glance Jump to any command's full reference: * [`agentmark init`](#agentmark-init): set up AgentMark in a new or existing project * [`agentmark doctor`](#agentmark-doctor): check that you configured your project correctly * [`agentmark dev`](#agentmark-dev): start the local dev environment (API, webhook, UI) * [`agentmark login`](#agentmark-login): authenticate with AgentMark Cloud * [`agentmark logout`](#agentmark-logout): clear stored CLI credentials * [`agentmark link`](#agentmark-link): link your local project to a Cloud app * [`agentmark run-prompt`](#agentmark-run-prompt): run a single prompt with test props * [`agentmark run-experiment`](#agentmark-run-experiment): run an experiment against its dataset * [`agentmark generate-types`](#agentmark-generate-types): generate type definitions from prompt schemas * [`agentmark generate-schema`](#agentmark-generate-schema): generate a JSON Schema for prompt frontmatter * [`agentmark build`](#agentmark-build): compile prompts to JSON for static loading * [`agentmark pull-models`](#agentmark-pull-models): pull and configure models from a provider ## Installation ```bash theme={null} # Install globally so the bare `agentmark` command is on your PATH npm install -g @agentmark-ai/cli agentmark # Or run without installing, via npx npx @agentmark-ai/cli ``` In a project scaffolded by [`agentmark init`](/getting-started/quickstart), `@agentmark-ai/cli` is also pinned as a local dev dependency, so `npm run dev` and CI resolve that pinned version from `node_modules/.bin` regardless of what's installed globally. ## Environment variables The CLI automatically loads environment variables from a `.env` file in the current working directory. This happens before any command execution, so you can store API keys and configuration there. ```bash theme={null} # .env OPENAI_API_KEY=sk-... AGENTMARK_API_KEY=... AGENTMARK_APP_ID=... ``` See [Environment variables](/configure/environment-variables) for the full list. ## Update notifications The CLI checks for updates asynchronously when you run commands. If a newer version is available, you'll see a notification after your command completes. This check is non-blocking. To disable update checks, set the environment variable: ```bash theme={null} export AGENTMARK_NO_UPDATE_NOTIFIER=1 ``` *** ## Commands ### `agentmark init` Set up AgentMark in a new or existing project. Writes `agentmark.json`, creates the `agentmark/` prompts directory, pins `@agentmark-ai/cli` as a local dev dependency (plus npm scripts), wires IDE MCP configs for the clients you pick, and installs the [agent skill](/coding-agents/agent-skill). Run it once per project: from inside an existing repo (the common case) or with a folder name to scaffold a fresh one. ```bash theme={null} agentmark init [folder] [options] ``` **Options:** | Option | Description | Default | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `[folder]` / `--path ` | Target directory | `.` inside an existing project, else `my-agentmark-app` | | `--client ` | IDE clients to wire MCP configs for, comma-separated: `claude-code`, `codex`, `cursor`, `vscode`, `zed` (or `all`) | prompt (pre-selects detected editors) | | `-y, --yes` | Non-interactive: accept the default for every prompt (folder default, all IDE clients, keep an existing `agentmark.json`). For CI and coding agents | interactive | | `--overwrite` | Replace an existing `agentmark.json` with the default config | keep existing | | `--api-url ` | Override the AgentMark gateway URL for the cloud MCP entry (internal staging / self-host) | `https://api.agentmark.co` | The local pin is non-destructive: an existing `dev` / `build` script is never overwritten. AgentMark's scripts land under namespaced keys (`agentmark:dev`, …) instead. `npm create agentmark` is a thin wrapper that runs this exact flow. ### `agentmark doctor` Check that you set up your AgentMark project correctly. The default pass is a static health check (no network calls, no servers spawned) that inspects your config, prompts, client, and dependencies and reports each finding with a fix. The optional `--smoke` tier adds a live end-to-end run. Run it right after scaffolding, or when a command fails with a config or dependency error. ```bash theme={null} agentmark doctor [options] ``` **Options:** | Option | Description | Default | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | | `--json` | Emit the report as JSON instead of human-readable text | human text | | `--strict` | Exit non-zero on warnings too (useful in CI) | off | | `--smoke` | Also run a live tier: execute a prompt against a running `agentmark dev` and verify the emitted trace's shape (model, tokens, input, output) | static only | | `--boot` | With `--smoke`, start `agentmark dev` automatically and tear it down afterward (one command, for CI / agents) | assumes dev is running | | `--prompt ` | Prompt to run for `--smoke` (defaults to the first prompt found) | first prompt | | `--props ` | Props as a JSON string for the `--smoke` prompt. Requires `--prompt` (for example `'{"name":"Alice"}'`) | none | | `--webhook-port ` | Webhook port `--smoke` targets (and `--boot` starts dev on) | `9417` | | `--api-port ` | API-server port `--smoke` reads traces from (and `--boot` starts dev on) | `9418` | **What it checks:** * **Project**: `agentmark.json` is present and valid (including field/schema shape: required keys present, no unknown-key typos); `agentmarkPath` resolves (catches the `"/"` mistake); the client file (`agentmark.client.ts` / `agentmark_client.py`), the dev-server entry, and the managed-deploy handler (`handler.ts` / `handler.py`) exist; `AGENTMARK_API_KEY` / `AGENTMARK_APP_ID` are present; `.env` is gitignored; Node.js is 20.6+. * **Config & prompts**: every `.prompt.mdx` parses and declares a `model_name`; you declare prompt models in `builtInModels` (doctor enforces a non-empty list as an allowlist); the catalog recognizes the models. * **Dependencies**: doctor checks that you installed the runtime packages (resolvable from the project), not just listed them in `package.json`. The client and dev-entry import `@agentmark-ai/prompt-core`, so doctor fails the check when `package.json` declares prompt-core but `node_modules` lacks it, and points you to `npm install`; a missing install is the most common reason `agentmark dev` or `doctor --smoke --boot` exits 1. `@agentmark-ai/sdk` carries tracing and the cloud-execution runner; doctor warns (rather than fails) when `node_modules` lacks it or `package.json` omits it. `agentmark init` pins `@agentmark-ai/cli` locally so CI and teammates run the same version; a missing pin is an advisory warning. For Python projects, doctor checks that `agentmark-prompt-core` and `agentmark-sdk` import, and skips with a reminder when it can't verify them. There is no SDK-specific adapter to require: you bring your own SDK through `@agentmark-ai/prompt-core`'s neutral render or an executor. * **Live run (`--smoke`)**: runs one prompt end-to-end through `agentmark dev` and confirms real output + token usage came back, then fetches the emitted trace and validates its shape, exercising your SDK, provider keys, and tracing indirectly, with no provider-specific knowledge. **Exit code:** `0` when nothing failed (warnings don't fail the run), `1` on any failure, or on any non-advisory warning with `--strict`. Advisory warnings (like a missing local `@agentmark-ai/cli` pin) are informational and never fail the run, so adding `agentmark init` to an existing project won't turn a green `--strict` CI gate red. So it drops straight into CI. **Example:** ```bash theme={null} # Human-readable report agentmark doctor # Machine-readable, for scripts agentmark doctor --json # CI: treat warnings as failures too agentmark doctor --strict # Live end-to-end check: boot dev, run a prompt, verify the trace, tear down agentmark doctor --smoke --boot # Smoke a specific prompt that requires variables agentmark doctor --smoke --prompt agentmark/greeting.prompt.mdx --props '{"name":"Alice"}' ``` For the failures it surfaces, see [Troubleshooting](/reference/troubleshooting). *** ### `agentmark dev` Start the local development environment: the API server, the webhook server, and the local dev UI app. Once you link the project (`agentmark link`), traces from local runs automatically forward to AgentMark Cloud. ```bash theme={null} agentmark dev [options] ``` **Options:** | Option | Description | Default | | ------------------------- | ------------------------------------------------------------------------------------------------------------- | -------- | | `--api-port ` | API server port | `9418` | | `--webhook-port ` | Webhook server port | `9417` | | `--app-port ` | Local dev UI port | `3000` | | `--no-forward` | Disable trace forwarding to AgentMark Cloud (forwarding is on by default once you link the project) | forward | | `--no-ui` | Skip the UI app (API + webhook only) for CI / headless / test use | UI on | | `--no-watch` | Don't restart on file changes; exit on a dev-entry crash so the error surfaces (for CI / headless / boot use) | watch on | For programmatic Cloud access, run the [`agentmark-mcp`](/coding-agents/gateway-mcp) MCP server or call the gateway REST API directly. **Project detection:** * **TypeScript projects**: `agentmark.client.ts` in the project root * **Python projects**: `pyproject.toml`, `agentmark_client.py`, or `.agentmark/dev_server.py` **Dev server entry points (TypeScript):** The CLI looks for dev server files in this order: 1. `dev-server.ts` (custom override, project root) 2. `dev-entry.ts` (default location, project root) **Python virtual environment:** For Python projects, the CLI auto-detects `.venv/` or `venv/` directories. **Example:** ```bash theme={null} # Default — API + webhook + UI on 9418/9417/3000 agentmark dev # Custom ports agentmark dev --api-port 9500 --webhook-port 9501 # CI / headless — no UI app agentmark dev --no-ui # Linked project, but don't forward traces to Cloud agentmark dev --no-forward ``` *** ### `agentmark login` Authenticate with AgentMark Cloud via browser OAuth. The CLI opens your default browser to complete the login flow, then stores credentials locally for subsequent commands. ```bash theme={null} agentmark login [options] ``` **Options:** | Option | Description | Default | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `--base-url ` | AgentMark Cloud URL | `$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co` | | `--print-url` | Print the auth URL instead of opening a browser (for SSH'd shells, CI runners, or IDE-embedded agents) | open browser | | `--json` | Emit a single line of JSON on completion instead of human text; useful for wrapper scripts that need to capture `user_id` / `email` programmatically | human text | | `--timeout ` | How long to wait for the browser handoff before failing | `120` (2 minutes) | `agentmark link`, trace forwarding in `agentmark dev`, and the [`agentmark-mcp`](/coding-agents/gateway-mcp) MCP server all use the stored credentials automatically. For the MCP server, the `AGENTMARK_API_KEY` environment variable takes precedence over the cached session bearer; `agentmark link` itself always requires a login session. *** ### `agentmark logout` Clear stored CLI authentication credentials. ```bash theme={null} agentmark logout [options] ``` **Options:** | Option | Description | Default | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `--base-url ` | AgentMark Cloud URL | `$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co` | | `--json` | Emit a single line of JSON on completion instead of human text. Shape: `{"logged_out": true, "was_logged_in": , "revoked_dev_key": }` | human text | *** ### `agentmark link` Link your local project to an app in AgentMark Cloud. The CLI prompts you to select an app from your account (or use `--app-id` to skip the prompt), then stores the app ID and app/org metadata in `.agentmark/dev-config.json`. The CLI creates no API key: trace forwarding authenticates with your login session. ```bash theme={null} agentmark link [options] ``` **Options:** | Option | Description | Default | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `--app-id ` | App ID to link (skips interactive selection) | None | | `--base-url ` | AgentMark Cloud URL | `$AGENTMARK_PLATFORM_URL` or `https://app.agentmark.co` | | `--json` | Emit a single line of JSON on completion (for example, for CI to capture the linked appId). Shape: `{"linked": true, "appId": "...", "appName": "...", "tenantId": "...", "orgName": "...", "baseUrl": "..."}` | human text | After linking, `agentmark dev` automatically forwards traces from local prompt runs to the linked app, with no flag needed. `agentmark dev` reads the linked `appId` from `.agentmark/dev-config.json` (per-developer, gitignored). The forwarder authenticates with the session bearer from `~/.agentmark/auth.json` (auto-refreshed). *** ### `agentmark run-prompt` Run a single prompt file with test props. ```bash theme={null} agentmark run-prompt [options] ``` **Arguments:** | Argument | Description | | ---------- | ------------------------------ | | `filepath` | Path to the `.prompt.mdx` file | **Options:** | Option | Description | Default | | --------------------- | ------------------------------------------ | ----------------------- | | `--server ` | Webhook server URL | `http://localhost:9417` | | `--props ` | Props as JSON string | - | | `--props-file ` | Path to JSON or YAML file containing props | - | **Example:** ```bash theme={null} # Run with inline props agentmark run-prompt ./agentmark/greeting.prompt.mdx --props '{"name": "Alice"}' # Run with props from file agentmark run-prompt ./agentmark/greeting.prompt.mdx --props-file ./test-props.yaml # Run against a remote server agentmark run-prompt ./agentmark/greeting.prompt.mdx --server https://my-webhook.example.com ``` *** ### `agentmark run-experiment` Run an experiment against its dataset, with evaluations by default. ```bash theme={null} agentmark run-experiment [options] ``` **Arguments:** | Argument | Description | | ---------- | ------------------------------------------------------ | | `filepath` | Path to the `.prompt.mdx` file with test configuration | **Options:** | Option | Description | Default | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | `--server ` | Webhook server URL | `http://localhost:9417` | | `--skip-eval` | Skip running evals even if they exist | `false` | | `--format ` | Output format: `table`, `csv`, `json`, `jsonl`, `junit` | `table` | | `--threshold ` | Fail if pass rate is below threshold (0-100) | - | | `--sample ` | Sample N% of dataset rows randomly (1-100) | - | | `--rows ` | Select specific rows by index/range (for example `0,3-5,9`) | - | | `--split ` | Train/test split (for example `train:80`, `test:80`) | - | | `--seed ` | Seed for reproducible sampling/splitting | - | | `--truncate ` | Truncate table cell content to N chars (0 = no limit) | `1000` | | `--concurrency ` | Dataset rows to run in parallel | `20` | | `--baseline-commit ` | Git ref (or tree hash) of a prior run to compare against; enables the regression gate via `test_settings.regression_tolerance` | - | **Example:** ```bash theme={null} # Run experiment with table output agentmark run-experiment ./agentmark/qa-bot.prompt.mdx # Run experiment with JSON output, skip evals agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format json --skip-eval # Run with CI threshold (fails if <80% pass rate) agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --threshold 80 # Emit JUnit XML for CI gating (GitHub Actions, GitLab, Jenkins, etc.) agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --format junit > results.xml # Gate against a baseline run — fails rows whose scorer regresses beyond # test_settings.regression_tolerance relative to the baseline commit agentmark run-experiment ./agentmark/qa-bot.prompt.mdx --baseline-commit main ``` *** ### `agentmark generate-types` Generate TypeScript type definitions from your prompt schemas. ```bash theme={null} agentmark generate-types [options] ``` **Options:** | Option | Description | Default | | --------------------------- | ------------------------------------------ | ------------ | | `-l, --language ` | Target language (`typescript` or `python`) | `typescript` | | `--local ` | Local server port to fetch prompts from | - | | `--root-dir ` | Root directory containing agentmark files | - | **Output:** The command outputs TypeScript definitions to stdout. Redirect to a file: ```bash theme={null} agentmark generate-types --root-dir ./agentmark > agentmark.types.ts ``` **Generated types include:** * Input types based on `input_schema` * Output types based on the model's `schema` * A mapping of prompt paths to their respective types * Tool argument types **Example:** ```bash theme={null} # Generate from local files agentmark generate-types --root-dir ./agentmark > agentmark.types.ts # Generate from local dev server agentmark generate-types --local 9418 > agentmark.types.ts ``` See [Type safety](/configure/type-safety) for usage examples. *** ### `agentmark generate-schema` Generate a JSON Schema file for `.prompt.mdx` frontmatter. This enables IDE validation (squiggles) for fields like `model_name` in your prompt files. ```bash theme={null} agentmark generate-schema [options] ``` **Options:** | Option | Description | Default | | ----------------------- | ---------------- | ------------ | | `-o, --out ` | Output directory | `.agentmark` | **Example:** ```bash theme={null} agentmark generate-schema agentmark generate-schema --out ./schemas ``` *** ### `agentmark build` Build prompts into pre-compiled JSON files for static loading with `FileLoader`. ```bash theme={null} agentmark build [options] ``` **Options:** | Option | Description | Default | | ----------------------- | ---------------- | ---------------- | | `-o, --out ` | Output directory | `dist/agentmark` | **Requirements:** * An `agentmark.json` config file must exist in the current directory * `agentmark build` reads prompts from the directory that `agentmarkPath` names in the config **Output structure:** ```text theme={null} dist/agentmark/ manifest.json # Build manifest with all prompts greeting.prompt.json # Compiled prompt (mirrors source structure) nested/ helper.prompt.json ``` **Example:** ```bash theme={null} # Build with default output directory agentmark build # Build to custom directory agentmark build --out ./build/prompts ``` See [Loaders](/configure/loaders) for using built prompts with `FileLoader`. *** ### `agentmark pull-models` Pull and configure models from a provider. Runs interactively by default; pass `--provider` + `--models` to skip the prompts (for example, for CI or agents). ```bash theme={null} agentmark pull-models [options] ``` **Options:** | Option | Description | Default | | ------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------- | | `--provider ` | Provider key (skips the interactive picker) | prompt | | `--models ` | Comma-separated model IDs to add (skips the interactive multi-select) | prompt | | `--list` | Print available providers (or models when combined with `--provider`) as JSON and exit; doesn't modify `agentmark.json` | off | With both `--provider` and `--models` set, the command runs fully non-interactively and is safe for CI. This command opens an interactive prompt (when you pass no flags) to: 1. Select a model provider 2. Choose models to enable 3. Update your local configuration **Agent / CI discovery workflow** Use `--list` to discover valid providers and model IDs before modifying config (no `agentmark.json` required): ```bash theme={null} # List all available providers agentmark pull-models --list # List models for a specific provider agentmark pull-models --list --provider anthropic # Add models non-interactively agentmark pull-models --provider anthropic --models anthropic/claude-opus-4-8 ``` `--list` output is stable JSON, safe to pipe into `jq` or capture in scripts. *** ## Programmatic gateway access (for agents and scripts) Programmatic gateway access is available through two protocol-level surfaces that stay in lock-step with the gateway's OpenAPI spec: * **IDE agents (Claude Code, Cursor, VS Code, Zed):** run the [`agentmark-mcp`](/coding-agents/gateway-mcp) MCP server. It fetches the gateway's OpenAPI spec at startup and exposes one MCP tool per operation (for example `create_app`, `list_traces`, `start_app_git_connect`). The agent calls those tools directly; no CLI invocation needed. * **CI / shell scripts:** call the gateway REST API with `curl` and an `AGENTMARK_API_KEY`. AgentMark generates the MCP tools from the same OpenAPI spec, so request shapes are identical; only the transport differs. Both targets honor the same auth chain: `AGENTMARK_API_KEY` env var first, then the session bearer from `~/.agentmark/auth.json` (written by `agentmark login`). *** ## Configuration files ### `agentmark.json` Project configuration file in your project root. See [Project config](/configure/project-config) for the full schema. ```json theme={null} { "agentmarkPath": ".", "version": "2.0.0", "mdxVersion": "1.0" } ``` | Field | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------ | | `agentmarkPath` | Base path for agentmark files (contains the `agentmark/` directory); use `"."` for the canonical layout, not `"/"` | | `version` | Configuration version | | `mdxVersion` | MDX syntax version | ### `.agentmark/dev-config.json` Auto-generated local development configuration (gitignored): ```json theme={null} { "createdAt": "2026-04-15T10:30:00.000Z", "appPort": 3000, "forwarding": { "appId": "1c5f9e2a-7b43-4a1e-9c0d-3f8a2b6e4d71", "appName": "my-app", "orgName": "my-org", "tenantId": "9d2b4c6e-1f3a-4e5d-8a7b-0c9e2f4a6b8d", "baseUrl": "https://api.agentmark.co" } } ``` This file stores: * `appPort`: local dev server UI port (updated when dev server starts). * `forwarding`: linked app metadata (app ID, app and org names, gateway base URL) used by `agentmark dev` when forwarding traces to AgentMark Cloud. Populated by `agentmark link` and cleared by `agentmark logout`. No credentials live here: the forwarder authenticates with your login session. The configuration expires 30 days after creation: the next CLI command that loads it regenerates a fresh file, and running `agentmark link` again restores the app binding.

Have questions?

Reach out any time: