> ## Documentation Index > Fetch the complete documentation index at: https://puzzlet-9ba7bb98.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Python dev server > Running the AgentMark development server with Python The AgentMark CLI automatically detects and runs Python projects with the appropriate dev server configuration. ## Starting the dev server ```bash theme={null} agentmark dev ``` The CLI detects Python projects and spawns the Python webhook server alongside the API server and UI. ## Project detection The CLI identifies Python projects by checking, in order: 1. **AgentMark setup files**: `agentmark_client.py`, `.agentmark/dev_server.py`, or a root `dev_server.py`. These are decisive: once you set up your project, an unrelated manifest file no longer overrides TypeScript/Python detection. (`agentmark.client.ts` is the equally decisive TypeScript marker.) 2. **Python manifests**: `pyproject.toml`, `requirements.txt`, or `setup.py`. These matter before any AgentMark files exist, so a fresh pip project gets Python guidance from `doctor` and `dev` on first contact. If none match, the CLI assumes TypeScript. ## Virtual environment detection The CLI automatically detects and uses virtual environments: ```text theme={null} Priority order: 1. .venv/bin/python (or .venv\Scripts\python.exe on Windows) 2. venv/bin/python (or venv\Scripts\python.exe on Windows) 3. System python ``` When the CLI finds a virtual environment, it prints: ```text theme={null} Using virtual environment: .venv/ ``` ## Entry point resolution The CLI resolves the dev server entry point in this order: | Location | Description | | -------------------------- | ----------------------------------------------------- | | `dev_server.py` | Custom dev server (project root) | | `.agentmark/dev_server.py` | Standard entry point (created by you or your AI tool) | The entry point serves a **webhook runner** so the CLI and the local dev UI can execute your prompts (once deployed, the same handler serves the Dashboard). You build the runner once from an executor (the function that calls your LLM SDK), and the runner's `dispatch` method handles every job it receives (`prompt-run`, `dataset-run`, and `get-evals`, which lists the evals registered on your client). ### Neutral client and runner The entry point builds a **webhook runner** from three pieces and serves it. Keep `agentmark_client.py` minimal (loader, client, and your evals) and put the executor, tracing init, and runner in `.agentmark/dev_server.py`: 1. **Client** (`agentmark_client.py`): `create_agentmark(loader=ApiLoader.local(base_url="http://localhost:9418"))`, plus your evals. 2. **Executor + runner** (`.agentmark/dev_server.py`): write a `create_executor` for your SDK, then `runner = create_webhook_runner(client, executor)`. The full executor contract and copy-paste handlers (object, streaming, every SDK) live on [Connect your SDK](/getting-started/client-setup#connect-your-sdk). 3. **Serve**: `serve_webhook_runner(runner)` (see [Entry-point file](#entry-point-file)). For the complete `agentmark_client.py` + `.agentmark/dev_server.py` files, see [Client setup](/getting-started/client-setup). **Local tracing differs from production.** The runner wires span hooks, but spans only export once you initialize tracing, and in local dev you point the exporter at the dev API server (unauthenticated, no cloud keys) so `agentmark doctor --smoke` and the local trace UI see your runs: ```python theme={null} import os from agentmark_sdk import AgentMarkSDK AgentMarkSDK( api_key="local-dev", app_id="local-dev", base_url=os.environ.get("AGENTMARK_DEV_SERVER", "http://localhost:9418"), ).init_tracing(disable_batch=True) ``` See [Tracing setup](/observe/tracing-setup) for the production configuration. ### Entry-point file The CLI boots `.agentmark/dev_server.py` (or `dev_server.py` at your project root) and passes `--webhook-port` / `--api-server-port`. It strips cloud credentials from the spawned [environment](#environment-variables), so a client that picks its loader by API-key presence (the [client setup](/getting-started/client-setup) pattern) lands on the local loader; the example above pins `ApiLoader.local` directly, which behaves the same in local dev. The entry point **must keep serving HTTP**: `serve_webhook_runner(runner)` is the line that does that. It serves `runner.dispatch`, the same handler a managed deployment exposes as `handler = runner.dispatch`. Without it the process builds the runner, exits, and `agentmark dev` reports `Webhook server stopped`. `serve_webhook_runner` is the Python counterpart of the TypeScript `createWebhookServer` (`@agentmark-ai/cli/runner-server`). `npm create agentmark@latest` scaffolds `agentmark.json`, the `agentmark/` prompt directory, MCP configs for your editor, and the AgentMark agent skill (plus `git init` in new directories). It doesn't create the client or dev-server entry: you (or the agent skill) write `agentmark_client.py` and `.agentmark/dev_server.py`. See [Client setup](/getting-started/client-setup) for the full dev-server and `handler.py` files. ## Environment variables The CLI sets the following environment variables for the spawned `dev_server.py`: | Variable | Value | Description | | ------------------------- | ----------------------------- | ------------------------------------------------------------------------------------ | | `AGENTMARK_DEV_SERVER` | `http://localhost:{api_port}` | Local API server URL; the example entry point reads it to point the tracing exporter | | `PYTHONPATH` | project root (prepended) | Lets `.agentmark/dev_server.py` import project-root modules like `agentmark_client` | | `PYTHONDONTWRITEBYTECODE` | `1` | Prevents `__pycache__` creation | | `PYTHONUNBUFFERED` | `1` | Ensures real-time output | | `PYTHONPYCACHEPREFIX` | per-run temp directory | Keeps stale bytecode from masking source edits between restarts | The CLI also removes `AGENTMARK_API_KEY`, `AGENTMARK_APP_ID`, and `AGENTMARK_BASE_URL` from the spawned process, so a client that picks its loader by API-key presence stays in local mode instead of loading deployed prompts from the Cloud. ## Server architecture When you run `agentmark dev`, three servers start: ```text theme={null} ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ API Server │────▶│ Webhook Server │────▶│ UI Server │ │ (port 9418) │ │ (port 9417) │ │ (port 3000) │ │ │ │ │ │ │ │ Telemetry API │ │ Python Process │ │ Next.js │ │ Trace Storage │ │ Prompt Executor │ │ Local dev UI │ └─────────────────┘ └─────────────────┘ └─────────────────┘ ``` The Python webhook server: * Receives prompt execution requests from the CLI * Uses your `agentmark_client.py` configuration * Executes prompts through your executor, which calls your LLM SDK * Returns streaming or non-streaming responses ## Port configuration Override default ports with CLI options: ```bash theme={null} agentmark dev --webhook-port 8080 --api-port 8081 --app-port 8082 ``` | Option | Default | Description | | ---------------- | ------- | ------------------- | | `--webhook-port` | 9417 | Webhook server port | | `--api-port` | 9418 | API server port | | `--app-port` | 3000 | UI server port | ## Wire contract You normally never implement this; `serve_webhook_runner(runner)` is the whole server. The contract below is for custom entry points (your own framework, extra routes) and for debugging with `curl`. Every job arrives as `POST /` with a JSON `{type, data}` body. Three event types exist: ### `prompt-run` Executes a single prompt: ```json theme={null} { "type": "prompt-run", "data": { "ast": { ... }, "options": { "shouldStream": true }, "customProps": { ... } } } ``` ### `dataset-run` Executes a prompt across a dataset: ```json theme={null} { "type": "dataset-run", "data": { "ast": { ... }, "experimentId": "exp-123", "datasetPath": "./datasets/test.jsonl" } } ``` ### get-evals Control-plane job: lists the eval names registered on your client, which populates the Dashboard's New Experiment dialog. Carries no AST; the response is a flat JSON body `{"type": "evals", "result": [...names], "traceId": null}`. ### Responses The CLI (`run-prompt`, `run-experiment`) and the Dashboard switch parsing on one response header: * **Streaming** (`runner.dispatch` returned a result with a `stream`): respond with `AgentMark-Streaming: true` and `Content-Type: application/x-ndjson`, then write the stream's NDJSON lines as they arrive (one JSON event per line: `text` / `object` / `dataset` / `error` chunks). After the stream drains, append a final `{"type": "done", "traceId": "..."}` line when the result carries a `traceId`; that's where the CLI reads the trace link from. * **Non-streaming**: respond `200` with the dispatch result as a plain JSON body. * **Errors**: respond with a non-2xx status and a `{"message": "..."}` JSON body (400 for malformed/unknown jobs, 500 for executor failures). This is the same contract the TypeScript dev server (`createWebhookServer`) and the managed-deployment server implement; `serve_webhook_runner` keeps the Python side pinned to it. ## Running prompts With the dev server running, execute prompts from another terminal: ```bash theme={null} agentmark run-prompt ./agentmark/.prompt.mdx ``` Or run experiments: ```bash theme={null} agentmark run-experiment ./agentmark/.prompt.mdx ``` Need a working starter? See [Example prompts](/build/example-prompts): four copy-paste recipes (object, text+tools, image, speech) you can drop into your `agentmark/` directory. ## Troubleshooting ### `agentmark doctor` reports missing dependencies `agentmark doctor` checks `.venv/`, then `venv/`, then the system `pip`, in that priority order, so it correctly finds packages installed inside a virtual environment. If you still see a missing-package warning after installing into your venv, confirm the venv pip resolves the packages: ```bash theme={null} .venv/bin/pip show agentmark-prompt-core agentmark-sdk ``` ### Virtual environment not found If you see "python not found" errors: ```bash theme={null} # Create a virtual environment python -m venv .venv # Activate it source .venv/bin/activate # macOS/Linux .venv\Scripts\activate # Windows # Install dependencies (plus your own LLM SDK, e.g. openai) pip install agentmark-prompt-core agentmark-sdk python-dotenv ``` ### Module not found Confirm you installed dependencies in the correct virtual environment: ```bash theme={null} pip install agentmark-prompt-core agentmark-sdk python-dotenv ``` ### Port already in use If ports are busy, specify alternative ports: ```bash theme={null} agentmark dev --webhook-port 9500 --api-port 9501 ``` ### `agentmark_client.py` not found The CLI requires `agentmark_client.py` in your project root: ```bash theme={null} # Create a new project npm create agentmark@latest # Or manually create agentmark_client.py ``` ## Agent frameworks An agent-framework executor works the same in the dev server. Your handler runs an agent loop instead of a single completion, but it's still a `create_executor` wired into the runner above. See [Connect your SDK](/getting-started/client-setup#agent-frameworks-pydantic-ai-mastra-claude-agent-sdk) for the agent-loop executor, including streaming with tool-call events. ## Next steps Python client, dev server, and deploy handler files Wire any LLM SDK with an executor Copy-paste executors for OpenAI, Anthropic, agent frameworks Execute prompts from CLI

Have questions?

Reach out any time: