> ## Documentation Index
> Fetch the complete documentation index at: https://puzzlet-9ba7bb98.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CI/CD

> Run AgentMark evals in your pipeline and gate pull/merge requests on the results.

Run AgentMark evals in CI and gate pull requests and merge requests on the results. The CLI's `--format junit` output is JUnit XML, the same format `pytest`, `jest`, and `vitest` emit, so every major CI system parses it natively: GitHub Actions (via marketplace parsers), GitLab CI (via `artifacts:reports:junit:`), Jenkins, and CircleCI. Failures show up alongside your other tests, with no third-party reporter to install.

## The gates

Three independent gates can fire in CI. They answer different questions and can all run at once:

* **Validation**: *does every prompt still compile?* Runs `agentmark build`. See [Status checks](/deploy/status-checks), which covers both the Cloud-managed check and the self-hosted CI job.
* **Absolute pass rate** (`--threshold <percent>`): *is this run good enough on its own?* Fails when the share of passing rows falls below a fixed floor. Needs no baseline. See [Running experiments](/evaluate/running-experiments#command-options).
* **Regression** (`--baseline-commit <ref>`): *did this change make anything worse than before?* Fails when a case scores below its own baseline, or a scorer's mean drops below a floor. See [Regression gates](/deploy/regression-gates) for the full mechanics.

This page covers wiring the eval run itself into your pipeline. The two eval gates (`--threshold` and `--baseline-commit`) are flags on the same `run-experiment` command.

## Run evals in CI (raw CLI)

`run-experiment` sends each prompt and dataset to a running AgentMark dev server, so a CI job boots one headless and waits for it before running the experiment. This is the platform-agnostic pattern: install dependencies, boot `agentmark dev --no-ui --no-forward`, wait for port 9417, run the experiment, and point your CI at the JUnit output. The GitLab job below is the fully worked example; the same boot-run-report shape ports to GitHub Actions, Jenkins, and CircleCI.

```yaml .gitlab-ci.yml theme={null}
agentmark_eval:
  image: node:20-bookworm-slim
  variables:
    GIT_DEPTH: "0"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    # Project dependencies: the dev server runs your project's client code
    - npm ci
    # Boot the dev server headless (webhook server on 9417, API server on 9418).
    # `npx` resolves the @agentmark-ai/cli pinned by `npm ci` (node_modules/.bin
    # isn't on PATH in a raw CI shell, so the bare `agentmark` command isn't).
    - npx @agentmark-ai/cli dev --no-ui --no-forward &
    # Wait until the webhook server accepts connections
    - timeout 60 bash -c 'until (echo > /dev/tcp/127.0.0.1/9417) 2>/dev/null; do sleep 1; done'
    # Run the experiment and emit JUnit XML
    - npx @agentmark-ai/cli run-experiment agentmark/qa.prompt.mdx --format junit > results.xml
  artifacts:
    when: always
    reports:
      junit: results.xml
```

The dev server is what executes your prompts, so it needs your model provider keys (for example `OPENAI_API_KEY`). Add them as masked CI/CD variables in **Settings → CI/CD → Variables**; the job environment passes through to the dev server. Without a running server, `run-experiment` exits with `❌ Could not connect to AgentMark server`.

List each prompt you want to gate as its own `run-experiment` line writing to its own XML file. Add `--threshold <percent>` for a pass-rate gate, or `--baseline-commit "$CI_MERGE_REQUEST_DIFF_BASE_SHA"` for the regression gate. The baseline lookup resolves from AgentMark Cloud when you set an `AGENTMARK_API_KEY` variable (see [Set up the API key](#set-up-the-api-key)), and `GIT_DEPTH: "0"` keeps the diff base resolvable. See [Regression gates](/deploy/regression-gates) for the gate mechanics.

<Note>
  `run-experiment` always executes through a webhook server (the boot-and-wait step above), on every platform; `AGENTMARK_API_KEY` / `AGENTMARK_APP_ID` don't change that. They point the **regression baseline** lookup at AgentMark Cloud (durable across CI runs) instead of the ephemeral local store. The [regression gate setup](/deploy/regression-gates#set-it-up-for-prompts-cli) has the complete GitHub Actions `.github/workflows/evals.yml`. To gate agents or workflows from inside your own test suite (no CLI and no dev server, since your `task` function is the execution), use the [SDK setup](/deploy/regression-gates#set-it-up-for-agents-and-workflows-sdk).
</Note>

## Set up the API key

Add `AGENTMARK_API_KEY` as a **masked**, **protected** CI/CD variable in your project's settings (**Settings → CI/CD → Variables** on GitLab, **Settings → Secrets and variables → Actions** on GitHub):

<Steps>
  <Step title="Get the key from AgentMark Cloud">
    In the [AgentMark Dashboard](https://app.agentmark.co), open **Settings → API Keys** and create a key scoped to the app whose prompts you're gating.
  </Step>

  <Step title="Store it as a masked variable">
    On GitLab, **Settings → CI/CD → Variables → Add variable**:

    * Key: `AGENTMARK_API_KEY`
    * Value: the key from step 1
    * Type: **Variable** (not File)
    * Flags: **Masked**, **Protected**

    On GitHub, add it as a repository **secret** and reference it as `${{ secrets.AGENTMARK_API_KEY }}`.
  </Step>

  <Step title="Reference it from the job">
    The CLI reads `AGENTMARK_API_KEY` directly from the job environment. Once the GitLab component ships, pass it via `inputs.api-key: $AGENTMARK_API_KEY`. Don't hard-code the key in your pipeline config.
  </Step>
</Steps>

Cloud-backed runs (regression-gate baselines, dataset sync) need the key. For fully local evals with no Cloud features, you can skip the variable and run without a key.

## What gets gated

Up to four independent gate predicates fire on every run; any failing fails the job.

1. **Per-row gate**: every `(row × scorer)` pair is a `<testcase>` in the JUnit XML. If the scorer's `passed` flag is `false`, the run emits `<failure>` and your CI reports it inline (in the MR widget on GitLab, the Checks tab on GitHub).
2. **Threshold gate** (optional): when you set `--threshold`, the job fails if the overall pass rate is below the threshold.
3. **Regression gate** (optional): when a baseline run resolves and the prompt sets `test_settings.regression_tolerance`, a row fails if a scorer's score dropped more than the tolerance below its baseline. This catches silent quality drops even when the scorer still "passes" in absolute terms.
4. **Per-scorer threshold gate** (optional): when the prompt sets `test_settings.score_thresholds` (a `{ scorer: minMeanScore }` map), the run fails if a scorer's mean score across the run falls below the configured minimum.

[Regression gates](/deploy/regression-gates) documents the full mechanics: how the gate resolves the baseline by tree hash, matches rows by input content, and keeps missing baselines inert.

## Packaged integrations

The `agentmark-ai/eval-component` GitLab CI/CD Catalog component and the `agentmark-ai/eval-action` GitHub Action automate the wiring: they diff each PR/MR, run `@agentmark-ai/cli` against the changed `.prompt.mdx` files, and emit the JUnit XML for you. They share a contract: both wrap the same CLI command (`run-experiment --format junit`), accept the same `threshold` / `baseline-ref` semantics, and emit the same JUnit XML schema.

<Warning>
  Neither the `agentmark-ai/eval-component` Catalog component nor the `agentmark-ai/eval-action` GitHub Action is **published yet**, so `include: component: gitlab.com/agentmark-ai/eval-component/eval@v1` and `uses: agentmark-ai/eval-action@v1` won't resolve. Use the [raw-CLI setup](#run-evals-in-ci-raw-cli) above; it produces identical JUnit output and gate behavior. The sections below document how the GitLab component works once it's published.
</Warning>

### Attribute eval traces to a PR's preview environment

When an eval runs in CI, its traces can land in the pull request's [preview environment](/deploy/environments-and-promotions), so a failing eval is one click from the trace that produced it. The SDK tracer reads two environment variables to pick the target environment: `AGENTMARK_PR_NUMBER` (the PR whose preview environment to use) and `AGENTMARK_ENVIRONMENT` (an explicit environment name). See [Attributing traces to an environment](/observe/tracing-setup#attributing-traces-to-an-environment).

This needs a key that can write to preview environments. Scope **one** key to the Preview environment kind (Dashboard → app → **Settings → API keys**, **Environment scope → Environment kinds → Preview**) and store it as `AGENTMARK_API_KEY`. A kind-scoped key authorizes every PR's preview environment as it's created, so you never provision a key per pull request. See [API key environment scope](/api-reference/authentication#api-key-environment-scope).

In the [raw-CLI job](#run-evals-in-ci-raw-cli), set the variable yourself in the job environment. On GitHub Actions, `AGENTMARK_PR_NUMBER: ${{ github.event.pull_request.number }}` on a pull-request workflow attributes the run to that PR's preview environment; on GitLab, `AGENTMARK_PR_NUMBER: $CI_MERGE_REQUEST_IID`. For a fixed target (a nightly eval against staging), set `AGENTMARK_ENVIRONMENT: staging` instead.

The GitHub `eval-action` does this for you: it exposes an `environment` input (which sets `AGENTMARK_ENVIRONMENT`) and, on a `pull_request` event, also sets `AGENTMARK_PR_NUMBER` from the PR number automatically. Leave `environment` empty for the PR-preview case; set it only for non-PR runs.

Once an env-scoped run lands its traces, the PR's AgentMark preview comment links straight to them: it carries a **View traces** link to that preview environment's trace list, alongside the **Open preview** link.

### GitLab component quick start

Once the component ships, the include replaces the hand-rolled job:

```yaml theme={null}
include:
  - component: gitlab.com/agentmark-ai/eval-component/eval@v1
    inputs:
      api-key: $AGENTMARK_API_KEY    # masked, protected CI variable

variables:
  GIT_DEPTH: "0"                     # required so the diff base resolves
```

On every MR, the component evaluates the `.prompt.mdx` files changed in the diff and surfaces results inline in the MR widget.

<Warning>
  The component needs `GIT_DEPTH: "0"`. GitLab's default shallow checkout doesn't contain the diff base, so the component can't resolve `$CI_MERGE_REQUEST_DIFF_BASE_SHA` to a tree hash. When that happens, the component disables the regression gate for the run rather than failing the job.
</Warning>

### Inputs

| Input               | Required | Default                     | Description                                                                                                                                                                           |
| ------------------- | -------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api-key`           | optional | None                        | AgentMark API key. Required for Cloud-backed runs; omit for fully local evals.                                                                                                        |
| `prompts`           | optional | changed `.prompt.mdx` files | Newline- or space-separated list of prompt files to evaluate.                                                                                                                         |
| `threshold`         | optional | None                        | Pass-rate threshold (0–100). Fails the job if overall pass rate is below this number.                                                                                                 |
| `baseline-ref`      | optional | MR diff base                | Git ref to compare scores against for the regression gate. Resolved to a tree hash and passed to the CLI as `--baseline-commit`. Requires `GIT_DEPTH=0`. Set empty (`''`) to disable. |
| `working-directory` | optional | `.`                         | Directory to run from.                                                                                                                                                                |
| `results-glob`      | optional | `agentmark-results-*.xml`   | Pattern for per-prompt JUnit XML output files. Must contain exactly one `*` wildcard; the prefix and suffix around it become the per-prompt filename template.                        |
| `cli-version`       | optional | `latest`                    | npm version specifier for `@agentmark-ai/cli`. Pin for reproducible CI.                                                                                                               |
| `image`             | optional | `node:20-bookworm-slim`     | Docker image. Must include npm, git, bash.                                                                                                                                            |

### When the job runs

The component's default `rules:` runs on:

* every **merge request** pipeline (the primary gate), and
* pushes to the **default branch** (so the run records a fresh baseline after merge).

Override in your `.gitlab-ci.yml` to change the cadence:

```yaml theme={null}
include:
  - component: gitlab.com/agentmark-ai/eval-component/eval@v1
    inputs:
      api-key: $AGENTMARK_API_KEY

# Run only on MRs — skip the default-branch baseline write.
agentmark_eval:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
```

The default-branch run is what populates the baseline that subsequent MRs gate against. Skip it only if you're recording baselines through a separate process (a scheduled job, a manual trigger, or the SDK).

### With a regression-tolerance threshold

Set the per-case tolerance and run-level floors in the prompt's frontmatter. The component reads them automatically from `test_settings`, so it doesn't need any extra inputs.

```yaml theme={null}
# agentmark/qa.prompt.mdx (frontmatter)
test_settings:
  dataset: ./data/qa.jsonl
  regression_tolerance: 0.05            # fail a case if a scorer drops >5% below baseline
  score_thresholds:
    groundedness: 0.9                   # fail the run if mean groundedness < 0.9
```

```yaml theme={null}
# .gitlab-ci.yml
include:
  - component: gitlab.com/agentmark-ai/eval-component/eval@v1
    inputs:
      api-key: $AGENTMARK_API_KEY

variables:
  GIT_DEPTH: "0"
```

`baseline-ref` defaults to `$CI_MERGE_REQUEST_DIFF_BASE_SHA`, so MR pipelines pick up the right comparison automatically. The first run on the default branch records the baseline; from then on every MR gates against the run captured at its base commit's tree hash. See [Regression gates](/deploy/regression-gates) for the full gate semantics.

## Coexists with your existing tests

The CLI job and the component both emit JUnit XML, the same format `pytest`, `jest`, and `vitest` already emit. Failures appear alongside any other failing test: in the MR widget and the pipeline **Tests** tab on GitLab, in the Checks tab on GitHub. No new UI to learn, no additional reporter to install.

## See also

* [Status checks](/deploy/status-checks): the validation gate (does every prompt compile?), Cloud-managed and self-hosted.
* [Regression gates](/deploy/regression-gates): full mechanics of the per-case and run-level gates, plus the GitHub Actions workflow and SDK setup.
* [Running experiments](/evaluate/running-experiments): CLI reference and JUnit output details.

<div className="mt-8 rounded-lg bg-blue-50 p-6 dark:bg-blue-900/30">
  <h3 className="font-semibold mb-3">Have questions?</h3>
  <p className="mb-4">Reach out any time:</p>

  <ul>
    <li>
      Email the team at <a href="mailto:hello@agentmark.co" className="text-blue-600 hover:text-blue-800 dark:text-blue-400 dark:hover:text-blue-200">[hello@agentmark.co](mailto:hello@agentmark.co)</a> for support
    </li>

    <li>
      Schedule an <a href="https://cal.com/ryan-randall/enterprise" className="text-blue-600 hover:text-blue-800 dark:text-blue-400 dark:hover:text-blue-200">Enterprise Demo</a> to learn about AgentMark's business solutions
    </li>
  </ul>
</div>