Skip to main content
AgentMark uses OpenTelemetry to collect distributed traces and metrics for your prompt executions. This provides detailed visibility into how your prompts perform in development and production.

​
Setup

Initialize tracing in your application: 
import { AgentMarkSDK } from "@agentmark-ai/sdk";
import { createAgentMarkClient, VercelAIModelRegistry } from "@agentmark-ai/ai-sdk-v5-adapter";
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";

const sdk = new AgentMarkSDK({
  apiKey: process.env.AGENTMARK_API_KEY,
  appId: process.env.AGENTMARK_APP_ID,
  baseUrl: process.env.AGENTMARK_BASE_URL  // defaults to https://api.agentmark.co
});

// Initialize tracing
const tracer = sdk.initTracing();

// Configure client
const modelRegistry = new VercelAIModelRegistry()
  .registerModels(["gpt-4o-mini"], (name) => openai(name));

const client = createAgentMarkClient({
  loader: sdk.getApiLoader(),
  modelRegistry
});

// Load and run prompt with telemetry
const prompt = await client.loadTextPrompt("greeting.prompt.mdx");
const input = await prompt.format({
  props: { name: 'Alice' },
  telemetry: {
    isEnabled: true,
    functionId: "greeting-function",
    metadata: {
      userId: "123",
      environment: "production"
    }
  }
});

const result = await generateText(input);

// Shutdown tracer (only for short-running scripts)
await tracer.shutdown();
Pass disableBatch: true to initTracing() for short-running scripts that need traces flushed immediately:
const tracer = sdk.initTracing({ disableBatch: true });

​
Collected Spans

AgentMark records these OpenTelemetry spans:
Span TypeDescriptionKey Attributes
ai.inferenceFull inference call lifecycleai.model.id, ai.prompt, ai.response.text, ai.usage.promptTokens, ai.usage.completionTokens
ai.toolCallIndividual tool executionsai.toolCall.name, ai.toolCall.args, ai.toolCall.result
ai.streamStreaming response metricsai.response.msToFirstChunk, ai.response.msToFinish, ai.response.avgCompletionTokensPerSecond

​
Span Attributes

Each span contains detailed attributes: Model Information:
  • ai.model.id - Model identifier (e.g., “gpt-4o-mini”)
  • ai.model.provider - Provider name (e.g., “openai”)
Token Usage:
  • ai.usage.promptTokens - Input tokens
  • ai.usage.completionTokens - Output tokens
Telemetry Metadata:
  • ai.telemetry.functionId - Your function identifier
  • ai.telemetry.metadata.* - Custom metadata fields
Response Details:
  • ai.response.text - Generated text
  • ai.response.toolCalls - Tool call information
  • ai.response.finishReason - Why generation stopped

​
Grouping Traces

Group related operations using the trace function. The trace function accepts a TraceOptions object and a callback that receives a TraceContext: 
import { trace } from "@agentmark-ai/sdk";

const { result, traceId } = await trace(
  { name: 'user-request-handler' },
  async (ctx) => {
    // All spans created here are grouped under this trace
    const prompt = await client.loadTextPrompt('handler.prompt.mdx');
    const input = await prompt.format({
      props: { query: 'What is AgentMark?' },
      telemetry: { isEnabled: true }
    });

    return await generateText(input);
  }
);

// traceId is available for correlation (e.g., logging, scoring)
console.log('Trace ID:', traceId);

// result is a Promise<T> — await it if you need the value
const output = await result;

​
TraceOptions

The trace function accepts the following options:
OptionTypeRequiredDescription
namestringYesName for the trace
metadataRecord<string, string>NoCustom key-value metadata
sessionIdstringNoGroup traces into a session
sessionNamestringNoHuman-readable session name
userIdstringNoAssociate trace with a user
datasetRunIdstringNoLink to a dataset run
datasetRunNamestringNoHuman-readable dataset run name
datasetItemNamestringNoSpecific dataset item name
datasetExpectedOutputstringNoExpected output for evaluation
datasetPathstringNoPath to the dataset file

​
TraceResult

The trace function returns a TraceResult<T>: 
interface TraceResult<T> {
  result: Promise<T>;  // The result of your callback (as a Promise)
  traceId: string;     // The trace ID for correlation
}
Note that result is Promise<T>, not T. You need to await it to get the resolved value.

​
TraceContext

The callback receives a TraceContext with methods for adding attributes, events, and child spans: 
interface TraceContext {
  readonly traceId: string;
  readonly spanId: string;
  setAttribute: (key: string, value: string | number | boolean) => void;
  addEvent: (name: string, attributes?: Attributes) => void;
  span: <T>(options: SpanOptions, fn: (ctx: TraceContext) => Promise<T>) => Promise<T>;
}

​
Creating Child Spans

Use ctx.span() to create child spans within a trace. Each child span also receives a TraceContext, so you can nest spans arbitrarily: 
import { trace } from "@agentmark-ai/sdk";

const { result, traceId } = await trace(
  { name: 'multi-step-workflow' },
  async (ctx) => {

    // Child span for validation
    await ctx.span({ name: 'validate-input' }, async (spanCtx) => {
      spanCtx.setAttribute('input.length', 42);
      // Validation logic
    });

    // Child span for main processing
    const output = await ctx.span({ name: 'process-request' }, async (spanCtx) => {
      const prompt = await client.loadTextPrompt('process.prompt.mdx');
      const input = await prompt.format({
        props: { query: 'process this' },
        telemetry: { isEnabled: true }
      });
      return await generateText(input);
    });

    // Child span for formatting
    await ctx.span({ name: 'format-response' }, async (spanCtx) => {
      spanCtx.addEvent('formatting-complete');
    });

    return output;
  }
);

​
Graph View for Complex Workflows

For complex AI agent workflows, use graph metadata to visualize execution flow: 
import { trace } from "@agentmark-ai/sdk";

const { result } = await trace(
  {
    name: "ai-agent-workflow",
    metadata: {
      "graph.node.id": "orchestrator",
      "graph.node.display_name": "Orchestrator",
      "graph.node.type": "router"
    }
  },
  async (ctx) => {

    // Step 1: Process input
    await ctx.span(
      {
        name: "input-processor",
        metadata: {
          "graph.node.id": "input-processor",
          "graph.node.parent_id": "orchestrator",
          "graph.node.display_name": "Input Processor",
          "graph.node.type": "agent"
        }
      },
      async (spanCtx) => {
        // Process and validate input
      }
    );

    // Step 2: Retrieve context
    await ctx.span(
      {
        name: "context-retrieval",
        metadata: {
          "graph.node.id": "context-retrieval",
          "graph.node.parent_id": "orchestrator",
          "graph.node.display_name": "Context Retrieval",
          "graph.node.type": "retrieval"
        }
      },
      async (spanCtx) => {
        // Fetch relevant context
      }
    );

    // Step 3: LLM reasoning (depends on both previous steps)
    await ctx.span(
      {
        name: "llm-reasoning",
        metadata: {
          "graph.node.id": "llm-reasoning",
          "graph.node.parent_ids": JSON.stringify(["input-processor", "context-retrieval"]),
          "graph.node.display_name": "LLM Reasoning",
          "graph.node.type": "llm"
        }
      },
      async (spanCtx) => {
        const prompt = await client.loadTextPrompt('reason.prompt.mdx');
        // ... reasoning logic
      }
    );
  }
);

​
Graph Metadata

PropertyDescriptionRequired
graph.node.idUnique node identifierYes
graph.node.display_nameHuman-readable nameYes
graph.node.typeVisual type: router, llm, tool, retrieval, memory, agentYes
graph.node.parent_idSingle parent node IDNo*
graph.node.parent_idsJSON array of parent IDsNo*
*Either parent_id or parent_ids required (except for root).

​
Node Types

  • router - Orchestration and routing
  • llm - LLM inference and reasoning
  • tool - External tool execution
  • retrieval - Information retrieval
  • memory - Memory operations
  • agent - Agent-specific processing

​
Scoring Traces

Use sdk.score() to attach quality scores to traces or spans: 
const sdk = new AgentMarkSDK({
  apiKey: process.env.AGENTMARK_API_KEY,
  appId: process.env.AGENTMARK_APP_ID
});

const { result, traceId } = await trace(
  { name: 'scored-workflow' },
  async (ctx) => {
    // ... your logic
    return output;
  }
);

// Score the trace
await sdk.score({
  resourceId: traceId,
  name: 'correctness',
  score: 0.95,
  label: 'correct',
  reason: 'Output matches expected result'
});

​
Best Practices

Use meaningful function IDs: 
// Bad
telemetry: { functionId: "func1" }

// Good
telemetry: { functionId: "customer-support-greeting" }
Add relevant metadata: 
telemetry: {
  isEnabled: true,
  functionId: "search-handler",
  metadata: {
    userId: user.id,
    environment: process.env.NODE_ENV,
    query: searchQuery,
    resultCount: String(results.length)
  }
}
Monitor in production:
  • Always enable telemetry in production
  • Use the AgentMark dashboard to monitor performance
  • Set up alerts for anomalies
  • Review traces when debugging user issues
Clean up resources: 
// For short-running scripts (like CLI tools)
await tracer.shutdown();

// For long-running servers, shutdown on process exit
process.on('SIGTERM', async () => {
  await tracer.shutdown();
  process.exit(0);
});

​
Viewing Traces

Traces are viewable in the AgentMark platform dashboard showing:
  • Complete execution timeline
  • Token usage and costs
  • Custom metadata
  • Error information
  • Graph visualization (when enabled)

​
Next Steps

Sessions

Group related traces together

AgentMark Platform

View traces in the dashboard