Optimizing Your AI Agent's Error Rates with ClawPulse

A Practical Instrumentation Wrapper (TypeScript)

The fastest way to start tracking error rates the right way is to wrap your agent invocations once and emit a structured event per turn. This is what the ClawPulse agent does under the hood, but you can run the equivalent yourself in 80 lines:

```ts

// runMonitored.ts — wrap any agent step with error-rate telemetry

import { randomUUID } from "node:crypto";

type FailureMode =

| "tool_validation" | "upstream_timeout" | "rate_limit"

| "guardrail_block" | "schema_drift" | "loop_runaway"

| "handoff_cascade" | "key_expiry" | "unknown";

interface TurnTelemetry {

turn_id: string;

agent: string;

model: string;

status: "ok" | "error";

failure_mode?: FailureMode;

latency_ms: number;

prompt_tokens: number;

completion_tokens: number;

cache_read_tokens?: number;

usd_cost: number;

retries: number;

error_message?: string;

}

const CLAWPULSE_INGEST = process.env.CLAWPULSE_INGEST_URL!;

const CLAWPULSE_TOKEN = process.env.CLAWPULSE_AGENT_TOKEN!;

function classify(err: unknown): FailureMode {

const msg = String((err as Error)?.message || err).toLowerCase();

if (msg.includes("429") || msg.includes("rate limit")) return "rate_limit";

if (msg.includes("invalid_api_key") || msg.includes("401")) return "key_expiry";

if (msg.includes("timeout") || msg.includes("etimedout")) return "upstream_timeout";

if (msg.includes("schema") || msg.includes("zod")) return "schema_drift";

if (msg.includes("guardrail") || msg.includes("policy")) return "guardrail_block";

if (msg.includes("max_iterations") || msg.includes("loop")) return "loop_runaway";

if (msg.includes("tool") && msg.includes("validation")) return "tool_validation";

return "unknown";

}

export async function runMonitored(

agent: string,

model: string,

fn: () => Promise,

opts: { maxRetries?: number; usdBudget?: number } = {},

): Promise {

const turn_id = randomUUID();

const started = Date.now();

let retries = 0;

let lastErr: unknown;

while (retries <= (opts.maxRetries ?? 1)) {

try {

const result = await fn();

// Caller exposes usage on `result` or via global hook; simplified here

const usage = (result as { usage?: { prompt: number; completion: number; cached?: number } }).usage;

const cost = usage ? (usage.prompt 3e-6 + usage.completion 15e-6) : 0;

void emit({

turn_id, agent, model,

status: "ok",

latency_ms: Date.now() - started,

prompt_tokens: usage?.prompt ?? 0,

completion_tokens: usage?.completion ?? 0,

cache_read_tokens: usage?.cached,

usd_cost: cost,

retries,

});

return result;

} catch (err) {

lastErr = err;

retries++;

if (classify(err) === "rate_limit") await sleep(2000 * retries);

}

}

void emit({

turn_id, agent, model,

status: "error",

failure_mode: classify(lastErr),

latency_ms: Date.now() - started,

prompt_tokens: 0, completion_tokens: 0,

usd_cost: 0,

retries,

error_message: String((lastErr as Error)?.message ?? lastErr).slice(0, 500),

});

throw lastErr;

}

function emit(t: TurnTelemetry) {

// Fire-and-forget — never block the agent path on telemetry

return fetch(CLAWPULSE_INGEST, {

method: "POST",

headers: { "content-type": "application/json", "authorization": `Bearer ${CLAWPULSE_TOKEN}` },

body: JSON.stringify(t),

}).catch(() => {});

}

function sleep(ms: number) { return new Promise((r) => setTimeout(r, ms)); }

```

Three things this wrapper buys you on day one:

1. Per-turn cost attribution (you can now compute "cost per intent" or "cost per user" in SQL).

2. Failure classification at the source — error logs are already noisy, so classifying at emit-time means dashboards stay clean even when the LLM provider returns the same generic 500.

3. Retry visibility — a 0% error rate that ships with `retries: 4` per turn is not a healthy fleet; it's a fleet that's burning 5x the tokens and 5x the latency to look healthy.

Three SQL Recipes That Reveal Hidden Errors

Once telemetry lands in ClawPulse (or any column-store you control), these queries surface failures that aggregate dashboards mask:

1. Failure mode share by agent — week-over-week

```sql

SELECT

agent,

failure_mode,

COUNT(*) AS turns,

ROUND(100.0 COUNT() / SUM(COUNT(*)) OVER (PARTITION BY agent), 2) AS pct_of_agent

FROM turn_events

WHERE status = 'error' AND ts >= NOW() - INTERVAL '7 days'

GROUP BY agent, failure_mode

ORDER BY agent, turns DESC;

```

This is the first query to run when an agent's error rate ticks up. A flat 2% rate split evenly across modes is a healthy fleet under normal load; a 2% rate that's 90% `schema_drift` means a tool contract changed.

2. Silent-retry cost burn

```sql

SELECT

agent,

COUNT(*) FILTER (WHERE retries > 0) AS turns_with_retry,

COUNT(*) AS total_turns,

ROUND(SUM(usd_cost) FILTER (WHERE retries > 0)::numeric, 2) AS retry_cost_usd,

ROUND(SUM(usd_cost)::numeric, 2) AS total_cost_usd

FROM turn_events

WHERE ts >= NOW() - INTERVAL '24 hours'

GROUP BY agent

HAVING COUNT(*) FILTER (WHERE retries > 0) > 0

ORDER BY retry_cost_usd DESC;

```

If `retry_cost_usd / total_cost_usd > 0.20`, you have a reliability problem disguised as a budget problem.

3. The "loop runaway" early warning

```sql

SELECT

date_trunc('hour', ts) AS hour,

agent,

AVG(prompt_tokens + completion_tokens) AS avg_tokens_per_turn,

PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY prompt_tokens + completion_tokens) AS p95_tokens

FROM turn_events

WHERE ts >= NOW() - INTERVAL '48 hours'

GROUP BY 1, 2

HAVING PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY prompt_tokens + completion_tokens) > 80000

ORDER BY hour DESC;

```

When p95 token-per-turn drifts above 80k, the agent is starting to accumulate context without converging — the precursor to a runaway. Catch it here, before the bill arrives.

Real Postmortem: The 4.5x Latency Drift That Looked Like a Healthy Error Rate

In February 2026, a ClawPulse customer running a multi-agent customer-support pipeline (3 agents, ~14 tools, GPT-4.1 mini + Sonnet 4.6) saw their error rate stay flat at 0.8% for six days while p95 latency tripled and per-turn cost grew 4.5x. Standard Datadog dashboards pinging only HTTP status said "all green."

Root cause: a typo in a context-compaction trigger flipped the threshold from 200,000 tokens to 20,000. Every turn was recompacting context after a few exchanges, so each compaction added ~1.4s of LLM round-trip time and re-billed the prompt without changing the output. Errors stayed "low" because the agent kept finishing — just slower and more expensively.

What surfaced it was recipe #3 above: p95 tokens-per-turn climbed from 12k to 87k over four days. The fix was a one-line config rollback. The signal was already in the telemetry; the dashboard just wasn't asking the right question.

Lesson: a low error rate on a fleet that's quietly getting slower and more expensive is not a healthy fleet — it's a fleet on the way to an outage that you'll pay for twice (in cost, and in customer churn).

How ClawPulse Compares to the Alternatives for Error-Rate Tracking

| Tool | Per-turn classified errors | Cost-per-error attribution | Real-time alerts | Self-hosted option |

|---|---|---|---|---|

| OpenAI dashboard | No (HTTP-status only) | No | No | N/A |

| LangSmith | Yes (trace-level) | Partial | Yes | Cloud-only |

| Langfuse | Yes (event-level) | Yes | Yes | Yes |

| ClawPulse | Yes (turn + tool + handoff) | Yes (per-user, per-intent) | Yes (sub-minute) | Yes (single-binary agent) |

For deeper alternative comparisons, see our Langfuse alternatives roundup, Helicone alternatives, and our deeper debugging playbook for production AI agents.

External references worth bookmarking:

• OpenTelemetry GenAI semantic conventions: opentelemetry.io/docs/specs/semconv/gen-ai/ — the standard your error events should align to.
• Anthropic prompt-caching docs: docs.anthropic.com/en/docs/build-with-claude/prompt-caching — caching cuts both cost and the surface area where context-staleness errors hide.
• OpenAI Agents SDK: platform.openai.com/docs/guides/agents-sdk — read the failure-mode notes before scaling beyond a single agent.
• LangChain error-handling guide: python.langchain.com/docs/how_to/agent_executor#error-handling — useful even if you're not on LangChain, for the taxonomy alone.

Frequently Asked Questions

Ready to put real error-rate observability in front of your fleet? Start a 14-day trial, book a 15-minute demo, or compare plans on the pricing page.