LLM API Rate Limiting Best Practices: Avoid 429 Errors and Save 40% on Costs

Every production LLM application eventually hits the same wall: a `429 Too Many Requests` error at 3 AM, right when traffic spikes. Whether you're calling Claude, GPT-4, or running multi-agent workflows, rate limiting isn't optional — it's the difference between a reliable product and a flaky one. This guide covers the patterns that actually hold up under load, with code you can ship today.

Why LLM Rate Limits Are Different

Traditional API rate limits count requests per second. LLM providers add a second dimension: tokens per minute (TPM). Anthropic's rate limit documentation shows tier-based quotas where Tier 1 caps you at 50 requests/minute and 40,000 input tokens/minute for Claude Opus. OpenAI's usage tier system works similarly.

The asymmetry matters. A single agent loop calling Claude with a 50K-token context window can exhaust your TPM budget in three calls — even though you've technically only made 3 of your 50 allowed RPM. Most teams blow through limits not because they send too many requests, but because their context windows balloon over time.

A real example from production: an agent built with LangGraph accumulates conversation history across iterations. By turn 8, each request carries 80K tokens. Three concurrent users → instant 429. The fix isn't "more retries" — it's measuring tokens before you send.

Cost dimension you're probably ignoring

A Claude Sonnet 4.6 request averages $0.003 per 1K input tokens and $0.015 per 1K output tokens. If your retry logic naively replays a failed 80K-token request three times, you've burned $0.72 to deliver one response. Multiply by 10,000 daily users and rate limit handling becomes a six-figure cost decision. Aggressive retries without circuit breakers are the most expensive bug in modern LLM apps.

Pattern 1: Token Bucket With Dual Counters

Forget the textbook token bucket — you need two buckets running in parallel: one for requests, one for tokens. Here's a minimal implementation in Python:

```python

import asyncio

import time

from dataclasses import dataclass

@dataclass

class DualBucket:

rpm_limit: int

tpm_limit: int

request_tokens: float

token_tokens: float

last_refill: float

async def acquire(self, estimated_tokens: int):

while True:

now = time.monotonic()

elapsed = now - self.last_refill

self.request_tokens = min(

self.rpm_limit,

self.request_tokens + elapsed * (self.rpm_limit / 60)

)

self.token_tokens = min(

self.tpm_limit,

self.token_tokens + elapsed * (self.tpm_limit / 60)

)

self.last_refill = now

if self.request_tokens >= 1 and self.token_tokens >= estimated_tokens:

self.request_tokens -= 1

self.token_tokens -= estimated_tokens

return

await asyncio.sleep(0.1)

```

The trick: estimate tokens before calling the API using `anthropic.count_tokens()` or tiktoken for OpenAI. Pre-flight measurement prevents the worst failure mode where you queue 500 requests, all fit RPM, none fit TPM, and your queue deadlocks.

Pattern 2: Exponential Backoff Done Right

Most "exponential backoff" implementations are wrong because they ignore the `retry-after` header. Anthropic and OpenAI both return this header on 429 responses with the exact wait time. Use it.

```javascript

async function callWithBackoff(fn, maxRetries = 5) {

for (let attempt = 0; attempt < maxRetries; attempt++) {

try {

return await fn();

} catch (err) {

if (err.status !== 429 && err.status !== 529) throw err;

const retryAfter = parseInt(err.headers?.['retry-after']) || null;

const jitter = Math.random() * 1000;

const wait = retryAfter

? retryAfter * 1000 + jitter

: Math.min(2 * attempt 1000 + jitter, 30000);

console.warn(`Rate limited. Retrying in ${wait}ms`);

await new Promise(r => setTimeout(r, wait));

}

}

throw new Error('Max retries exceeded');

}

```

Three non-obvious things this gets right:

1. Handles 529 (overloaded) — Anthropic returns this during capacity events; treat it like 429.

2. Adds jitter — without it, all your servers retry at exactly the same millisecond and recreate the spike.

3. Caps wait at 30s — beyond that, fail fast and let the user retry manually.

The official Anthropic SDK does this internally with `max_retries`, but if you're calling via raw `fetch` or curl, you need it explicit.

Pattern 3: Request Coalescing for Agents

If you're running multi-agent systems (think: 5 Claude agents each making decisions in parallel), naive parallelism guarantees rate limit hell. Coalesce identical requests:

```python

from functools import lru_cache

import hashlib

class CoalescingClient:

def __init__(self, client):

self.client = client

self.in_flight = {}

async def complete(self, messages, **kwargs):

key = hashlib.md5(

str(messages).encode() + str(kwargs).encode()

).hexdigest()

if key in self.in_flight:

return await self.in_flight[key]

future = asyncio.create_task(

self.client.messages.create(messages=messages, **kwargs)

)

self.in_flight[key] = future

try:

return await future

finally:

self.in_flight.pop(key, None)

```

In production agent swarms, we've seen this cut request volume by 35-60% because agents frequently make redundant context-checking calls. Combine with Anthropic's prompt caching for an additional 90% cost reduction on cached portions.