Rate limits will kill your agent at scale. Here's how to design around them.

Your agent makes three LLM calls and five tool calls per user request. At one concurrent user, that's eight calls total. At fifty concurrent users, that's four hundred calls hitting your upstream APIs simultaneously.

Most LLM providers rate limit by requests per minute and tokens per minute. Most external APIs rate limit by requests per second or requests per minute. At fifty concurrent users, your agent's call rate is fifty times higher than at one user — but your rate limits haven't changed.

You will hit the limit. The question is whether your agent is designed to handle it gracefully.

Two kinds of rate limits to manage

LLM provider limits (RPM + TPM). Your LLM provider limits how many requests and tokens you can consume per minute. An agent that makes three LLM calls per user request consumes three times the RPM quota of a single-call system. At twenty concurrent users making three calls each, that's sixty RPM. At a 100 RPM limit, you have headroom. At sixty concurrent users, you're over.

Token limits are sneakier. An agent accumulating context over a long session uses more tokens per call as the session grows. Token consumption is not linear — it grows with session length. Plan for the worst-case token consumption, not the average.

Upstream tool API limits. Every external API your agent calls has its own rate limit. These limits are set for the API's total traffic across all your users, not per-user. When thirty concurrent agent sessions each call get_user_record simultaneously, that's thirty calls to your user service in a tight burst. If the service rate limits at twenty requests per second, you get throttled.

The patterns

Token bucket at the agent gateway. Before any agent session makes calls to external services, it requests tokens from a token bucket that governs the total call rate. The bucket refills at the maximum sustainable rate. Sessions that arrive when the bucket is empty wait rather than proceed.

This pattern is the standard for preventing thundering herd: many concurrent sessions trying to call the same API simultaneously. The token bucket smooths the burst into a steady flow.

class AgentGateway:
    def __init__(self, max_calls_per_second: float):
        self.bucket = TokenBucket(capacity=max_calls_per_second * 10, 
                                   refill_rate=max_calls_per_second)
    
    async def call_tool(self, tool: str, args: dict) -> dict:
        await self.bucket.acquire()  # Blocks if rate limit reached
        return await tools[tool].call(args)

Per-tool rate limiters. Different tools have different limits. A search API might allow 100 calls per minute. A payment API might allow 10. Implement a separate rate limiter for each tool, configured with that tool's actual limit.

Exponential backoff with jitter on 429 responses. When a 429 (rate limit exceeded) response arrives, back off before retrying. Use exponential backoff (double the wait time on each retry) with jitter (add randomness to the wait time). The jitter prevents all waiting sessions from retrying simultaneously when the rate limit window resets.

async def call_with_backoff(tool: str, args: dict, max_retries: int = 3) -> dict:
    for attempt in range(max_retries):
        try:
            return await call_tool(tool, args)
        except RateLimitError as e:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.uniform(0, 1)  # Exponential + jitter
            await asyncio.sleep(wait)

LLM call batching and caching. For tool calls that return the same result for the same input (user record lookup, product catalog, etc.), cache the result within the session and across sessions with a TTL. If ten concurrent sessions all need the same user record, make one API call and share the result.

For LLM calls, batch similar requests where possible. If multiple low-priority tasks can be processed together, batch them into a single LLM call rather than making ten sequential calls.

Queue with priority and rate shaping. For high-load systems, move the agent's calls through a work queue. The queue accepts calls at any rate but dispatches them to upstream services at the rate those services allow. Priority queuing ensures that high-priority sessions (paying customers, SLA-bound users) get dispatched first.

Designing for your actual limits

Before you deploy, document the rate limits for every upstream dependency your agent calls:

The binding constraint is your lowest-capacity upstream relative to your call multiplier. In the example above, the payment API's 10 RPM limit, combined with one call per user request, caps you at ten concurrent users before you hit rate limits.

Design your backpressure architecture around the binding constraint. If the payment API is the bottleneck, implement a payment API-specific queue with priority handling. Don't design for the average limit — design for the minimum.

Test under load before you ship. Run your agent at 2x your expected concurrent user count. Measure which upstream services get throttled first. Verify that your backpressure implementation actually throttles requests rather than cascading failures to users. Fix the gaps before production.

The agent that works for one user and fails for fifty didn't fail because of the agent. It failed because no one modeled the call amplification at scale. Model it now.