Once an AI workflow moves past a pilot, the first production error is often a 429 Too Many Requests. The model is fine and the prompt is fine. The account sent more requests or tokens in a minute than the provider allows.
A common first idea is to create more API keys and rotate through them, a practice usually called API key rotation. We built two open-source tools that do key and target rotation, a Load Balancer for OpenAI-compatible APIs and a Vertex AI Load Balancer for Google Cloud, so we have spent time on what rotation actually solves. The short answer: it solves reliability and visibility problems well. It usually does not raise your limit.
How LLM rate limits are counted
Most providers count usage at the account or project level, not per key. That one fact decides whether rotation helps.
- OpenAI. The rate limits guide says limits are defined at the organization level and at the project level, not the user level. Limits are measured in requests per minute and tokens per minute, among others, and they rise as your usage tier goes up.
- Gemini API. Google's Gemini rate limits page says it directly: rate limits are applied per project, not per API key.
- Vertex AI. Vertex AI quotas are tracked per Google Cloud project, and many are also per region.
So ten keys in the same project share one limit. Rotating through them changes which key signs each request. The project can still send the same amount.
OpenAI's guide adds a second warning: unsuccessful requests count toward your per-minute limit, so resending a failed request in a tight loop makes the problem worse.
What actually raises throughput
If you need more capacity, the reliable options are the boring ones:
- 1Move up a usage tier or request a quota increase. This is the provider's supported path, and it is usually the cheapest one once you count engineering time.
- 2Send fewer tokens. Shorter prompts, smaller context, and cached system prompts reduce tokens per minute, which is often the limit you hit first.
- 3Use a smaller model for easy steps. Routing classification or extraction to a cheaper model leaves headroom for the hard calls. Our AI agent cost guide covers this tradeoff.
- 4Queue instead of burst. A queue that smooths a batch of 500 requests over several minutes stays under a per-minute limit that a burst would blow through.
- 5Retry with backoff and jitter. OpenAI recommends honoring the Retry-After header when present, then falling back to exponential backoff with random jitter, with limits on attempts and total time.
We'll find which limit your workflows hit and add the queuing, retries and failover that fix it without breaking provider terms.
What key rotation is good for
Rotation still earns its place as a reliability tool. It just does not add capacity.
- Failover. Keys get revoked, expire, hit a billing problem, or start failing for reasons unrelated to load. A rotator that marks a failing key and moves on keeps the workflow running while someone fixes it.
- One key for your apps, many behind it. Your applications call one internal endpoint with one internal key. Provider keys stay in one place, so rotating or revoking them does not mean redeploying every app.
- Separate budgets that you already own. An agency or a company with several departments may keep separate provider projects per client or team, each with its own billing and limits. A gateway can route each workload to the right project.
- Regions. On Vertex AI, quotas are often per region. Routing across regions within your own project can spread load and give you a fallback if one region has trouble.
- Visibility. A single gateway is a natural place to log every request, error, model, and response time, which most teams lack until something breaks.
One caution: creating extra accounts or projects only to get around a provider's limits can break its terms of service. Read the terms for each provider before pooling capacity, and use the quota increase process when that is what you need.
What our two load balancers do
Both projects are self-hosted Next.js apps with a SQLite database, and both are public on our projects page.
The Load Balancer sits in front of any OpenAI-compatible API. Its README describes key management with automatic rotation, failure detection that deactivates a key after a set number of failures, a cooldown after rate-limit responses, request logs and usage statistics, masked keys in the UI, and an optional master API key so only your apps can call it. The target endpoint is configurable.
The Vertex AI Load Balancer does the same job for Vertex AI. Each target is a project, location, model, and service account key, with an optional daily limit. It picks the next healthy target, tracks per-target usage, errors, and response times, and supports import and export of targets.
Neither tool raises a provider's limit, and we do not describe them that way here. They keep a workflow running when one key or target fails, and they show you what is happening across all of them.
The older Gemini Load Balancer is deprecated in favor of the OpenAI-compatible version, which is why it sits at the bottom of our projects list.
When a hosted gateway is the better choice
A self-hosted gateway is one more service your team has to keep running and patched. If you need multiple providers, team-level budgets, and a dashboard without running anything, a hosted LLM gateway may suit you better. If you want the data and keys to stay on your own server and your needs are simple, a small self-hosted proxy is easy to audit.
Either way, treat the gateway as production infrastructure. It holds every provider key you have, so it needs access control, secret storage, and alerts when error rates rise. Our post on catching automations that silently stop working covers the alerting side.
A short checklist
- Find which limit you are hitting: requests per minute, tokens per minute, or daily requests. The error and response headers usually say.
- Check whether the limit is per project. If it is, more keys will not help.
- Add Retry-After handling and backoff with jitter before anything else.
- Queue batch work so it does not arrive as one burst.
- Ask for a higher tier or quota once the workload is real.
- Add a gateway for failover, central keys, and logging if you run several apps or projects.
How OrchestriAI handles this
Rate limits are an integration problem as much as an AI problem. In our systems integration work we design the queues, retries, and alerts around model calls, and in AI agent builds we size prompts and model choices so the workflow fits the limits you have.
If your automations are failing with 429 errors or you are not sure what your AI usage looks like, book a free automation audit. We'll find which limit you are hitting and what the cheapest fix is.
