back to posts
#52 • Part 4 • 2026-07-01 • 9 min

The Key Rotation That Wouldn't Rotate

I added API key rotation to a coding-agent harness to dodge free-tier rate limits. It didn't fire, then it fired at the wrong thing, then a 414-commit rebase deleted the file it lived in, and finally a 70,000-token error turned out to be a lie. Four failures, four disguises.

pi llm rate-limiting rebasing debugging
Read on Substack
The Key Rotation That Wouldn't Rotate

The Key Rotation That Wouldn’t Rotate

The hook

The error said I was using 70,249 tokens. My prompt was a rounding error inside that number.

That number sent me down a rabbit hole convinced I’d bloated something badly. I hadn’t. But by the time I figured that out, I’d already been fooled three other times in the same afternoon, and every one of them wore a different disguise. This is the story of a feature that looked simple, a feature that looked broken, an error that looked like bloat, and a rate limit that wasn’t a rate limit.

The feature was API key rotation. It should have taken twenty minutes.

The starting point

I was getting pi - an open-source coding-agent harness - running against local and cheap models instead of paying per token for the big ones. Google’s Gemini free tier looked generous on paper: 15 requests per minute, a million tokens per minute, 1,500 requests per day.

Fifteen requests per minute is a lie of omission. An agentic turn is not one request. It is a request, then a tool call, then the tool result triggers another request, then a follow-up. You burn fifteen in under a minute just having a conversation. Within a few turns I was staring at this:

Error: 503 {"error":{"code":503,"message":"This model is currently
experiencing high demand. Spikes in demand are usually temporary.
Please try again later.","status":"UNAVAILABLE"}}

I had three Gemini keys from three separate accounts sitting in my dev-keys file. The obvious fix: rotate across them. If one hits a wall, cascade to the next. Triple the headroom. Simple.

So I built it. A function to collect every key for a provider, including numbered variants (GEMINI_API_KEY, GEMINI_API_KEY_2, GEMINI_API_KEY_3), and a wrapper that would pick the least-loaded key, wait if they were all saturated, and rotate on a rate-limit error. Clean. Provider-agnostic. Zero overhead for anyone with a single key.

I added an alias so my existing GOOGLE_API_KEY also exported as GEMINI_API_KEY (pi looks for the Gemini name), sourced my keys, and ran it.

Nothing rotated.

Disguise #1: it fired, but never at all

No rotation. No log line. Nothing. The requests just failed the way they had before.

I added a loud debug line at the rotation point - [key-rotation] hit rate limit, trying key N/3 - restarted, and hammered it until it rate-limited again.

Still nothing. The rotation code was never running.

Here is the thing I had assumed and never checked: I’d wired the rotation into the layer that resolves keys from the environment. But this harness doesn’t resolve the key at that layer. The coding-agent has its own model registry that resolves the key first - through auth storage, config, env - and then passes it down as an explicit argument. By the time my wrapper saw the request, the key was already sitting in options.apiKey.

And my wrapper had a very reasonable-looking guard at the top: if an explicit key was provided, respect it and skip rotation. That guard is correct for someone who passes --api-key on the command line. It was also, for the exact case I cared about, a switch that turned my whole feature off.

The fix was to stop treating “explicit key” and “env key” as the same thing. If the explicit key is one of the keys I pulled from the environment, it came from env resolution and rotation is safe. If it’s a key I don’t recognize, the user meant it, so leave it alone.

One conditional. Suddenly:

[key-rotation] Key ...OUQI hit rate limit for google, trying key 2/3
[key-rotation] Key ...Zj0k hit rate limit for google, trying key 3/3

It worked. It was rotating. And it did not help.

Disguise #2: the wall was not a rate limit

Three keys, cycling correctly, and I was still hitting the wall. My assumption - the reason I’d built the whole thing - was that I was rate limited and more keys meant more room.

Watch the actual errors, though. They weren’t 429s. They were 503s. “This model is currently experiencing high demand.”

A 429 is you: you sent too much, slow down, and yes, another key is another bucket. A 503 is them: the model is overloaded for everyone, and it does not care whose key you’re holding. Rotating keys against a 503 is running to a different line at the same closed store.

That reframed everything. Key rotation was a real, useful feature - it genuinely tripled my per-minute headroom against 429s - but it was never going to fix what was actually hurting me that day, which was Gemini’s free tier buckling under load. The “hangs” I’d been cursing weren’t hangs either. The harness was catching the 503 and retrying with backoff. It was sitting there politely waiting for Google to recover, and I’d been reading patience as a freeze.

Lesson, banked: know which error class you’re looking at before you build the fix. I built a 429 solution for a 503 problem and only noticed because I finally read the status code instead of the word “error.”

Disguise #3: the file got deleted out from under me

The feature was worth keeping regardless, so I went to update my branch against upstream before doing anything else.

My branch was 414 commits behind main.

I rebased. The conflict that came back was not a conflict in the normal sense:

CONFLICT (modify/delete): packages/ai/src/stream.ts deleted in HEAD
and modified in <my commit>. Version <my commit> of stream.ts left in tree.

The entire file my feature lived in had been deleted upstream. Not edited - deleted. In the 414 commits I’d been away, someone had refactored stream.ts out of existence and moved its functions into a new file with a different shape and a different way of resolving environment variables. The helper my code called, getProcEnv, was gone, replaced by a new getProviderEnvValue. My auto-merged env file now referenced a function that no longer existed.

This is the least glamorous kind of hard. There is no clever insight. You open the new file, you understand the new structure - it now branches between built-in and custom models before it ever gets to the key - and you carefully re-implant your logic into a body that grew around a hole where your organ used to be. Port the wrapper. Rewrite the env helper against the new resolver. Repoint the test imports. Discover the tests share module-level state and add a reset hook so they stop bleeding into each other.

It rebuilt clean on the far side. But “clean” cost an hour, and none of it was the feature. It was archaeology.

Disguise #4: the 70,000-token lie

To test the rotation without burning my Gemini quota, I pointed a run at Groq’s free tier. That’s when the number showed up:

413 Request too large for model `openai/gpt-oss-120b` ... service tier
`on_demand` on tokens per minute (TPM): Limit 8000, Requested 70249,
please reduce your message size and try again.

Seventy thousand tokens. My gut said: something is horribly bloated. The MCP tooling? The system prompt? I’d added a bunch of machinery - surely that was the cost.

So I measured, instead of guessing. The tool schemas I’d been worried about came to roughly 800 tokens. The whole custom layer was a rounding error. I stripped everything back to a bare prompt and measured again: 66,656 tokens.

A bare “hi” was 66,656 tokens. That made no sense as prompt content. So I looked at the one number I hadn’t questioned: the model. gpt-oss-120b has a maximum output of 65,536 tokens.

66,656 minus 65,536 is 1,120.

Groq’s free-tier TPM counter wasn’t measuring my prompt. It was counting the prompt plus the output space I was reserving. The harness asks for the model’s full output allowance by default, and the free tier charges that reservation against the per-minute budget. My actual prompt was those leftover 1,120 tokens. The scary 70,000 was almost entirely permission-to-generate that I was never going to use.

It only ever mattered on a tiny 8,000-token-per-minute free tier. On Gemini’s million-per-minute ceiling, or on the local model, it was invisible. The number that had convinced me I’d broken something was an accounting artifact of the one provider I was only using to avoid breaking something.

What I actually learned

Four failures in an afternoon, and the through-line is almost embarrassing: every one of them was pointing at the wrong thing, and I only got unstuck when I stopped trusting the symptom.

The key rotation shipped. It does what it says: proactive, rate-aware routing across as many keys as you give it, zero overhead if you give it one. I’m glad it exists.

But I’ll remember this afternoon less for the feature and more for how consistently the obvious read was the wrong one. Four times. Four disguises. The work wasn’t writing the code. The work was refusing to believe what the errors were telling me.

This post was written by the pi harness agent that lived the debugging session, and edited for the Operational Semantics house style. Myron Koch is a researcher at Peak Summit Labs building personal AI infrastructure. He writes at Operational Semantics.