Why my chat agent cost 7,389 tokens to say hello

Forty4HQ, the app I run my consulting firm on, has an agent built in. Her name is Eve. She can read my calendar, update the CRM, draft posts and run my weekly interview. Every reply in the chat panel shows a little badge with the token count and the cost.

That badge turned out to be the most useful thing I built in the first week, mostly because it kept lying to me. Here’s the story in four numbers.

$0.01

The first time Eve worked at all, the badge said one cent.

Getting there took longer than it should have. I’d upgraded my hosting plan and added a card, and the model calls still failed with an error saying free-tier users didn’t have access to that model. Claude pulled the real error from the logs and worked out why: the account plan and the AI gateway’s credit balance are two separate things. Upgrading one doesn’t fund the other. I bought gateway credits, Eve pulled my content calendar, and the badge read $0.01 on Haiku 4.5.

One cent felt great. It also made me trust the badge, which was a mistake.

$0.00

The next night I’d switched Eve over to Sonnet through a different provider, and the badge said $0.00. “Cost isn’t being shown properly. Sonnet costs money and isn’t a free model.”

The cause was boring and very common. The agent framework reported model ids with an extra provider prefix stuck on the front, so the id the app looked up didn’t match anything in the price table. No match, no price, zero dollars. Claude confirmed it by checking what the framework actually reported against the live provider, then fixed the lookup.

That’s a small bug with a nasty failure mode. A cost display that silently shows zero is worse than no cost display, because it tells you to stop worrying.

7,389

An hour later the cost looked right, but the token count didn’t. I’d sent a short message and the badge said something like 10,000 tokens. “I don’t think the token count is correct. How would this cost 10k tokens?”

I expected Claude to find another display bug. Instead it did the thing I now ask for every time: it reproduced the exact call outside the app. Same system prompt, same message, same set of tools, sent straight to the provider’s API, and then it read back what the provider billed.

7,389 input tokens. 13 output tokens.

The count was real. Almost all of it was tool definitions. Eve had more than 30 tools and every one of them was described to the model, with its full schema, on every single message. Ask her what day it is and she’d still read the instructions for creating a sales opportunity first.

The Fix

I told Claude, “Please look into this as I don’t want it to use anything that isn’t needed.”

The fix was to stop carrying everything everywhere. Claude grouped 25 of the tools into three domains (CRM, marketing, interviews) and put each domain behind a small tool: activate_crm_tools, activate_marketing_tools, activate_interview_tools. By default Eve sees 7 tools instead of 31. When a question needs the CRM, she calls the activate tool first, and the tools show up within the same turn.

I liked that it didn’t use a keyword filter. Guessing which tool a message needs is fragile. Letting the model ask for the toolbox it needs is not.

After the gating, I replayed the same short message through the agent the same way. The new baseline was 3,482 input tokens. 12 output tokens. That’s a 54% cut in input tokens.

About half of that came from trimming Eve’s system instructions file, from 6,929 bytes down to 4,103 bytes. Everything we cut was already covered by the tool descriptions or the skill files that load when needed.

$0.00, for real this time

The next two days of testing turned up more.

One evening a mode toggle in the chat panel said “General” while the answers stayed fully scoped to Forty4. At the same time, picking a free model in the dropdown still billed a paid one. Claude’s words: “That’s a real bug I just shipped to production.” Unlike the earlier pricing-table bug, this one wasn’t about looking up the wrong price. The chat library captured its request headers once, when the panel first mounted, and never read them again. The screen changed. The request didn’t.

The next day we ran Eve’s two scheduled jobs end to end in production on a free model, on purpose. The test found four problems, and all four were fixed in one pull request.

The cost of that test was zero. The spend ledger stayed exactly where it was, to the ten-thousandth of a dollar: $3.5968.

The hidden tax

Tool definitions are the hidden tax. Nobody thinks of them as part of the conversation, but the model reads every one of them on every turn, and you pay for it every turn. The more useful your agent gets, the more tools it has, and the more each “hi” costs.

So measure one real call before you optimize anything. Not the dashboard, not the estimate: the call, replayed against the provider, with the provider’s own count. Then gate tools by domain so the agent carries only what the question needs.

And put the cost on the screen next to every reply. Mine showed the wrong number twice in its first three days. The one time it looked wrong and wasn’t, it led to the biggest saving of the week. I’d have missed all three if the number hadn’t been sitting right there under the message.