The Tokens That Made Me Rethink Everything

Asking a model to explain its reasoning out of habit was multiplying its output tokens five to tenfold.

It was a Monday morning, coffee still too hot to drink, when I opened my AI usage dashboard and just stared at it for a second. The number at the top wasn’t wrong, exactly—it was just bigger than anything I’d let myself expect. Month over month, it had been creeping up so quietly that I’d never stopped to really look at it. That morning, I finally asked myself: where is all this money actually going?

I didn’t have a good answer. So I started digging.

It didn’t take long to find the culprit. Almost every dollar traced back to one thing: tokens. Every prompt I’d sent, every response I’d gotten back, all of it billed in these small text chunks I’d never once thought twice about. I’d been sending prompts padded with pleasantries, letting the model ramble on with explanations nobody ever read, and resending entire chat histories on every single call without realizing it.

The first thing that stopped me in my tracks was learning that output tokens cost two to three times more than input tokens. I’d been letting responses run long by default, and I’d basically been paying a premium to read paragraphs I skimmed past anyway.

So I started small. I went back through my prompts and cringed a little—”Could you please kindly help me understand . . .” did the same job as “Explain . . .” for a fraction of the cost. I cut the pleasantries, stopped repeating instructions already in my system prompt, swapped paragraphs for bullet points, and started stripping headers/footers/nav junk from pasted documents before feeding them in.

The real discovery came on the output side. I’d never set max_tokens deliberately, so the model always took full advantage of that. Once I capped responses and got explicit—”answer in two sentences,” “just give me the JSON”—my costs dropped in a way that surprised me. I also realized asking the model to “explain its reasoning” out of habit was multiplying my output tokens five to tenfold. Cutting that was one of the cheapest wins I found.

Then I looked at my chat history and felt a little sick—every call was resending the entire conversation, forever. So I started summarizing older turns into a short recap every 10 turns or so and sending only the relevant section of long documents instead of the whole file.

The moment that really changed things was switching to retrieval instead of brute force. Instead of pasting whole reference documents into every prompt, I chunked everything up, used embeddings to pull just the two to five most relevant pieces per question, and sent only those. Input tokens dropped 50-90% on document-heavy work—probably the single biggest thing I did.

Around the same time, I learned my provider supported prompt caching. I restructured prompts so static content—instructions, examples, unchanging reference material—came first, and the variable part came last. That static portion caches at a steep discount, sometimes 90% cheaper, on every call after the first.

I also had to admit I’d been using the biggest, most capable model for everything, even simple stuff like classifying a ticket or pulling a name from text. I moved simple tasks to smaller, cheaper models and set up a lightweight routing step that decides whether a request needs escalation. I also started batching similar requests instead of firing one call per item, and caching answers to questions that kept recurring instead of regenerating them every time.

By then I’d learned my lesson about flying blind, too. I started logging usage prompt_tokens / usage.completion_tokens on every call, built a simple dashboard to track spend by feature, and now audit my most expensive prompts monthly. The 80/20 rule held up exactly like people say; a handful of prompts were responsible for most of my costs.

Here’s what it looked like on paper: a system processing 100-M input tokens and 50-M output tokens a month, at $0.005/1K input and $0.015/1K output, cost $1,250/month before I touched anything. After trimming input 40% and output 25%, the same workload cost $862.50—$387.50 saved every month, a 31% reduction, or $4,650 a year—without switching providers or downgrading a single model (just an illustration and not actual, results vary based on different parameters).

If I’m honest, this changed how I think about working with AI day to day. Every word has a cost and once you notice that, you can’t really unsee it. If your own AI bill has been creeping up the way mine did, start where I did: look at how long your responses are running, then look at what you’re sending as context—the rest tends to follow naturally.

Disclosure:  Generative AI tools were used to assist with editing and presentation. This blog post reflects my own views and not those of any current or former employer.

Vijay Sudhakar is a Senior Technical Project Manager with more than two decades of experience in enabling digital transformations for Enterprises on Finances.

You Just Read

The Tokens That Made Me Rethink Everything

© 2026 Copyright held by the owner/author(s).