Why I Hate Benchmarks: Moving From GPT to GLM-5.3 Flash

I hate benchmarks because they make the model look like the product.

A benchmark can tell you whether a model solved a task in a controlled environment. It usually does not tell you whether the model will return the exact structure your application requires, call the right tool once, use the result, preserve the right state, run through your actual agent runtime, expose which provider answered, or survive retries and partial failures.

I learned this again while trying to reduce Polyform's OpenAI spend.

We use OpenAI heavily. Poly relies on OpenAI-style model and tool contracts, and our Deep Work product runs through Codex. The bill is material. So when I saw how inexpensive GLM-5.3 Flash was, and how good it looked, ignoring it felt irresponsible.

The model was not bad. In several of our tests, it was excellent.

The surprise was how much of the experience I thought came from the model actually came from the system around it.

The migration

I thought I had found free money

The spreadsheet case looked obvious. GLM-5.3 Flash was dramatically cheaper than the models we were using. It was fast, capable, and available through interfaces that looked OpenAI-compatible. OpenRouter offered one API across providers. Z.ai documented JSON mode and tool use.

I thought I was making a model substitution:

It was not a model substitution. It was a systems migration.

Over four days, we merged a provider unification through OpenRouter, repaired a production Codex startup failure, restored Azure as the default, added a native Z.ai path, built model-specific regression gates, moved Deep Work back to direct OpenAI, and then merged a broad rollback to the native-provider architecture.

The first OpenRouter rewrite touched 150 files. The rollback touched 174.

That is not evidence that OpenRouter or GLM is bad. It is evidence that a model is attached to far more of your application than its endpoint URL.

“Returns JSON” is not one capability

One of the first surprises was structured output. People use “JSON mode” and “structured output” as if they mean the same thing. They do not.

OpenAI's Structured Outputs documentation distinguishes schema-constrained output from the older JSON mode, which only ensures valid JSON. Its function-calling controls can also require a tool call or force one specific function, while strict schemas constrain the arguments.

That changes how you build a product. If an application requests a label, score, and acceptance decision, it does not merely need something that can be parsed as JSON. It needs the required keys, correct native types, allowed values, no invented fields, and a clear failure when the contract cannot be met.

Z.ai's structured-output guide uses json_object, puts the expected structure in the messages, and demonstrates application-side JSON Schema validation. That is reasonable. It is also a different production contract.

In our live testing, GLM did not reliably enforce the final structured response from the API option alone. So we added a typed final_response tool, validated its payload with our application schema, prompted the model to call it, and retried once if the model answered in text instead.

if response_schema is not None and model_is_glm:
    tools.append(final_response_tool(response_schema))
    response_schema = None

response = call_model(tools=tools)

if response.called("final_response"):
    return validate(response.arguments, expected_schema)

if not retry_used:
    retry("Call final_response now. Do not respond with text.")

raise RuntimeError("Model did not call the required final response tool")

That wrapper made the model more usable. It also meant we were no longer comparing two models. We were comparing OpenAI's native contract with GLM plus a contract-emulation layer that we owned.

A tool call is a transaction, not a suggestion

The same thing happened with tools. For a demo, success can mean that the model called a tool. For Poly, that is only the middle of the transaction.

We had to add duplicate-action suppression because the model could repeat a tool that had already succeeded. We returned the prior result and explicitly told it not to perform the action again. We capped the agent loop at eight rounds. We added hard failures for duplicate mutating calls in evaluation.

“Called the tool twice” is not a small quality issue when the tool creates a task, changes a metric, sends a message, or starts another agent.

An OpenAI-compatible API is not an OpenAI-compatible system

OpenRouter solved real problems for us. It gave us one boundary, broad model access, provider routing, fallbacks, and provider-reported cost. I still think that is useful.

But a normalized request shape does not create normalized behavior. OpenRouter documents structured outputs for compatible models and provides controls such as require_parameters to keep routing on providers that support requested parameters. Its provider-routing documentation also makes the underlying behavior explicit: the router can load balance across providers and use fallbacks.

Those are features. They also create work for a production application.

When we introduced fallback routing, we had to separate three facts that had previously felt like one: the tier the user requested, the model that actually answered, and the cost reported by the provider.

Our first implementation lost that distinction in Poly. Then we fixed the main response but discovered that multi-call turns were still attributing all usage to the final model. Then we fixed that and discovered a separate metadata-generation call was still being recorded under the requested tier instead of the runtime model.

The same migration missed saved model selections in dashboards. Then it missed dataset reports. Then it missed a camel-case discriminator used by one persisted shape.

The router made it easy to send a request. It did not make every part of our product understand what happened.

The failures benchmarks miss

The model passed. The runtime failed.

The most painful failure had almost nothing to do with intelligence.

Our first OpenRouter change included a live GLM probe through Codex. It returned successfully. Focused tests passed. The change merged.

Then production Deep Work tasks failed at Codex startup.

The generated model catalog was missing one field required by the pinned Codex parser: supports_parallel_tool_calls. Adding that single field fixed the startup path.

This is exactly the kind of failure a model benchmark cannot see. The model may be perfectly capable. The API may respond correctly. The agent can still be unusable because the catalog schema, CLI version, authentication boundary, sandbox network, or result capsule differs from the environment used in the test.

We also learned that provider-specific reasoning items could not simply be replayed across providers. A conversation could keep its user messages, assistant text, and completed tool context, but opaque reasoning identifiers from one provider had to be dropped before another provider took the next turn. The API shape looked similar. The state was not portable.

Then the benchmark became its own product

At that point, the obvious response was: build a better benchmark. So we tried.

We designed a production-derived evaluator that could import real Poly, Workflow, and Deep Work cases; sanitize them; freeze tool results; replay them without external effects; compare tier-specific candidates; use a stronger model as a blind judge; enforce a spend cap; and preserve enough provenance to know exactly what ran.

That evaluator change grew to 7,467 added lines.

Review found 31 high-priority problems that we reproduced and fixed. The evaluator had to learn how to reconstruct semantic tool operations, bind frozen results to exact arguments, preserve region, isolate side effects, select the right workflow output, prevent history contamination, detect embedded credentials, verify the runtime model, and constrain a multi-call run to its remaining budget.

Three high-priority problems were still open when we closed it.

The irony was useful. The benchmark needed many of the same contracts as the product. If it did not reconstruct the exact history, tool effects, output, region, runtime model, and failure boundary, it could confidently grade a situation that never happened.

What the real runs told us

Once we separated the product surfaces, the results became much more informative.

GLM looked viable for bounded Regular work and not yet viable for our Codex-based Deep Work. Azure also produced an intermittent empty response and a post-tool 500 during live Poly testing. No provider was flawless. The right answer was workload-specific.

The cheapest tokens can create the most expensive system

Token price matters. At our volume, it matters a lot.

But token price is only one part of the cost of an accepted outcome. The real cost also includes the provider adapter, schema and tool-call failures, repeated turns, latency, duplicate actions, runtime compatibility, migrations, rollback, observability, evaluation, and human review when the system is uncertain.

A model that costs one tenth as much per token is not cheaper if it needs three times as many turns, produces more failed outcomes, or requires a dedicated compatibility layer that your team must maintain.

The opposite is also true. An expensive model is not automatically worth it because its API is pleasant. If a cheaper model passes the exact production contract for a bounded workload, use it.

The unit I care about is cost per accepted outcome.

What I benchmark now

I still use benchmarks. I just no longer let them make the decision.

1. Use the real endpoint.

   Run through the exact production provider and endpoint, not a convenient proxy or playground.

2. Use the real contract.

   Keep the production prompt, schema, tools, permissions, and tool results intact.

3. Use the real runtime.

   Include the actual agent, resume behavior, versioned catalog, state, and failure paths.

4. Inspect every effect.

   Require consequential actions to happen exactly once with the complete authorized payload.

5. Preserve the evidence.

   Verify identity, provenance, model attribution, cost, and the artifacts the product needs.

6. Repeat the safety case.

   Look at failure categories across runs, not only an average score.

7. Price the accepted result.

   Include retries, latency, engineering, and human review in the final cost.

A 96 percent pass rate can hide the one failure that duplicates a refund or publishes an internal control file. Failure categories matter more than a smooth average.

What I actually learned about OpenAI

I started this experiment thinking we were paying too much for a model.

I ended it with a better understanding of what we were buying.

We were buying models, but we were also buying a set of contracts that had grown together: the Responses API, strict function schemas, structured outputs, tool controls, reasoning state, model metadata, and Codex. Those contracts matched the way Polyform already worked.

That does not mean we will use OpenAI for everything forever. We should not. GLM was genuinely impressive, and I expect cheaper models to keep winning more bounded workloads. OpenRouter remains useful when broad access and routing matter more than exact provider semantics.

But switching is not a row change in a benchmark spreadsheet.

It is a product migration. The farther your AI has moved from chat into real work, the more true that becomes.

This is also why Polyform's own AI story is not “we found a smarter model.” We use strong underlying models, including Codex, inside a complete execution environment with maintained product state, permissions, visible work, validation, and recovery. When the AI is blocked or the result needs judgment, the work has an accountable owner instead of becoming a dead end.

That is why I hate benchmarks.

They show you the price and capability of the engine. They do not show you the cost of making the whole machine run.

References and further reading