Every AI product needs a way to change models

AI models are changing faster than most products built on them.

New models arrive with better benchmark results, lower prices, longer context windows, or a capability that looks meaningfully different. Kimi K3 caught my attention because its benchmark performance looked strong. I had been building an AI tutor in Telegram, and I wanted to know whether Kimi could make it better.

The question in my head was simple: how quickly could I put a promising new model into the same job and learn whether it worked?

I had never designed the tutor for that kind of experiment. What I expected to be a model test became a difficult migration. The experience changed how I think about the architecture of an AI product.

Models will keep changing. The product needs a reliable way to test them without starting over each time.

My first test was too close to a full migration

The tutor used AI across live conversation, lesson planning, grading, diagnosis, and the summaries that helped it decide what to teach next. Its existing provider was woven through all of those paths.

Kimi looked relatively easy to try. It offered an OpenAI-compatible API, predictable subscription pricing, and stateless requests. The tutor could continue to own the learner's history instead of depending on a provider session.

The first unit tests and end-to-end checks passed. Then I started using Kimi in lessons.

The shared API format had hidden important differences between the models. Some replies took several minutes. A subscription limit could interrupt a session. The model sometimes produced a sensible learner-facing answer but omitted the markers the tutor needed to advance the lesson, causing an exercise to repeat.

I also learned how easy it was to blame the model for the wrong problem. One awkward English exercise looked like a Kimi failure. I traced the sentence through the system and found it in one of my own templates. Switching providers would have carried the same mistake into the next version.

I was trying to judge Kimi while also discovering all the places where the product depended on the behavior of the previous model. That made the evaluation slower and the evidence harder to interpret.

API compatibility is a small part of portability

Before this experiment, I thought portability mostly meant giving providers the same input and translating their outputs into a common format.

That gets a request from one endpoint to another. A production product depends on much more.

A tutor needs a response soon enough to preserve the rhythm of a conversation. It needs to know whether an answer was correct, where the learner is in the lesson, and what should happen next. It needs predictable behavior when a provider is slow, reaches a usage limit, or returns only part of a response.

Those requirements became the stable contract I wanted the product to own.

I moved important lesson instructions into ordinary code so they could not drift with the model. I started treating model output as a proposal, validating it before it could change lesson state. I made latency, usage limits, and provider transitions visible. I also separated product failures from provider failures so the next model would not be credited or blamed for the wrong thing.

This work made the Kimi evaluation fairer. It also gave me a much clearer definition of portability: the ability to put another model inside the same product job, observe how it performs, and reverse the decision without rebuilding the product around it.

The use case needed its own evaluation

Kimi's benchmark results were a good reason to investigate. They could not tell me whether a French lesson would feel responsive or whether the tutor would advance reliably after grading an answer.

I needed to evaluate the model on the job the product was asking it to do.

For the tutor, that meant looking at several things separately:

1. Did the learner-facing response help?
2. Did it arrive quickly enough for a conversation?
3. Could the product validate the model's decisions before changing lesson state?
4. What happened when the provider timed out or reached a usage limit?
5. How much extra machinery did the product need to make the provider dependable?

The fifth question became especially important. Kimi could produce strong individual answers, but dependable operation required long timeouts, partial-response recovery, a second paid endpoint for quota limits, and repairs when structured instructions were missed.

That surrounding code belonged in the model decision. It would remain part of the product after the benchmark comparison was forgotten.

I used the next switch to build a repeatable path

I eventually moved the tutor's production inference to OpenAI's Codex. Strict structured output was a better fit for a product that needed to validate a result before advancing a lesson.

This time, I did not switch the whole product in one move. I first routed only my own live lessons to Codex. Then I moved planning, diagnosis, safety checks, and the remaining lesson paths one at a time while Kimi stayed available. Claude helped me engineer and review the migration.

The canary caught a mistake in my Codex integration. One schema treated optional fields in a way the API rejected. I fixed it and added a permanent check before expanding the rollout.

That failure was useful. A portable setup should make a provider-specific problem small enough to understand and contain. It should also leave a clear way back while the new path is still unproven.

Once every inference path had moved and the full test suite passed, I removed Kimi from the runtime.

What I am setting up for the next model

I do not think I have solved model portability. The next promising model will have its own strengths, constraints, and failure modes. I now know which parts of the setup need to remain stable.

The tutor needs a provider-independent contract for the work that affects the learner: what context goes in, what validated result comes back, who owns lesson state, and how failures appear to the user.

It also needs a small set of representative lessons and product tasks that every new model can run. Benchmarks can tell me which models deserve attention. These product evaluations tell me whether a model deserves responsibility inside the tutor.

Canary routing and rollback need to be normal product capabilities. I want to test a new provider on my own lessons, expand it one workflow at a time, compare what happened, and stop without disrupting everyone else.

There is also a balance to strike. A common interface can make providers interchangeable, but reducing every model to the same lowest-common-denominator features would hide the capabilities worth testing. I want the core product contract to stay stable while each provider adapter can use the model's particular strengths.

The Kimi experiment was harder than I expected. It gave me a better product than a simple provider comparison would have. The tutor now owns more of its state, evaluation, and recovery, and the next model test has a clearer path through the product.

I expect model exploration to remain continuous. The goal is to make the next question much cheaper to answer: does this new capability actually improve the experience I am building?