Two Hours of Work That Takes a Week

For Descript, testing a new model was a couple of hours of work and a week of waiting. The team removed the waiting. Evaluations run in an hour or two now, and nobody has to ask an engineer for time. The full architecture is in the Descript case study.

When model access is wired provider by provider, testing a new model starts with engineering work. When the person who wants to try a model is not the person who can add it, the request has to be explained, scheduled, and waited on. That is a queueing problem, and at Descript it set how many models the team tried.

What the queue cost Descript

Descript, the AI video and audio editing platform, ran into this while building Underlord, its video editing agent. Before OpenRouter, the team maintained three direct integrations, with OpenAI, Anthropic, and Google, and wrote its own fallback logic between them. Adding a model took a couple of hours. Getting those hours onto an engineer’s calendar took days, so a promising model could sit for a week before anyone knew whether it was worth pursuing.

The expensive part was the waiting. The team tried the models that were worth interrupting an engineer for and skipped the rest.

How Descript evaluates a model now

Aleks Mistratov, Head of AI Product at Descript, describes the loop now.

“I’m on a walk, I’m very online to my own detriment, and I see a tweet that a new model is out. I go into Slack and say: run evals on this model on OpenRouter, and link to it. An hour or two later we’ve run evals in our harness on that model.”

The mechanism is Claude Tag, Anthropic’s Slack integration, which Descript configured for this loop. It runs the evaluations and opens the pull requests, and a human reviews and approves.

What made that workflow automatable is that trying a model no longer requires building a provider integration first. Testing one does not mean opening a vendor relationship or wiring up another API endpoint. There is still work in it. Nobody has to be asked for it.

What weekly evaluation changed

Descript now evaluates models multiple times a week. Most of what the team tests does not ship. Three things follow from that.

Model choice is empirical. Descript measures new releases against its own test cases and changes what it ships when something clears the bar. The alternative the team had before was standardizing on a model early and revisiting the decision when something forced it.

New releases are not events. On a recent Anthropic release, Descript went from announcement to live in a couple of hours. The steps that remain are internal. Run the evaluations, decide whether the model beats the incumbent, add it to the model picker, deploy. Most of that work comes from how Descript’s own code is structured rather than from the connection to the model.

The field widened. Descript never built a direct integration with xAI. Grok 4.5 runs in production now, added through the same path as everything else.

How Descript runs the models it commits to

None of this means routing everything through a shared endpoint and accepting what comes back. Descript brings its own key to Baseten for a dedicated deployment of an open-weight model, with OpenRouter routing to Baseten first and other providers configured as fallbacks behind it. Those fallbacks swap the inference provider and keep the model constant, so a failure at one host does not change which model answers.

Mistratov’s advice to another engineer weighing the same change is short.

“It’s far easier than trying to figure out all these different connections. You do it once and everything else is a parameter.”

Descript uses the same integration to test new models and to run the ones it keeps. Testing is how the team finds a model. Routing configuration is how it runs one.

Read the case study

The Descript case study has the rest of the architecture, including how Descript sets inference-provider priority per model, what happens when a provider degrades mid-traffic, and what the team’s on-call looks like now.