Every June I run the same exercise: pull the current crop of open-weight models, put them on hardware we'd actually rack for a client, and see what's changed since last summer. This year the honest headline is that the gap between "open-weight" and "frontier proprietary" narrowed faster than at any point since we started doing this.

What we benchmarked

The short list this year was Llama 3.3, the latest Qwen releases, and a Mistral variant we'd fine-tuned on a client's internal documentation. We ran all three against the same task set we use for enterprise field-service and healthcare workflows: instruction following on multi-step procedures, tool-call reliability, and factual grounding against a retrieved document set.

Findings

Llama 3.3 continues to be the safest default — broad community support, predictable behavior, and strong enough performance that most partners don't need to look further. Qwen's multilingual handling pulled ahead noticeably, which matters more than people expect once you're deploying across a field-service org with technicians who don't all work in English. And the fine-tuned Mistral variant, unsurprisingly, beat both general models on our client's own documentation — the lesson there isn't "Mistral is better," it's that a smaller model fine-tuned on your own SOPs will usually beat a bigger general model that's never seen them.

The infrastructure story

The more interesting shift wasn't in the models — it was in how cheap it's gotten to serve them. A configuration that needed a rack of GPUs two summers ago now runs comfortably on a single modern inference server for the model sizes most of our partners actually need. That's the change that matters most for on-premise deployment: sovereignty used to come with a real infrastructure tax. This year, for most of our use cases, it basically doesn't.

Next summer's list is already forming. If you're evaluating open-weight models for a regulated deployment and want to compare notes, get in touch — this is the kind of thing we'd rather share than gatekeep.