I keep an open-weight model running on my own laptop: Qwen3.8-Flash-Next, quantized to 3 bits, served by llama.cpp on a MacBook Pro with an M5 Max and 128 GB of memory. It costs nothing per token and the server only listens on localhost, so I use it for first drafts and document work, and lean on frontier models for research and for grading.
But "free and private" is not a reason to trust it. So before leaning on it, I ran it through an eval, the same way I would gate a model at work.
The setup
31 model calls across nine categories: extracting deal terms from two real SEC 8-K filings, finance math, factual recall with a hallucination probe, current events with no search, logic puzzles, rubric-based triage of real items, writing, strict JSON instruction following, and low versus high reasoning effort. A frontier model graded each answer and fact-checked it against live sources. Every prompt ran once, so the results are indicative, not statistical.
Where it was excellent
- Reading what I give it. 48 of 48 required deal terms extracted correctly across four runs. Every fee, date, and condition matched. (Once, it added a per-share figure that was not in the text.)
- Logic. 6 of 6 reasoning runs correct, including catching a trick in a tender-offer proration problem.
- Following instructions. 7 of 7 checks on raw JSON with exact keys and word limits.
- Skepticism. In triage, it caught a real bug in the triage pipeline's input data: amended filings were being counted twice.
Where it failed, quietly
- Current facts: 0 of 3, though it did say its knowledge stopped around mid-2025.
- Obscure facts: it invented a mountain's height rather than say "not sure."
- Side remarks: the math was right, then it attached a false state-tax rule as a helpful caveat.
- Long multi-step math: a 30-year amortization returned nothing in three tries. It kept re-checking itself until it ran out of tokens.
- Overthinking: high reasoning effort was 2 to 7 times slower with no accuracy gain.
One small, funny one: unprompted, it used em dashes in 13 of 31 answers. Told not to, it stopped. Instructions work when you give them.
The rules I use it by now
Trust it for pulling terms from a document I paste in, logic and checklists, structured output, and first drafts. Double-check any number it computes, any tax or legal claim, any fact that was not in my document, and any rubric score (it runs one to three points generous). Never use it for anything current or for long multi-period math; a spreadsheet does that better.
Why this matters beyond my laptop
Models are strongest when they read and weakest when they recall. That is why our natural-language BI work at Visa put deterministic retrieval and a governed metrics catalog underneath, and the model on top for interpretation and phrasing only. It is an architecture decision, not a model choice. Give the model the source, constrain what it may assert, and test the side remarks as hard as the headline answer. Running a model yourself, failures and all, is the cheapest way to build that instinct. It also makes you a much better judge of the vendor demos that skip this part.
Next on the bench: the same model, several samples per prompt, and a first pass at an eval plan for an AI shopping agent's checkout.