This is week 3 where we breakdown the conversations we're having with clients that we wish all data and AI leaders understood in October 2026. Next up:
Model testing for the purpose of changing models.
When your team asks, "Is the new AI model better? Should we switch?"
You don't want to be stuck saying, "I don't know. We don't have a way of knowing."
You need a test set and to perform quality assessments (QA) regularly. Running quality assessments on AI is different than testing traditional software.
Traditional software fails loudly. You click a button, it goes to the wrong page, the test breaks. Pass or fail. You know immediately.
AI fails quietly. Confidently. It returns an answer that looks fluent, feels grounded, cites a real document but still gets it wrong.
“If you cannot measure AI, you cannot trust it.” - Archana Raghu at CANalytics 2026
This is why you need a test set.
Your test set is a collection of your questions, your data, and the answers you expect. It’s how you measure how well a model performs over time, and how you test a new model before you trust it with your work.
Your test set isn't just for QA, it's also IP. A well-built test set is an asset your company owns. It captures what “good” looks like for your business, and it keeps its value no matter which model you’re running. And AI companies are paying large sums for high-quality evaluation data like this. Treat yours accordingly.
Remember, your test set isn't helpful if you aren't using it consistently. It's important to build a QA SOP into your workflows. It can be as simple as:
- Identify what you’re evaluating.
- Choose your evaluators based on the problem you’re solving and the risk you’re measuring.
- Establish a baseline.
- Compare every model update and every new model against that baseline.
- Build an eval loop you can rerun and build upon.
When you’re using AI agents, test more than the final answer. An agent plans, calls tools, and makes decisions across multiple steps. The failure could be in any one of them.
If the truck is your systems and workflows; and the AI model is your engine, your test set is your test track. Before you swap in a new engine, run it on the track.
Missed last week's post on AI Model's changing and being retired? View past newsletters here!
Around the Data and AI World
Speaking of models just going off-line when they're just fine and you're still using them in your applications...
OpenAI Models going offline tomorrow
dev.to pulled together a list of AI models going offline soon, and a two-minute repo check, and 5 'gotchas' to look for.
Forward this to your AI engineers.
|
US Gov bands DeepSeek
Since at least July, the Department of Homeland Security has been monitoring "flows of traffic in and out of government websites to address potential risks from DeepSeek."
This week, the DHS banned the use of and aquisitions of products by DeepSeek products and third-party services from all US civilian agency IT systems.
Prior to this week, several US agencies, including the Navy and at least 3 states had issued their own bans against the use of DeepSeek.
Later this month, the FCC is voting on if electronics can be tested in China:
The Federal Communications Commission will vote on October 29 to bar all Chinese labs from testing electronic devices such as smartphones, cameras and computers for use in the US. Last year, 82% of US electronics were tested in China.
Read more.
If you've been reading the newsletter for a long time, then you may remember we told you to abandon DeepSeek about 37 seconds after we read the terms of service. Now the government is proving our point... I guess it pays to read HIGHLY TECHNICAL.
|
Data + AI EVents
November 5th - AI Omaha
5:30 PM to 7:00 PM at Werner Enterprises
What happens when AI helps your teams move faster than your processes were designed to handle? Join us at Werner Enterprises for a behind-the-scenes look at how one of Omaha's leading companies is addressing that scenario and putting AI to work to do it.
AI is changing the way software is built. But the real transformation goes beyond writing code faster. It's changing how teams collaborate, make decisions, and get work done.
Jason Nash, Director of Architecture and AI, and Jason Schwarz, Agile Coach & Release Train Engineer, will share how Werner has evolved its software engineering practices with AI and what they've learned as they've adapted their processes to keep pace with increasingly capable teams.
RSVP on Meetup!
|
November 19th - Q4 Data + AI Meetup
Join us from 4:30-6:30 on November 19th at the Casual Pint
Most data engineers, ai engineers, and data scientists are working on an island, all alone inside their company. For some of you, there is no one else on your teams that understand tech (let alone AI and Data Engineering). For others, even your closest friends don't really understand what you do.
If you ever find yourself answering "What do you do for work?" with a generalized...
- “I’m just a programmer”
- “I work with data”
- “I build AI”
...then our Data and AI meetup is created for you.
Join us for a night of eating, drinking, and debating things like GEPA vs. MIPROv2, ReAct vs. deterministic workflows, DuckDB vs. Spark, or whether your “AI architecture” really needed a vector database at all.
RSVP on LinkedIn to let us know you'll be there!
|
Look out for next week's newsletter. We're digging into why you shouldn't trust your AI vendors privacy policy at face value, and what you can do to mitigate privacy risk.
Rev your engines, run your evals,
|
|
NATE WATSON
CEO, Contemporary Analysis
|
|
|
Was this email forwarded to you?
Join our weekly HIGHLY TECHNICAL Newsletter for your weekly dose of AI, Tech, and Data Science.
|
WAYS TO WORK TOGETHER
Let us build your Data Solution
We help businesses of all sizes to increase visibility, drive more traffic and improve commercial success.
Schedule a Call
|
|
Enroll for Classes at ODSA
Come learn from 6-9pm, what a data scientist did from 8-5pm”
Talk to an Advisor about your next steps!
|