This is week 2 of 5 where we breakdown the conversations we're having with clients that we wish all data and AI leaders understood in October 2026. Next up:
AI Model Deprecation
A common myth about AI models is that they operate similarly to software products.
Take Microsoft Windows or Apple iOS. Once you download the operating system to your phone or computer, you don't expect it to change. When a change does happen, you're prompted to download the latest update. You know your system won't change until the update is installed. You can also depend on keeping the operating system on your device without fear the version will be deprecated and disappear from your device all together.
While open model AIs work similarly (you download it and it doesn't change or disappear), proprietary and frontier models are a different story.
1. Proprietary and frontier models can change or drift without warning.
A study in the Harvard Data Science Review found GPT-4’s accuracy at identifying prime numbers dropped from 84% to 51% from March to June 2023. This isn't rare. Just because a GPT-5 or Claude Opus may be the best model for the task at hand today, doesn't mean that in six months, that "same exact model" will continue performing as well on the same task.
A model's performance can change for a number of reasons: providers swap in new versions under the same name, adjust hidden instructions and settings, change the model weights, or retrain on user feedback. The problem is updates are automatic and often lack transparency.
2. Your favorite AI model (or version of a model) will be removed after about 16-20 months.
And that's for the API version, most model versions only make it a few months before being switched in consumer apps.
Unlike traditional software, an AI model can’t be "patched." Fixing a problem requires training a whole new version. Running multiple old models side by side is expensive, so the older models get cut. You can expect anywhere from 60 days to 6 months notice before a model's APIs are removed.
This is why it's so important to build a test set and perform quality assessment's regularly. It's only a matter of time before the model you built your AI on doesn't work anymore. You need a way of knowing when it's time to change models. (More on this next week).
As a reminder from last week:
Build the truck (the systems and workflows). Then connect the engine (the AI model). And when that AI model is no longer the best option, switch out the engine for another. When built correctly, you get to choose the engine you want based on the need you have.
See you next week for part 3 in our series: How to combat model deprecation. Model testing for the purpose of changing models.
Around the Data and AI World
Who’s Winning With AI and How
This week Wharton released it's fourth annual AI Adoption Report. The study tracks the evolution of AI adoption, ROI, agents, training, and workforce impact with the goal of helping leaders understand "benchmarks that reveal not only whether investments are paying off, but also how transformation is being experienced across the workforce."
This year's report zooms in on three markers of successful AI implementation: proving value, workflow transformation, organizational alignment. Only 7% of organizations succeed on all three markers.
We're going to focus in one just one: Proving Value
When it comes to ROI, one of the leading signals of success is which KPIs are tracked. Specifically, if a company tracks for growth (net-new revenue, sales conversion, output volume, and customer satisfaction) or just for efficiency (operational cost, hours saved, turnaround time, and error rates).
When tracking at least one growth metric, 35% of respondents reported "strong positive returns", against just 13% of those tracking efficiency alone.
Which introduces the question: If you're not tracking for an outcome, how can you expect to achieve said outcome?
According to the study:
Companies are spending as though AI will transform their industries, but measuring the return on that spending merely by how much time it has saved them.
83% of managers+ expect AI to fundamentally transform or majorly impact their industry within two to five years. Yet the most common way departments judge that investment is hours saved (40%), and net-new revenue is the least used of the major measures (27%).
The distance between what leaders expect AI to do to their industry and what they are asking it to prove inside their own departments is the central tension in this year's data.
In your organization, are you tracking for only time-saved? Or also for growth created?
Keep reading the full report or executive summary.
|
Are Data Teams Solving the Right Problems?
If you've been a reader of HIGHLY TECHNICAL for any amount of time, you know we're big on asking the right questions.
SeattleDataGuy is sharing a roundup of 5 questions you need to ask before starting your next data project:
- Are we just trying to substitute an easy problem for a hard one?
- What question or decision is this data supposed to support?
- What is the real source of truth for this problem?
- What knowledge exists only in someone's head or spreadsheet?
- What business process are we modeling?
And my personal favorite:
6. What happens if we don’t take on this project?
Write these on a sticky note and keep them close by.
|
Mark Your Calendars: We're 1 month away from our Q4 Data + AI Meetup!
Join us from 4:30-6:30 on November 19th at the Casual Pint
Most data engineers, ai engineers, and data scientists are working on an island, all alone inside their company. For some of you, there is no one else on your teams that understand tech (let alone AI and Data Engineering). For others, even your closest friends don't really understand what you do.
If you ever find yourself answering "What do you do for work?" with a generalized...
- “I’m just a programmer”
- “I work with data”
- “I build AI”
...then our Data and AI meetup is created for you.
Join us for a night of eating, drinking, and debating things like GEPA vs. MIPROv2, ReAct vs. deterministic workflows, DuckDB vs. Spark, or whether your “AI architecture” really needed a vector database at all.
I guarantee you'll make new friends who love being as nerdy about AI and Data Engineering as you do. And you'll finally be able to tell someone what you actually do for work.
Because meeting kindred spirits IS the point of our meetup. (Leave the learning to YouTube classes, other monthly AI meetups, or schools designed for skill upgrades like Omaha Data Science Academy).
Let’s drink and tell stories on the Contemporary Analysis tab!! RSVP on LinkedIn!
Join us. You’ll be glad you did.
Insights are just a bunch of hocus pocus!
|
|
NATE WATSON
CEO, Contemporary Analysis
|
|
|
Was this email forwarded to you?
Join our weekly HIGHLY TECHNICAL Newsletter for your weekly dose of AI, Tech, and Data Science.
|
WAYS TO WORK TOGETHER
Let us build your Data Solution
We help businesses of all sizes to increase visibility, drive more traffic and improve commercial success.
Schedule a Call
|
|
Enroll for Classes at ODSA
Come learn from 6-9pm, what a data scientist did from 8-5pm”
Talk to an Advisor about your next steps!
|