A Weekly Upskill for Yourself, Your Data, Your Team

June 11, 2026

Issue 115

"If you cannot measure AI, you cannot trust it."

This is an excerpt from “Quality Assurance in AI: From Measurement to Trust" presented by Archana Raghu at CANalytics 2026.

Archana spent her talk walking through what it actually takes to build a quality evaluation program for AI systems: covering output quality measurement, fairness testing, adversarial testing, and agentic QA. She explained that the tools and instincts that work for traditional software testing don't work for testing AI.

Traditional software fails loudly. You click a button, it goes to the wrong page, the test breaks. Pass or fail. You know immediately.

AI fails quietly. Confidently. It returns an answer that looks fluent, feels grounded, cites a real document but still gets it wrong. In one example, Archana showed a model correctly retrieving information about two employees but attributed the wrong performance data to the wrong person. Faithfulness score? Passed. Context relevance? Passed. The answer was still wrong.

With AI, a passing score doesn't mean a correct answer. You're no longer checking whether the output exists and is formatted correctly. You're measuring quality signals — faithfulness, context utilization, attribution — across multiple layers, because any one metric can pass while the system quietly produces something harmful.

Now multiply that problem by ten when you move to agentic AI.

A single-response model is testable. Input goes in, output comes out, you evaluate it. An agent is different. It plans. It calls tools. It observes results and decides what to do next, often across five or six intermediate steps before producing a final answer.

The failure could be in the answer, or any of the steps.

Did the agent choose the right plan? Call the right tool? Interpret the retrieved data correctly before deciding what to do next? When you're testing an agent you're evaluating every decision in the chain.

This is unsolved territory. The evaluation frameworks that work for RAG systems don't map cleanly onto agentic systems. The same task can produce a different plan, different tool calls, and different intermediate steps every time it runs, and when the final answer is wrong, tracing the failure back to a specific decision in the chain is difficult.

"Did the AI get the right answer?" is the wrong question. The right questions are "where in the chain could this have gone wrong, and do I have the observability to find it?"

Here's how to start:

Identify what you're evaluating, choose your evaluators based on the problem you're trying to solve and the risk you're trying to measure, establish a baseline, and compare every iteration against it. Create an eval loop you can rerun and build upon.

Claude Fable 5

On Tuesday, Anthropic launched Claude Fable 5, "a Mythos-class model that [they've] made safe for general use" with a host of invisible guardrails.

The new class of models also comes with new data retention policies.

"Fable 5, Mythos 5, and future models with similar or higher capability levels... will require 30-day retention for all traffic... on both first- and third-party surfaces."

Anthropic clarifies:

"We won’t use this data to train new Claude models, or for any non-safety-related purpose, and we’ve instituted new privacy protections including logging all human access to the data and ensuring its deletion after 30 days in almost all cases. The data will help us defend against complex and novel attacks (including new jailbreaks and attacks that operate across many requests) as well as help us identify and reduce false positives."

However, just a day after releasing Fable 5, Anthropic has walked back it's invisible guardrails after backlash from the AI research community. ​

Preventing AI-aided bioweapons

On June 3rd, 50 prominent players in AI, biotechnology, and national security, including leadership at OpenAI, Anthropic, and Google DeepMind, penned a letter to Congress urging stricter regulation and reporting from synthesized biology and DNA companies.

The letter asks Congress to require DNA synthesis companies and hardware manufacturers to vet who they're selling to and keep detailed records of every order, so that anything that slips through initial screening can be traced back to its source.

Right now, reporting is voluntary. The concern is that LLMs and AI tools trained on biological data are lowering the barrier for non-experts to access sophisticated information on building deadly pathogens (and using equipment that is getting cheaper by the year.)

​Read the full article.​

From the CAN Slack Channel

​A Cypherpunk’s Library​

Great resource on cryptography: "A personal collection of good public-domain reads. Nothing for sale, nothing to take down."

​What to do when your executives pull the plug on your BI tool.​

Data and analytics expert, Ryan Dolley realized a Tableau exodus has begun. He gives a few recommendations for what to do if your leadership decided to remove Tableau from the tech stack.

Nebraska Data and Analytics Forum

The Data Journey That's Powering OPPD's Future

Thursday, June 25th at Omaha Public Power District

OPPD will be sharing about how they are leveraging data to Power Omaha's future. This will cover the creation of their first data strategy, the launch of a formal data governance council, and the implementation of a data catalog and data platform.

​RSVP on Meetup​

OMA x AI

By UNO

Tuesday, June 30th

"At OMA x AI, we're focused really on the intersection of different industries with small businesses, with the education sector and nonprofits. Because that's really where innovation happens." - Jason Coleman, UNO's associate vice chancellor for Innovative and Learning Centric Initiatives on the TEN Podcast. ​

​Get Your Ticket Today​