At work, I have been talking more and more about AI fluency as a skill that companies need if they want to be successful in using AI. AI literacy is about knowing how to use AI tools. AI fluency goes a level deeper: understanding, on a conceptual level, certain aspects of AI, and how these tools and use cases are actually built. You don’t need to write the code, but you do need to understand what is happening under the hood, because that understanding is what separates teams that ship dependable AI from teams that ship demos.
In that spirit, I want to touch upon one aspect that sits at the heart of every serious AI application and is rarely explained in plain terms: evals, and specifically online evals for agent applications.
Picture this: a few weeks after you put an agent into production, someone on the team asks a simple question: “How do we know it’s still working?” The test suite is green. The demo went well. But nobody can say, with any confidence, whether the agent is doing a good job for real users at that moment. That question is the reason online evals exist.
Read what online evals are and how to pick and choose one for your production agents.
https://medium.com/@georgekar91/your-agent-passed-every-test-now-what-4b355a710323
[link] [comments]