What has been the biggest production bottleneck for your AI agents?

Building an agent that works in a controlled demo is one thing; keeping it reliable in production is another.

For those who have actually deployed AI agents, what has caused the most problems?

Tool/API reliability? Context management? Memory? Authentication and permissions? Evaluation? Hallucinations? Cost and latency? Observability? Human-in-the-loop workflows? Integration with legacy systems?

I'm especially interested in what changed between the prototype and production. What problem did you underestimate initially, and how did you eventually solve it?

Real implementation experiences would be much more useful than theoretical answers.