Lessons learned from building AI assistants for cloud infrastructure

We learned a lot of lessons building our AI assistant, Pulumi Copilot. Here are some key insights:

1. Minimize LLM Usage: Let traditional code handle deterministic tasks, reserve LLMs for natural language work
2. Decompose into Skills: Break complex tasks into modular units that combine LLM and traditional code appropriately
3. Test Rigorously: Use multiple validation approaches, including LLMs testing LLMs
4. Learn from Hallucinations: Sometimes incorrect outputs reveal user expectations
5. Learn from Users Continuously: User interactions improve our AI systems - from training better skills to catching hallucinations and revealing product opportunities.

Here is the longer more detailed blog post. Be curious on yall's thoughts.