Building an AI voice agent from scratch: the parts that actually took our time

We recently finished our first proper phone-based AI agent and did a postmortem on where the engineering time actually went. It wasn't the LLM.

We spent maybe 15% of the time getting the actual conversation behaviour right. The rest went into all the boring stuff around it.

The biggest time sinks were:

1. Telephony SIP setup, call routing, dealing with weird edge cases. This took considerably longer than expected.
2. Turn detection + barge-in Getting the agent to stop talking when someone interrupts sounds simple until you have to make it work reliably on an actual phone call.
3. Observability Our first version basically dumped transcripts and events into logs. Technically we had logging. Practically, nobody wanted to search through raw JSON to figure out why a call went wrong.
4. Handling failures Timeouts, dropped calls, tools taking too long, STT returning something weird, etc.

This was the stuff that didn't show up in the first demo and then suddenly became everyone's problem.

We looked at managed voice agent platforms halfway through the project, including Vapi, Retell and Dasha.

In retrospect, I think we would've been better off using a managed runtime from the start and spending our engineering time on the actual business logic.

If you've built a production voice agent, what part ended up eating most of your time? Not the cool demo stuff. The annoying part nobody puts in the architecture diagram.